How to make picking work correctly

[arkdemon]

Hello everyone. With the help of the forum, I implemented a picking function a while ago. However, I found that it is not very precise and it does not pick sometimes when it should.
I think the problem comes from my unproject function(fov is 60).
so, here is my unproject function:

Code: Select all

vec3 unproject(float x, float y, float w, float h,
                    mat4 proj, mat4 mv, float depth){
    vec4 rw = inverse(proj * mv) * fst::vec4((x/w)*2.f-1.f,(y/h)*2.f-1.f,depth*2.f-1.f,1.f);
    return fst::vec3(rw) / rw.w;
}

The Picking class:

Code: Select all

struct RayCastInfo
{
   NewtonBody* hit;
   float dist;

   void Reset (){
      dist = 1000.f;
      hit = 0;
   }

   RayCastInfo () : dist(1000.f), hit(){ Reset(); }
};

static unsigned nRayCastPrefilter (const NewtonBody* body, const NewtonCollision* collision, void* userData){
    return 1;
}

static float nRayCastFilter(const NewtonBody* const body, const NewtonCollision* const shapeHit,
                      const float* const hitContact, const float* const normal, long long collisionID,
                      void* const userData, float intersectParam){
   RayCastInfo *rc = (RayCastInfo*) userData;
   if (intersectParam < rc->dist && intersectParam != 0.f){ // hit is closer than previous, so update
      rc->dist = intersectParam;
      rc->hit = (NewtonBody*)(body);
   }
   return intersectParam;
}


static void RayCast (float *p0, float *p1, RayCastInfo &rc){
    rc.Reset();
    NewtonWorldRayCast(world, p0, p1, nRayCastFilter,&rc, nRayCastPrefilter, 0);
}

and how I use it:

Code: Select all

vec3 p0 = unproject(x, y, w, h, m_proj, m_cam->view(), 0.f);
    vec3 p1 = unproject(x, y, w, h, m_proj, m_cam->view(), 1.f);

    RayCastInfo ray;
    RayCast(&p0.x,&p1.x, ray);

   if(!ray.hit){
       picked = 0;
   }
   else{
      //I treat it

Thank you for your answers and suggestions.

[Julio Jerez]

you can no just do the matrix inverse and expect the transform to be right because there is a perspective dive involve
here is the function that I use

Code: Select all

dVector DemoCamera::ScreenToWorld (const dVector& screenPoint) const
{
   GLdouble winX = screenPoint.m_x; //Store the x cord;
   GLdouble winY = screenPoint.m_y; //Store the y cord
   GLdouble winZ = screenPoint.m_z; //Store the Z cord

   //Now windows coordinates start with (0,0) being at the top left
   //whereas OpenGL cords start lower left so we have to do this to convert it:
   //Remember we got viewport value before
   winY = (dFloat)m_viewport[3] - winY;

   GLdouble objx;
   GLdouble objy;
   GLdouble objz;
   gluUnProject (winX, winY, winZ, m_modelViewMatrix, m_projectionViewMatrix, (GLint*)&m_viewport, &objx, &objy, &objz);

   return dVector (dFloat(objx), dFloat(objy), dFloat(objz));
}

if your code does not have function gluUnProject, you can look up the code for it by searching in google, the function is open source, you can make you own version
https://www.opengl.org/wiki/GluProject_ ... oject_code

[arkdemon]

Hello. Thank you for your answer. It works now! The error was that I forgot the "height - winY". Thank you again.

PS: The code for glUnproject is the same as mine at the exception that I do not take into account the position of the viewport nor the winZ (but I missed the "height - winY"y). But what is winZ?

Thank you.

[Julio Jerez]

winz id the depth value of the screen, it is a value that goes form zero at the front clipping plane to one at the back clipping plane