dgType.h: dgSort asserts

A place to discuss everything related to Newton Dynamics.

Moderators: Sascha Willems, walaber

dgType.h: dgSort asserts

Postby Krystian » Thu Oct 13, 2011 2:00 pm

I have updated Newton from SVN (rev 922, there was a few updates, but only in dCompilerKit), and I got ASSERT in dgTypes.h - dgSort:
Code: Select all
   
void dgSort (...
...
for (dgInt32 i = 1; i < elements; i ++) {
      dgInt32 j = i;
      T tmp (array[i]);
      //for (; j && (compare (&array[j - 1], &tmp, context) > 0); j --) {
      for (; compare (&array[j - 1], &tmp, context) > 0; j --) {
         _ASSERTE (j > 0);  <-----
         array[j] = array[j - 1];
      }
      array[j] = tmp;
   }
...

Call stack:
   Game.exe!dgSort<dgIsland>(dgIsland * const array, int elements, int (const dgIsland *, const dgIsland *, void *)* compare, void * const context)  Line 485 + 0x29 bytes   C++
    Game.exe!dgWorldDynamicUpdate::UpdateDynamics(dgWorld * const world, int archModel, float timestep)  Line 175 + 0x1a bytes   C++
    Game.exe!dgWorld::Update(float timestep)  Line 686   C++
    Game.exe!Newton::UpdatePhysics(float timestep)  Line 115   C++
    Game.exe!NewtonUpdate(const NewtonWorld * const newtonWorld, float timestep)  Line 720   C++
    Game.exe!zwGame::NextFrame()  Line 250 + 0x16 bytes   C++
    Game.exe!RunGame(zwGameInit * init)  Line 158   C++
    Game.exe!main(int argc, char * * argv)  Line 201 + 0x9 bytes   C++



Yesterday there was a few changes (svn) inside that function. Is it a 'know' issue?
Warning: I'm learning C/CPP/ARM-NEON/Newton/AndroidNDK/OpenGL ES, so whatever I post here, keep in mind that I can be totally wrong ;)
Krystian
 
Posts: 26
Joined: Thu Jul 21, 2011 3:11 pm
Location: Poland

Re: dgType.h: dgSort asserts

Postby Julio Jerez » Thu Oct 13, 2011 2:42 pm

I made and optimization, but and I tested,
It is very, very importnat that j is larger that zero. it is is it will crash very badlly.

I was sure I test that, but maybe I made a mistake.
can you tell me what the value of element is?
Julio Jerez
Moderator
Moderator
 
Posts: 12426
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: dgType.h: dgSort asserts

Postby Krystian » Thu Oct 13, 2011 2:52 pm

Hope this help. If not, you can connect to my PC through for example teamviewer ;)
Attachments
dgSort.jpg
dgSort.jpg (229.16 KiB) Viewed 5605 times
Warning: I'm learning C/CPP/ARM-NEON/Newton/AndroidNDK/OpenGL ES, so whatever I post here, keep in mind that I can be totally wrong ;)
Krystian
 
Posts: 26
Joined: Thu Jul 21, 2011 3:11 pm
Location: Poland

Re: dgType.h: dgSort asserts

Postby Julio Jerez » Thu Oct 13, 2011 3:05 pm

Ok I see J is zero, that is very bad.
Can you print the values of arrays by doing this in the debugger?
Array, 19
Julio Jerez
Moderator
Moderator
 
Posts: 12426
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: dgType.h: dgSort asserts

Postby Krystian » Thu Oct 13, 2011 3:13 pm

Code: Select all
-      array[0]   {m_bodyCount=2 m_bodyStart=38 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   38   int
      m_jointCount   1   int
      m_jointStart   19   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[1]   {m_bodyCount=2 m_bodyStart=38 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   38   int
      m_jointCount   1   int
      m_jointStart   19   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[2]   {m_bodyCount=2 m_bodyStart=36 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   36   int
      m_jointCount   1   int
      m_jointStart   18   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[3]   {m_bodyCount=2 m_bodyStart=34 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   34   int
      m_jointCount   1   int
      m_jointStart   17   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[4]   {m_bodyCount=2 m_bodyStart=32 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   32   int
      m_jointCount   1   int
      m_jointStart   16   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[5]   {m_bodyCount=2 m_bodyStart=30 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   30   int
      m_jointCount   1   int
      m_jointStart   15   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[6]   {m_bodyCount=2 m_bodyStart=28 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   28   int
      m_jointCount   1   int
      m_jointStart   14   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[7]   {m_bodyCount=2 m_bodyStart=26 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   26   int
      m_jointCount   1   int
      m_jointStart   13   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[8]   {m_bodyCount=2 m_bodyStart=24 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   24   int
      m_jointCount   1   int
      m_jointStart   12   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[9]   {m_bodyCount=2 m_bodyStart=20 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   20   int
      m_jointCount   1   int
      m_jointStart   11   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[10]   {m_bodyCount=2 m_bodyStart=18 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   18   int
      m_jointCount   1   int
      m_jointStart   10   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[11]   {m_bodyCount=2 m_bodyStart=16 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   16   int
      m_jointCount   1   int
      m_jointStart   9   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[12]   {m_bodyCount=2 m_bodyStart=14 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   14   int
      m_jointCount   1   int
      m_jointStart   8   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[13]   {m_bodyCount=2 m_bodyStart=12 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   12   int
      m_jointCount   1   int
      m_jointStart   7   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[14]   {m_bodyCount=2 m_bodyStart=10 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   10   int
      m_jointCount   1   int
      m_jointStart   6   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[15]   {m_bodyCount=2 m_bodyStart=8 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   8   int
      m_jointCount   1   int
      m_jointStart   5   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[16]   {m_bodyCount=2 m_bodyStart=6 m_jointCount=1 ...}   dgIsland
      m_bodyCount   2   int
      m_bodyStart   6   int
      m_jointCount   1   int
      m_jointStart   4   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[17]   {m_bodyCount=3 m_bodyStart=3 m_jointCount=2 ...}   dgIsland
      m_bodyCount   3   int
      m_bodyStart   3   int
      m_jointCount   2   int
      m_jointStart   2   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[18]   {m_bodyCount=3 m_bodyStart=0 m_jointCount=2 ...}   dgIsland
      m_bodyCount   3   int
      m_bodyStart   0   int
      m_jointCount   2   int
      m_jointStart   0   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[19]   {m_bodyCount=3 m_bodyStart=0 m_jointCount=2 ...}   dgIsland
      m_bodyCount   3   int
      m_bodyStart   0   int
      m_jointCount   2   int
      m_jointStart   0   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int


Edit: added missing arrray[15]
array[19] is probably out of bound?
Warning: I'm learning C/CPP/ARM-NEON/Newton/AndroidNDK/OpenGL ES, so whatever I post here, keep in mind that I can be totally wrong ;)
Krystian
 
Posts: 26
Joined: Thu Jul 21, 2011 3:11 pm
Location: Poland

Re: dgType.h: dgSort asserts

Postby Julio Jerez » Thu Oct 13, 2011 3:26 pm

I do no unstastnad what could be wrong, the assert should not happens.

what that loop is doing is sorting the array by using the m_jointCount as the key.
the optimization is that if you make sure that the first element in the array is the smallest or equal to teh smallest; then it does not have to check for index j to to reach zero.
since it can no possible happens. in the array m_joint all joint count seem to be 1,

the only way if will fail is if in array zero, m_joint count is equal 2 before it enter the loop.
can you track teh array before it enetr the loop and see if thsi is correct?
Julio Jerez
Moderator
Moderator
 
Posts: 12426
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: dgType.h: dgSort asserts

Postby Krystian » Thu Oct 13, 2011 3:35 pm

Read PM if possible. That assert happened after about 500 frames, and I'm not sure if I'll be able to easily to reproduce it (currently it crashes more like once per 10 runs).
Edit: I could try to set conditional breakpoint, but you need to tell me what exactly condition should be and where
Edit2: my simple animation depends where mouse pointer is in window - I'm 'simulating' accelerometer, which I'm using as a gravity vector, so simulation is not determinant/repeatable.
Edit3: I'll try to make it repeatable and debug that assert more
Warning: I'm learning C/CPP/ARM-NEON/Newton/AndroidNDK/OpenGL ES, so whatever I post here, keep in mind that I can be totally wrong ;)
Krystian
 
Posts: 26
Joined: Thu Jul 21, 2011 3:11 pm
Location: Poland

Re: dgType.h: dgSort asserts

Postby Julio Jerez » Thu Oct 13, 2011 3:50 pm

can you add a hack, to teh caller funtion, then step and see whe the value in m_jointcout are before they are sorted.


Basically what the function is, is a cache friendly sorting function.
It use quick sort to partially sort the array in stride.
After the quick sort face it place the array in strides of 8 or less number of elements that satisfy the condition that all element in one stride are equal or smaller that the element in the next stride.
Then the second part uses a insertion sort to sort an array that is already almost pre-sorted.
Under this Conditions insertion sort is run on average is only one pass.
What the optimization does is that checks for the first stride and place the smaller element at location zero. Since no other element can be large that that, the inner loop does no need to check.
I have tested that against Radix sort before the optimization, and in all case it is up 5 times faster for smaller array and about 3 times more than that for array of a 65000 elements.
The reason the function is faster is that radix is does a similar number of passes over the array as Radix sort does for array under 65000 elements but function always access the array sequentially so It is very cache friendly were Radix sort extremely hostile to cache locality
Julio Jerez
Moderator
Moderator
 
Posts: 12426
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: dgType.h: dgSort asserts

Postby Krystian » Thu Oct 13, 2011 4:09 pm

for (dgInt32 i = 1; i < elements; i ++) { // elements==9

dump of array just before any iteration of above loop:
Code: Select all
-      array[0]   {m_bodyCount=11 m_bodyStart=0 m_jointCount=10 ...}   dgIsland
      m_bodyCount   11   int
      m_bodyStart   0   int
      m_jointCount   10   int
      m_jointStart   0   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[1]   {m_bodyCount=11 m_bodyStart=11 m_jointCount=10 ...}   dgIsland
      m_bodyCount   11   int
      m_bodyStart   11   int
      m_jointCount   10   int
      m_jointStart   10   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[2]   {m_bodyCount=11 m_bodyStart=22 m_jointCount=10 ...}   dgIsland
      m_bodyCount   11   int
      m_bodyStart   22   int
      m_jointCount   10   int
      m_jointStart   20   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[3]   {m_bodyCount=11 m_bodyStart=33 m_jointCount=10 ...}   dgIsland
      m_bodyCount   11   int
      m_bodyStart   33   int
      m_jointCount   10   int
      m_jointStart   30   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[4]   {m_bodyCount=11 m_bodyStart=44 m_jointCount=10 ...}   dgIsland
      m_bodyCount   11   int
      m_bodyStart   44   int
      m_jointCount   10   int
      m_jointStart   40   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[5]   {m_bodyCount=21 m_bodyStart=55 m_jointCount=21 ...}   dgIsland
      m_bodyCount   21   int
      m_bodyStart   55   int
      m_jointCount   21   int
      m_jointStart   50   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[6]   {m_bodyCount=11 m_bodyStart=76 m_jointCount=10 ...}   dgIsland
      m_bodyCount   11   int
      m_bodyStart   76   int
      m_jointCount   10   int
      m_jointStart   71   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[7]   {m_bodyCount=11 m_bodyStart=87 m_jointCount=10 ...}   dgIsland
      m_bodyCount   11   int
      m_bodyStart   87   int
      m_jointCount   10   int
      m_jointStart   81   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
-      array[8]   {m_bodyCount=11 m_bodyStart=98 m_jointCount=9 ...}   dgIsland
      m_bodyCount   11   int
      m_bodyStart   98   int
      m_jointCount   9   int
      m_jointStart   91   int
      m_hasUnilateralJoints   0   int
      m_isContinueCollision   0   int
Warning: I'm learning C/CPP/ARM-NEON/Newton/AndroidNDK/OpenGL ES, so whatever I post here, keep in mind that I can be totally wrong ;)
Krystian
 
Posts: 26
Joined: Thu Jul 21, 2011 3:11 pm
Location: Poland

Re: dgType.h: dgSort asserts

Postby Julio Jerez » Thu Oct 13, 2011 4:51 pm

So the elemen at stride may be smalled bu that elemen is not tested;
Ok let use search fo the smallest in the first two strides, that will guarantee to find the smallest
on that function int this line
Code: Select all
   if (elements < stride) {
      stride = elements;
   }


do this
Code: Select all
       stride = stride * 2;
   if (elements < stride) {
      stride = elements;
   }


see if the problem goes away;


edit: actually I fixed sync again, and it should work now.
Thanks for the test.
Julio Jerez
Moderator
Moderator
 
Posts: 12426
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: dgType.h: dgSort asserts

Postby Krystian » Thu Oct 13, 2011 5:15 pm

With stride = stride * 2; (svn 933) no crash. Thanks!

But how can that change 'fix' it? This my last test cases I understand (somehow), array[8].m_jointCount=9 was a 'problem'?, but what if similar case will happend when smaller element will be somewhere in array[] > 16, "that will guarantee to find the smallest" - I haven't 'read/understood' whole dgSort(), so it might be obvious ;)
Or I get it totally wrong? (I'll need re-read you posts tomorrow, and learn more about new sorting algos)

Edit: btw I hope that m_jointCount is not any count of 'newton joints', as my test application doesn't use joints at all ;)
Warning: I'm learning C/CPP/ARM-NEON/Newton/AndroidNDK/OpenGL ES, so whatever I post here, keep in mind that I can be totally wrong ;)
Krystian
 
Posts: 26
Joined: Thu Jul 21, 2011 3:11 pm
Location: Poland

Re: dgType.h: dgSort asserts

Postby Julio Jerez » Thu Oct 13, 2011 6:30 pm

Like I said the first part divide the array M into M/N smaller sub arrays in which
All element of subarrayN[I] are equal or smaller that all elements of array[I + 1 ]
However the size of N is not known, because they is not equal size, all we know is that each sub area contains N + 1 or fewer elements.
So to find the smallest element of the entire array of M element all we need to do is find the smallest in the first N + 1 elements, as a measure of precaution I scan 2 * N first elements.
BTW this is not a new sorting algorithm this is actually a very old one, but a very good one. It is a combination fo quick sort with inserion sort.
They both are mediocre when uses indiviually, but when combined together not other algorithm can come even close to the performance they yeild on totally random data.

As for the second question, yes those are joints, In Newton contact are joints, Newton is no like the other physics engine when contacts are different empties and they all have three on methods, in newton all contacts and all joint are the same entities, they just have different way of settin the Degress of freedom.
Julio Jerez
Moderator
Moderator
 
Posts: 12426
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: dgType.h: dgSort asserts

Postby Krystian » Fri Oct 14, 2011 6:05 pm

Thanks for response and clarifications.

I've played a bit with dgSort and ...
1. Assert in code:
Code: Select all
for (; compare (&array[j - 1], &tmp, context) > 0; j --) {
_ASSERTE (j > 0); // if this assserts (compare()>0), then there was a read from array[-1], but when compare()<=0, then no assertion, but still read from array[-1]
array[j] = array[j - 1];
}
...

will not check condition (j>0) if fn. compare() returns <=0, so this assert-check might not be 'reliable'. When j==0, then compare() reads from array[j-1] (so read from 'garbage' or you can get here access violation).

Example:
Comment out //stride = stride * 2; in dgSort. You won't get assert failure, but there will be read from array[-1] so from array2[0].m_jointCount==-1234, in the end everything might look ok, because that m_jointCount = 9 will be finally stored in array[0], and validation at end of dgSort will also succeed:
Code: Select all
   dgIsland array2[10]={0};

   // 'marker' for test reading array[j-1] when j=0
   array2[0].m_jointCount = -1234; // value must be < 9

   array2[1].m_jointCount = 10;
   array2[2].m_jointCount = 10;
   array2[3].m_jointCount = 10;
   array2[4].m_jointCount = 10;
   array2[5].m_jointCount = 10;
   array2[6].m_jointCount = 21;
   array2[7].m_jointCount = 10;
   array2[8].m_jointCount = 10;
   array2[9].m_jointCount = 9;

   dgSort (&array2[1], 9, CompareIslands);

I know that it's with commented out "stride = stride * 2", and might not be important right now. I think it could be a case why it worked fine when testing before (but still could AV or asserts).

2. You have wrote:
"After the quick sort face it place the array in strides of 8 or less number of elements that satisfy the condition that all element in one stride are equal or smaller that the element in the next stride."
but as I understand in this case after first sort in "while (stackIndex)..." (which anyway doesn't change anything in above array?!)
it's not true for above example data. I don't know how strides will look in 9-elements array (like first stride [0-7], second stride [8]), but in all cases I see that m_jointCount=9 will be in second stride, so "all element in one stride are equal or smaller that the element in the next stride" is false for above data?

Second loop "for (dgInt32 i = 1; i < stride; i ++) {" is for "What the optimization does is that checks for the first stride and place the smaller element at location zero.", but in this case, first stride was not enough, so there was a "stride = stride * 2" fix.

If possible, could you show me how strides would look like in an array of 9-elements [10,10,10,10,10,21,10,10,9] (this is an array before and after first sort)? This would make my doubts go away :) For now I think that there might be a bug in first sorting algo, or I've misunderstood some things here (which is more likely possible).
Warning: I'm learning C/CPP/ARM-NEON/Newton/AndroidNDK/OpenGL ES, so whatever I post here, keep in mind that I can be totally wrong ;)
Krystian
 
Posts: 26
Joined: Thu Jul 21, 2011 3:11 pm
Location: Poland

Re: dgType.h: dgSort asserts

Postby Krystian » Fri Oct 14, 2011 6:15 pm

Hmm, changing:
Code: Select all
if ((hi - lo) > stride)

to:
Code: Select all
if ((hi - lo) >= stride)


seems to fix it?

Edit: also with that change, previous fix "stride = stride * 2;" isn't needed, and everything makes sense :)
Warning: I'm learning C/CPP/ARM-NEON/Newton/AndroidNDK/OpenGL ES, so whatever I post here, keep in mind that I can be totally wrong ;)
Krystian
 
Posts: 26
Joined: Thu Jul 21, 2011 3:11 pm
Location: Poland

Re: dgType.h: dgSort asserts

Postby Julio Jerez » Fri Oct 14, 2011 6:32 pm

yes that will fix it too, but it will make the first part to do a lot more work on randome accesed data.
to you give you an idea, say you have an array of 64 entries, the stride is 8
if you use (hi - lo) < 8
then on average you will get 64 / 8 strides of 8 elemenet each.
each stride the element will be almost sorted. the top part do less work and the lower part do more work,
but because the sub array are almost sorted, the second part get away with small amound work because
The secrect is that insertion sort is linear on arrays that are almost sorted.

if you use (hi - lo) <= 8 now the top part devide the 64 elemnet into 16 stride of 4 elements each, now the top part will be dowing a lot of work on very short arrays.
the secund part will be doing the same amont of time because the arrays are still almost sorted an dthe are small, so not saving at all in teh secudn part.
it just increases the work of the first part to make smaller arrays for not gain at all.

It is bether to let the stride be relativally large (8 or 16 elements) and scan the first two strides in search of the smallest element afte that.

basically using (hi - lo) <= 8 sudivode the array into sub arrays that are stride / 2
so the secund part scan for the first stride, but in really the first stride is tow stride of have size each.
I do no thonk that is not a good optimization. But it does work.
Julio Jerez
Moderator
Moderator
 
Posts: 12426
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles


Return to General Discussion

Who is online

Users browsing this forum: No registered users and 1 guest