Multi-World and/or Multi-Thread

[Gianluca]

Hello,
I'm having problems in running simulataneous NewtonWorld instances :-S

I firstly tried to run only one NewtonWorld using multi-thread capability... and all works fine.

But If I try to run more than one NewtonWorld instances on various thread (with SetThreadCount(1) ), the application will crash in some Newton function... some time CreateBody or CreateCollision or dgSortArray::Sort, dgBodyMasterList::RemoveConstraint().

I can't understand why multiple instances of NewtonWorld will crash ... some hints ??

Basically, I have a pool of threads, and each of them has a private variable "NewtonWorld".

Thanks,
Gianluca.

[Julio Jerez]

Hello,
But If I try to run more than one NewtonWorld instances on various thread (with SetThreadCount(1) ), the application will crash in some Newton function... some time CreateBody or CreateCollision or dgSortArray::Sort, dgBodyMasterList::RemoveConstraint().

That very well be possible.
this is because In Netwon teh Memory manager is thread safe, but allocationg from chuncks is not thread safe.
I never tested running several world in a single instance, I tested running mutiple copies of the engine in separate window but that runs seprate window intances that do no see each other.

The solution will be making the chunk allocation thread safe, but that will be costly if done at the Manage level, or very elavorated.
Or the solution I would like to try but that scape my abilities as programer, islto make the Memory Manager local to the Netwon world.
Bu I do not know how that can be done.
I beleieve this is also the same bug Marc may be having if he is running world in seprate threades

In any case, that seem a very big bug that need fixing, do you have a test that I can try for fixing?
I will try the first solution making allocation from chunks thread safe.

Obviustly It should work even if you run separated version each one running separate chield threads.

[Gianluca]

Hello,
ok, now I understand why it will crash
I don't have a little application that replicate the bug... but I'll do it as soon as possible.
Anyway, if the problem is the allocation of memory, do you think that it's possible to avoid these problem using the memory callbacks in NewtonWorldCreate ???

[Julio Jerez]

No using the callback will not solve it, bacause the allocation call back only allocate new pool wich are thread safe, but small allocation fix size allocated are indices into that pool.

you can test thes by allocation a group of bodies, you will see that not all badies get a call back, they are all created from one pool until the pool does not have
room for another chunk then a new Chunk is allocated, it is that chunk that is requested from the allocator callback.
Basicall we get racing condition in intenal Chunk allocations.

[Gianluca]

I'm not sure that I get it where is the problem.
Because
-1) when I run one NewtonWorld using more than one thread (i.e. SetThreadCounts(4)) I don't have problems..
but
-2) when I run more than one NewtonWorld concurrently even if I set ThreadCounts(1) ... the application will crash...
so,
why the chunck allocation doesn't create problem in condition 1) ... but It cause crashes in case 2) ??
Is the pool a global variable shared amongs all NewtonWorld created ??
If yes, one solution it could be to create differents pools for different worlds... right ?!?!

Thanks,
Gianluca.

[Julio Jerez]

-1) when I run one NewtonWorld using more than one thread (i.e. SetThreadCounts(4)) I don't have problems..
but
-2) when I run more than one NewtonWorld concurrently even if I set ThreadCounts(1) ... the application will crash...
so,

that is correct, in the first case the engine is designed with per thread buffers so that critical section are minimazed,
in the case teh is need to allocate memory it uses a critical section.
when it make a callback it passes the thread index hint teh applciation tah it can use local buffer to eliminate critical sections.

In the secund case there is not per engine memory manager so thread may be allocation data at the same time,
for example you can have a joint allocation space for contact in threa zero, in one Newtonworld, and at the same time another joint allocation space for contact in tread 0 in a different newtonworld.
both the world have their own buffers but at the moment of allocation they will clash.
that is a exreme case, but clashes can happens anywhere an allocation happens.

If you can produce a test I thonk I can fix it very eassy.

[Julio Jerez]

Is the pool a global variable shared amongs all NewtonWorld created ??
If yes, one solution it could be to create differents pools for different worlds... right ?!?!

yes it is a class that is global, but it is not so eassy to make it local to the workld as the workd is a sub class and the memory manager is the memory manager for every thing I do.

Making it local to the world so teh every thing is compeletely seprate would be the ideal solution,
but this is not as eassy as it sound.
It have being a very long time since I look into that, I will check again maybe there is a solution at the memory manager level.

[Gianluca]

Hello,
I created a very simple program that create various NewtonWorld in different threads.
It's very dummy, but enough for the bug to appear , essantialy it create, simulate and destroy some boxes forever.
The code is just one file:

Code: Select all


#include <pthread.h>
#include "Newton.h"
#include "dVector.h"
#include <iostream>
using namespace std;

//--- number of threads per world
#define NEWTON_THREADS   1
//--- number of worlds to create
#define NEWTON_WORLDS   4
#define NUM_STEPS      500

void applyForceAndTorqueHandler(const NewtonBody* body, dFloat timestep, int) {
   NewtonWorld* world = NewtonBodyGetWorld( body );
   NewtonWorldCriticalSectionLock( world );
   // apply the gravity force
   dVector force( 0.0f, 0.0f, -9.8 );
   NewtonBodySetForce( body, &force[0] );
   NewtonWorldCriticalSectionUnlock( world );
}

void* newtonSimul( void* ) {
   NewtonWorld* world = NewtonCreate( NULL, NULL );
   dVector minP(-20,-20,-20), maxP(20,20,20);
   NewtonSetWorldSize( world, &minP[0], &maxP[0] );
   NewtonSetMinimumFrameRate( world, 100 );
   NewtonSetSolverModel( world, 0 );
   NewtonSetFrictionModel( world, 0 );
   NewtonSetThreadsCount( world, NEWTON_THREADS );
   cout << "World Created" << endl;;
   //--- ForEver
   while( 1 ) {
      //--- creating objects
      NewtonBody* bodies[30];
      NewtonCollision* collisions[30];
      for( int i=0; i<30; i++ ) {
         collisions[i] = NewtonCreateBox( world, 1, 2, 1, 1, 0 );
         bodies[i] = NewtonCreateBody( world, collisions[i] );
         NewtonBodySetAutoSleep( bodies[i], 0 );
         dVector inertia;
         dVector centre( 0, 0, 0 );
         NewtonConvexCollisionCalculateInertialMatrix( collisions[i], &inertia[0], &centre[0] );
         NewtonBodySetMassMatrix( bodies[i], 1.0, inertia[0], inertia[1], inertia[2] );
         NewtonBodySetLinearDamping( bodies[i], 0.0 );
         dVector zero(0,0,0);
         NewtonBodySetAngularDamping( bodies[i], &zero[0] );
         NewtonBodySetAutoSleep( bodies[i], 0 );
         NewtonBodySetForceAndTorqueCallback( bodies[i], applyForceAndTorqueHandler );
      }
      cout << "Objects Created" << endl;;
      //--- simulating a bit
      cout << "Simulating" << endl;
      for( int i=0; i<500; i++ ) {
         NewtonUpdate( world, 0.01 );
      }
      //--- Destroying objects
      for( int i=0; i<30; i++ ) {
         NewtonDestroyBody( world, bodies[i] );
         NewtonReleaseCollision( world, collisions[i] );
      }
      cout << "Objects Destroyed" << endl;
   }
   return NULL;
}

int main( int argc, char* argv[] ) {
   pthread_t threads[NEWTON_WORLDS];
   for( int i=0; i<NEWTON_WORLDS; i++ ) {
      pthread_create( threads+i, NULL, newtonSimul, NULL );
   }
   
   for( int i=0; i<NEWTON_WORLDS; i++ ) {
      pthread_join( threads[i], NULL );
   }
}


[Julio Jerez]

nice, I set up teh project and I have teh firs crash here

Code: Select all

dgBroadPhaseCell* dgBroadPhaseLayer::FindCreate (dgInt32 x, dgInt32 z)
{
   dgInt32 key;
   dgTreeNode * node;

   key = GetKey (x, z);
   node = dgTree<dgBroadPhaseCell, dgUnsigned32>::Find (key);
   if (!node) {
      dgBroadPhaseCell cell;
      cell.m_count = 0;
      node = Insert(cell, key);
      node->GetInfo().Init (m_layerIndex);
   }

    return &node->GetInfo();
}

If I set #define NEWTON_WORLDS 1
it does no crash so it is definitly teh memory allocation reponsable fo rteh crash.

now I have to work in the solution, stand by.

[Julio Jerez]

It is a good thing I change my very, very old version of source visual sorce save to CVS Torstoy, now revision are automatic.
I beleieve that teh best solution for this is to soleve at teh very mamory m,aneger level
Thes are my macros I use in ever class

Code: Select all

#define DG_CLASS_ALLOCATOR_NEW   inline void *operator new (size_t size) { return dgMalloc(size); }                                         
#define DG_CLASS_ALLOCATOR_NEW_ARRAY   inline void *operator new[] (size_t size) { return dgMalloc(size); }
#define DG_CLASS_ALLOCATOR_DELETE      inline void operator delete (void *ptr) { dgFree(ptr); }
#define DG_CLASS_ALLOCATOR_DELETE_ARRAY   inline void operator delete[] (void *ptr) { dgFree(ptr); }

#define DG_CLASS_ALLOCATOR      \
   DG_CLASS_ALLOCATOR_NEW   \
   DG_CLASS_ALLOCATOR_DELETE   \
   DG_CLASS_ALLOCATOR_NEW_ARRAY   \
   DG_CLASS_ALLOCATOR_DELETE_ARRAY   

It is use like this fo rexample the vertor class

Code: Select all

template<class T>
class dgTemplateVector
{
   public:
   DG_CLASS_ALLOCATOR

   dgTemplateVector ();
   dgTemplateVector (const T *ptr);
   dgTemplateVector (T m_x, T m_y, T m_z, T m_w);
   dgTemplateVector Scale (T s) const;

   T& operator[] (dgInt32 i);
   const T& operator[] (dgInt32 i) const;

   dgTemplateVector operator+ (const dgTemplateVector &A) const;
   dgTemplateVector operator- (const dgTemplateVector &A) const;
   dgTemplateVector &operator+= (const dgTemplateVector &A);
   dgTemplateVector &operator-= (const dgTemplateVector &A);

   // return dot product
   T operator% (const dgTemplateVector &A) const;

   // return cross product
   dgTemplateVector operator* (const dgTemplateVector &B) const;

   // component wise multiplication
   dgTemplateVector CompProduct (const dgTemplateVector &A) const;
   T m_x;
   T m_y;
   T m_z;
   T m_w;
};

what I am goin to do is to change the macros to thsi style

Code: Select all

[code]#define DG_CLASS_ALLOCATOR_NEW(allocator)         inline void *operator new (size_t size, void* allocator) { return dgMalloc(size, allocator); }                                         
#define DG_CLASS_ALLOCATOR_NEW_ARRAY(allocator)    inline void *operator new[] (size_t size, void* allocator) { return dgMalloc(size, allocator); }
#define DG_CLASS_ALLOCATOR_DELETE(allocator)          inline void operator delete (void *ptr, void* allocator) { dgFree(ptr, allocator); }
#define DG_CLASS_ALLOCATOR_DELETE_ARRAY(allocator) inline void operator delete[] (void *ptr, void* allocator) { dgFree(ptr, allocator); }

#define DG_CLASS_ALLOCATOR(allocator)      \
   DG_CLASS_ALLOCATOR_NEW(allocator)      \
   DG_CLASS_ALLOCATOR_DELETE(allocator)   \
   DG_CLASS_ALLOCATOR_NEW_ARRAY(allocator)   \
   DG_CLASS_ALLOCATOR_DELETE_ARRAY(allocator)[/code]

thes will cause a whl lot of sysnatact error fixing all pve the place, bu teh idea i fthat Netwon itlset will have teh pointe to teh allocator.
and will nee to pass the allcator to each calss

for exampel say a array of veros is allocated

I will ne to change form

dgVetoc* array = new dgVector(100)

to

dgVetoc* array = allocator new dgVector(100);

and so every where.

I think this is easier than changing all the container by inserting critical sections and spinlocks the will slow down the engine by a great deal.
The word horse of the engine are the containers and the simple use new and delete, but the rely on new and free to be real fast, changing that will be catastrofic
in fact when I tarted the muttreaded solution I did it that way and I learnd the spin lock and critical sections are a really bad was for doing threading.

I will try teh memory solution that and see how is goes.

[Julio Jerez]

The good new is that it does no seems it is goin to be too difficult, I already mane teh modification to tteh funtion tah I nee to change and replace in about a docent files.
now I just need to go over then make the adjustment, an dI beilev it will be fine.
I have to leave for work know but I believ I can eget it completed tomorrow.

I was a very long time that it was bodering me that the memory manager was global, but I tolerated because I never tried multopels wordk running in parallels,
but is was abput time to do it.

now Netwon will be able to run mutiples worlds in separated threads and each one runing mutiples mutiples children threads, and all of them deterministic.
no bad fo and engine from the backyard

[Julio Jerez]

Good new and Bad news.

The good new is that to make a test I replne teh memory allocator form pool to use just main malloc and free, and the demo runs fine

Code: Select all

void* dgApi dgMalloc (size_t size)
{
   void *ptr;
   ptr = NULL;

   if (size) {
//      ptr = dgInternalMemory::g_allocaror.Malloc (dgInt32 (size));
      ptr = dgInternalMemory::g_allocaror.MallocLow (dgInt32 (size));
   }
   return ptr;
}

// general deletion allocation for all data in the library
void dgApi dgFree (void *ptr)
{
   if (ptr) {
//      dgInternalMemory::g_allocaror.Free (ptr);
      dgInternalMemory::g_allocaror.FreeLow (ptr);
   }
}

I made some chnage so tha I can see what is happing an dthe is the output

World Created 0
World Created 3
World Created 1
World Created 2
Objects Created 0
Simulating
Objects Created 3
Simulating
Objects Created 2
Simulating
Objects Created 1
Simulating
Objects Destroyed
Objects Created 0
Simulating
Objects Destroyed
Objects Created 2
Simulating
Objects Destroyed
Objects Created 3
Simulating
Objects Destroyed
Objects Created 1
Simulating

The Bad news is the because new use New and delete as the inteface for all objects,
if the memory manager is not very, very efficien the perofiome gos bow the toiler, nor only that bu teh memory fragmantion beforme attroisus.

basicall a new sodul almos resole to

Code: Select all

new (int size)
{
  void *ptr;
   size += (1<<granulatiry);
   size >> granulatiry;

   ptr = Pool[size]->Last;
   Pool[size]->Last =  Pool[size]->Last->Last;
   return  ptr;
}

plus few more instrution for maintance when a pool is full.

anyway I implemneted teh new version and I have crashes all over the plase, so I made a mistake some where.
I decide to rervese all together rathe tah serchin for teh Bug, and I will try again.
This is not as hard as I thought, I soudl be eable to have ready in a day or two.
but this is definitlly the way of solving it.

[Julio Jerez]

woohoo now I got you demo fooly working in separate threads, I still nee to make sur eteh SDK still runs since I commonet out some funtions tha now will requres teh pointer to one world as agument.
The wrold have the local allocator.
Basically my solution is thii.

I keep the Global allocator, byu thei is inly use fo memory that is no pool base, liek large contact buffer, contranstraint, vertex array,
all class take the engine class allocator form teh work as argument, so teh tehy can use it for internal pool alocation.
it work leo a cham.

I want to do this very, long time ago, becaus eit always bother me tha when using mutiple world teh pool where shared across world and the crate a lot of fragmention, now each world keeps it pool neatly packed.
Thsi sure make teh mutipwpord jsu as efficent as single worlds. Plus they are fully thread safe.

I guess one more day and this baby is ready to go.

[Julio Jerez]

Ok I finished the changes and now each world own its memory manager. you can download 2.13
I made a small change to your demo so that you can see hwo it is set up.
also there is not need to use critical section in a force and torque call back if you are only doing newton operations,
critical sections is for when aplication are using local data that can potencially cause race condition.
mutiple worlds aare like ships at sea, they do not see each others.
here is the modifiec demo.

Code: Select all

// testMultithread.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"


#define __restrict__ __restrict


#include <pthread.h>
#include "Newton.h"
#include "dVector.h"
#include <iostream>
using namespace std;

//--- number of threads per world
#define NEWTON_THREADS   3
//--- number of worlds to create
#define NEWTON_WORLDS   4
#define NUM_STEPS      500

void applyForceAndTorqueHandler(const NewtonBody* body, dFloat timestep, int)
{
   NewtonWorld* world = NewtonBodyGetWorld( body );
//   NewtonWorldCriticalSectionLock( world );
   // apply the gravity force
   dVector force( 0.0f, 0.0f, -9.8f );
   NewtonBodySetForce( body, &force[0] );
//   NewtonWorldCriticalSectionUnlock( world );
}

static int worldIndex;
void* newtonSimul( void* )
{
   int mem;
   NewtonWorld* world = NewtonCreate ();
   mem = NewtonGetMemoryUsed();

   dVector minP(-20,-20,-20), maxP(20,20,20);
   NewtonSetWorldSize( world, &minP[0], &maxP[0] );
   NewtonSetMinimumFrameRate( world, 100 );
   NewtonSetSolverModel( world, 0 );
   NewtonSetFrictionModel( world, 0 );
   NewtonSetThreadsCount( world, NEWTON_THREADS );

   int index = worldIndex ++;
   cout << "World Created " << index << endl;;

   //--- ForEver
   while( 1 ) {
//   for (int i = 0; i < 1; i ++) {
      //--- creating objects
      NewtonBody* bodies[30];
      NewtonCollision* collisions[30];
      for( int i=0; i<30; i++ ) {
         collisions[i] = NewtonCreateBox( world, 1, 2, 1, 1, 0 );
         bodies[i] = NewtonCreateBody( world, collisions[i] );
         NewtonBodySetAutoSleep( bodies[i], 0 );
         dVector inertia;
         dVector centre( 0, 0, 0 );
         NewtonConvexCollisionCalculateInertialMatrix( collisions[i], &inertia[0], &centre[0] );
         NewtonBodySetMassMatrix( bodies[i], 1.0, inertia[0], inertia[1], inertia[2] );
         NewtonBodySetLinearDamping( bodies[i], 0.0 );
         dVector zero(0,0,0);
         NewtonBodySetAngularDamping( bodies[i], &zero[0] );
         NewtonBodySetAutoSleep( bodies[i], 0 );
         NewtonBodySetForceAndTorqueCallback( bodies[i], applyForceAndTorqueHandler );
      }
      cout << "Objects Created " << index << endl;;
      //--- simulating a bit
      cout << "Simulating" << endl;
      for( int i=0; i<500; i++ ) {
         NewtonUpdate( world, 0.01f );
      }

      mem = NewtonGetMemoryUsed();
      //--- Destroying objects
      for( int i=0; i<30; i++ ) {
         NewtonDestroyBody( world, bodies[i] );
         NewtonReleaseCollision( world, collisions[i] );
      }
      cout << "Objects Destroyed" << endl;
   }

//   NewtonDestroy (world);
   return NULL;
}

int main( int argc, char* argv[] )
{
   pthread_t threads[NEWTON_WORLDS];

   NewtonSetMemorySystem (NULL, NULL);

   for( int i=0; i<NEWTON_WORLDS; i++ ) {
      pthread_create( threads+i, NULL, newtonSimul, NULL );
   }

   for( int i=0; i<NEWTON_WORLDS; i++ ) {
      pthread_join( threads[i], NULL );
   }
}

[Gianluca]

Woowww... thank you very much !!
There is people that paying lot of money for having a such kind of support ... and you gave me for free
Thanks

Now, I have few time for trying the new version... but I'll let you know as it works.

Just a question: Do you introduced a new function "SetMemorySystem", but what's the purpose ??

Thanks again,
Gianluca M.