A place to discuss everything related to Newton Dynamics.
Moderators: Sascha Willems, walaber
by middletail » Sat Jun 13, 2009 10:04 pm
Hello,
I'm trying to use Newton 2.01 in my app and I found there was a lack of determinism. Then I looked around this forum and I found the following posts:
http://www.newtondynamics.com/forum/viewtopic.php?f=9&t=5146http://www.newtondynamics.com/forum/viewtopic.php?f=9&t=5054 In these posts,NewtonInvalidateCache() is mentioned which seems to be the solution for my problem. I actually tried using it in my app but it seemed that it stopped Newton from working. Here is what I'm trying to do: My app is an animation tool and I have a keyframe animation. Then I'd like to switch it to physics animation at some point but the object stayed at the end point of keyframe animation when I used NewtonInvalidateCache(), which was supposed to keep moving by Newton. It did so when I didn't use NewtonInvalidateCache() but I had the problem with determinism.
Then my question is what does NewtonInvalidateCache() do? And How and when do you use it?
I'm using Newton 2.01 with Ogre and OgreNewt on Mac OSX 10.5.7.
Could anyone tell me? Thanks in advance.
middletail
-
middletail
-
- Posts: 4
- Joined: Sat Jun 13, 2009 9:15 pm
by Julio Jerez » Sat Jun 13, 2009 11:45 pm
Oh cool you are using the Mac SDK, can you please download 2.01 again I fix a Bug and I just updated all the platforms.
Has you have any problem with geting it to run on you Mac?
NewtonInvalidateCache()
Clear all of the internal contact joints and sort the body list and the joint list.
This function is only useful for people doing multiplayer app that work in lock step.
To get determinism all you need to do is call that functions after you create your scene, and before you start the main Loop.
You also need to play at Fix time step, and you must not use multithreading.
As a side Note, I believe it is time that people start to forget about determinist in Physics,.
With the New Hardware with many core keep the same simulation after is have being unitized is impossible.
The new CPU come with a least two cores and in order to make the physics competitive we nee to use multithreads, but the end application do not have control of what is run on what core, it is the OS that control that the treads assignment. When you go to GPU it is even harder
Even using something as simple as simd code can throw determinism off, and simpler yet, even using x87 code you never know when and external library is going to change the rounding mode of the CPU behind your back.
The moral is that the advances in hardware are making physics determinist more and more difficult each time.
I do not have a solution for this; in fact about five year ago a mention that to one of eth Torque people and they consider me a complete moron for even suggesting that what is wrong is that the multiplayer code should run in sinc.
I never like that idea of lock step code, because in my opinion the weakest system in the network dictate the rule.
-
Julio Jerez
- Moderator

-
- Posts: 12426
- Joined: Sun Sep 14, 2003 2:18 pm
- Location: Los Angeles
-
by JernejL » Sun Jun 14, 2009 7:35 am
Julio: determinism is a seriously important issue and might actually be the future of low-latency online gaming, i know a whole lot of games using "input-tick-sync" and deterministic simulation to achieve game synchronizing and it works remarkably well, for example old-style ismetric C&C games use it, they could never sync 500 tanks over network using conventional methods, i believe using a centralized server and extending input over several frames (30 frames per second * 100ms ping in theory goes thru in 1 second, but thats unrealistic, you need to take into account dropped packets and sudden lag-bursts, so using identical input for several frames this can easily end up as say every 4 frames - (30fps / 4) = 7.5 * 100ms ping = 750ms ), basically with this and deterministic game engine you get PERFECT game world sync with only slight and barely noticeable input lag and no way for cheaters to teleport around and try to cheat the simulation. In future where games might become very complex, syncing everything with conventional methods will be just simply impossible if the internet won't be able to transport large udp packets with very low latency (this is a issue already).
So while ordinary games like FPS will still want to squeeze more physics out of all cores and use conventional network sync, mmo games will likely take into this direction with deterministic simulations, and i think it's really great that newton currently supports determinism.
-

JernejL
-
- Posts: 1587
- Joined: Mon Dec 06, 2004 2:00 pm
- Location: Slovenia
-
by agi_shi » Mon Jun 15, 2009 11:25 pm
Delfi wrote:for example old-style ismetric C&C games use it, they could never sync 500 tanks over network using conventional methods,
The problem is that we're talking about real-time 3D physics, not 2D logic-based movement. The latter is much easier to make deterministic.
Some of the puzzles in my project require a deterministic simulation, but those puzzles turn off multithreading, so everything is fine. The other puzzles use multithreading for the speed.
-
agi_shi
-
- Posts: 263
- Joined: Fri Aug 17, 2007 6:54 pm
by JernejL » Tue Jun 16, 2009 3:29 am
The same approach works identically for 2D and 3D, as long as it is supported in physics engine itself.
-

JernejL
-
- Posts: 1587
- Joined: Mon Dec 06, 2004 2:00 pm
- Location: Slovenia
-
by Julio Jerez » Tue Jun 16, 2009 8:30 am
The problem is not that lock step networking does not work, obviously does, it is why it is used by many games.
But the short comings of lock step multiplayer far outweigh the benefic.
I give you a short list of the problems of lockstep multiplayer.
Your game is controlled by the weakest computer in the system.
Force the game logic to come up with a horrendous replay system that is some case it needs to kept track of up to hundreds of frames of game play.
Anything that can be is simulated faster than what the network allow for transmission, needs to be interpolated. As a result of that your game runs with a lag, that in my opinion, is unacceptable.
The game code is horrendously complex since basically every object or every entity in the game, have to be network aware, you end up with two games build in one.
It is extremely hard to debug since any programmer not familiar with the game engine, may make a mistake of not placing some essential variable saving and loading newtwork api of the engine.
Lock step synchronization was invented by ID about 15 years ago when it was OK to simulate game logic at 10 frames per seconds like Quake, and Doom.
It was great for that time, but the hardware have evolved a lot and we have system with more that one CPU, GPU, or even different vector units in the same prossesor.
Games that wish to keep the Lock step technology disregard all these advances in favor of an x87 architecture that is 35 years old.
To make an analogy Lock Step networking it teh equevalent of transmitting Morse code when we have cell phones and video conference.
Sure the telegraph was an awesome technology for its time but the enology made it obsolete.
It is my opinion that somebody need to figure out a way around Lock Step using a client / server method.
But I know I am in the minority here.
-
Julio Jerez
- Moderator

-
- Posts: 12426
- Joined: Sun Sep 14, 2003 2:18 pm
- Location: Los Angeles
-
by middletail » Tue Jun 16, 2009 8:54 am
Julio,
Thank you for your explanation. I've understood what the function does and how to get determinism. I downloaded the latest Newton a few days ago and tried to use the function but again it seemed it had stopped Newton from working. I actually had a second thought after I read your post and I would think of other ways to do without determinism but it would be good for me to know the right way to use the function.
I used NewtonInvalidateCache() after I reach the end of keyframe animation in my app and before I switch it to physics animation where I start increasing time in Newton world. Then once I do NewtonInvalidateCache(), though I increse the time, I get same position and orientation everytime I call NewtonBodyGetMatrix() which I actually call from getPositionOrientation() in OgreNewt. Is this the wrong timing to use it? Or am I doing something wrong?
I haven't had any problem except this so far and seems working fine but if something happens I'll let you know.
Thanks again for your help.
-
middletail
-
- Posts: 4
- Joined: Sat Jun 13, 2009 9:15 pm
by Julio Jerez » Tue Jun 16, 2009 9:16 am
I will check if teher is a bug in NewtonInvalidateCache()
you say that if you call that funtion the engine freezes?
-
Julio Jerez
- Moderator

-
- Posts: 12426
- Joined: Sun Sep 14, 2003 2:18 pm
- Location: Los Angeles
-
by middletail » Fri Jun 19, 2009 2:25 am
Julio,
I tried using NewtonInvalidateCache() in the Friction Coefficients Demo of Newton 2.02 and it seemed there was nothing wrong with it. I inserted the function at the last of Friction() after all the bodies were created. I didn't see the same problem as in my app which looked Newton got frozen. So I guess I just used it in the wrong way. I'll look into my code again. Anyhow, thank you for your help.
-
middletail
-
- Posts: 4
- Joined: Sat Jun 13, 2009 9:15 pm
by middletail » Mon Jun 22, 2009 6:56 am
Julio,
This might sound sticky but I have a question again about NewtonInvalidateCache(). Though I still have the problem with the function which seems it stops Newton working, I found I had the problem only when I used it with a tree collision. Is it possible you cannot use the function when you have that kind of collision in the Newton world?
By the way, thanks for updating the demo for Mac.
-
middletail
-
- Posts: 4
- Joined: Sat Jun 13, 2009 9:15 pm
by Julio Jerez » Mon Jun 22, 2009 8:40 am
that function does not deal with collision shapes, only with bodies and contacts joints.
I just pasted this and the end of the render funtion in the demos
- Code: Select all
static int xxx;
xxx ++;
if ((xxx % 500) == 0)
NewtonInvalidateCache (demo.m_world);
and at least with the demos, it seems to work fine.
-
Julio Jerez
- Moderator

-
- Posts: 12426
- Joined: Sun Sep 14, 2003 2:18 pm
- Location: Los Angeles
-
by JernejL » Mon Jun 22, 2009 8:41 am
all you need to do is call that functions after you create your scene, and before you start the main Loop.
Which i think means that you don't use it while creating collision trees or inside callbacks, you just call this once after you built the game scene, just before you start performing the game loop.
-

JernejL
-
- Posts: 1587
- Joined: Mon Dec 06, 2004 2:00 pm
- Location: Slovenia
-
by Julio Jerez » Mon Jun 22, 2009 9:35 am
Delfi wrote:all you need to do is call that functions after you create your scene, and before you start the main Loop.
Which i think means that you don't use it while creating collision trees or inside callbacks, you just call this once after you built the game scene, just before you start performing the game loop.
Also you call it each time a sycronization will mbe made.
For example when yuo get a message that every tiong is in Sync you call that funtion to clean up all the memory history and organize the data.
do not call it every frame as that funtion will severelly affect the performance of the engine.
-
Julio Jerez
- Moderator

-
- Posts: 12426
- Joined: Sun Sep 14, 2003 2:18 pm
- Location: Los Angeles
-
by Marc » Wed Jul 15, 2009 7:04 am
Julio Jerez wrote:As a side Note, I believe it is time that people start to forget about determinist in Physics,.
With the New Hardware with many core keep the same simulation after is have being unitized is impossible.
The new CPU come with a least two cores and in order to make the physics competitive we nee to use multithreads, but the end application do not have control of what is run on what core, it is the OS that control that the treads assignment.
For all these different hardware and rounding errors that occur I agree. But for multithreading: Isn't it possible to divide the scene in islands of objects that interact in the current timestep and calculate the resulting islands in parallel (no more than 1 thread per island) while still maintining deterministic behaviour?
-

Marc
-
- Posts: 281
- Joined: Sun Mar 14, 2004 4:07 pm
- Location: Germany
-
by Julio Jerez » Wed Jul 15, 2009 9:05 am
Newton does solve inland in parallel, it has to, it is the only way to have some gain with very large set of objects.
As for determinism, it is totally impossible with the way hardware manufactures take control of the hardware.
If you look at stuff like Nvidia code and OpenCL, thet do not let you decide how the hardware will be committed, plus the number of core is complete undermined.
if you look a multicourse then the OS, control how task are assigned to threads, it is possible the one core is busy with one OS task while other is working, or even the one thread is interrupted while other keeps working and solving more task that can change the outcome of some intermediate values.
With such gridlock control of the hardware it is impossible for us to control determinism.
My Guess is the someday companies like Nvida and Intel will be able to do it, because they write their stuff at a drive level, we cannot do that.
With these companies, Nvidia being the most dishonest of all, you cannot count on these things to be solved any time soon, I believe that now they discourage people to abandonee determinism but that is only until the figure out the solution and they the change position recommending the opposite, Nvidia is not a trustworthy Company.
And the OS do not give the kind of control to the user unless you too write at the driver level, which I will never do.
-
Julio Jerez
- Moderator

-
- Posts: 12426
- Joined: Sun Sep 14, 2003 2:18 pm
- Location: Los Angeles
-
Return to General Discussion
Who is online
Users browsing this forum: No registered users and 1 guest