How to remove ndModel

[JoeJ]

I had a little breakthrough yesterday, and i have just tested it again: Yes, seems i'm pretty much done with the basics. The simulation is perfect and i have precise control. It's now impossible for the ragdoll to loose balance. It's stable like a rock, and agile like Bruce Lee. \ /
Now the fun part starts, and i can start work on walking and other behaviors. I'm still sure that's the easier part, but i'll see...

Regarding contacts, the improvements did not help me. I have to stick at the custom contact joint. At least for now.
The problem with default contacts is: Becasue they are distributed over the whole contact area, some contact force is generated for all of them, and we have no precise control about how much.
Imagine you stand straight, and then you want to push your com backwards, so you would fall on your back.
To do so, you will move your center of contact pressure to the toes.
There will be also contacts under your heels, but ther will be no force there. All force is focused on the front side at the toes.
But in the simulation you can not do this. There will be force also on the heels, slowing your backwards movement down. And beside this constant error, it's unpredictable. Cached contacts change their position at random time, generating unexpected discontinuities. I've tried to compensate the error using a PD controller and such things. This works, but it's not rliable enough.

Even ML will have a hard time to compensate those unptredictable events.
Likely it can deal with it, but then the only way to do so is being very cautious, and thus moving slowly. Bruce Lee might be out of reach.

Only now that i have the custom contact joint workaround fully working, i can see the impact of this contact problem. And i tell you: It's huge. It's much bigger than you think it is, i guess. I knew about this problem for years, but i'm really surprised now myself.

Now i'll see how far i can get using the custom joint. It might be the perfect solution for games, since it also allows to cheat by making the feet virtually larger very easily, but i have not yet tried to walk over dynamic objects or such things. I expect some problems in the future.

So how would a general solution look like, still using Newtons default contacts and collision detection?
I think it might work using an additional constraint, affecting multiple contacts.
Basically i'd need to do something like: 'For all contacts of the feet, ensure that all genreated contact forces are clipped by some given user plane.'
Or in other words: 'The toe contact force needs to be x times stronger than the heel force'.
Something like that.

Ideally, the simulation would figure out this without any extra constraints or information. It should, since the joint accelerations i give should generate the center of pressure at exactly the desired point. And because it does not, i could call it an error that should be fixed.
But i see it's not that easy, due to the discrete nature of contacts in such physics engines. They turn on and off with nothing in between, the caching with it's discontinuous updates, etc. And it works well for anything beside robotics.
So maybe an extension where needed is better than trying to improve general accuracy even further.

Not sure. But i guess contacts was the first problem you worked on for this, and it looks like the journey isn't over yet...

[Julio Jerez]

I had a little breakthrough yesterday, and i have just tested it again: Yes, seems i'm pretty much done with the basics. The simulation is perfect and i have precise control. It's now impossible for the ragdoll to loose balance. It's stable like a rock, and agile like Bruce Lee. \ /

wow, that awesome

Regarding contacts, the improvements did not help me. I have to stick at the custom contact joint. At least for now.
The problem with default contacts is: Becasue they are distributed over the whole contact area, some contact force is generated for all of them, and we have no precise control about how much.

that's where the dModel and it derive ndModelArticulation class are useful.
Ther provide that app with a callback at the start, in between contact and solver and after solver
the another are a full step (mean all the substep of a tick as executed.

you can use that to encapsule your contraptions of bodies and joints, that is the update you can get the contact of for the bodies that are attached so joint you are interested it, and condition them to do you want. You can do stuff like add or remove contact points, you can also changed the contact parameters.

If you look at the vehicle code.
after the contact are calculated.
the update does many things. It does things liek
-align the frictions direction to the tire

them it checks if the vehicle has some speed, because the tire model for rubber tire and undefine at low speed.
if the vehicle has low speed or if it has more than one conctat, they just work ad normal rigid body coulomb friction.

when the vehicle has some speed, and it has only one contact located near the contact patch.
the tire friction is not linear, and it is given by the contact point and the distance to the tire center contact point.
it is given by Brush or Pacejka friction model for rubber tires.
I like the Brush model because it is analytic, but does no generate the force decay, so not tire burn.

here the equation gives you the estimated force friction force that the tire can assert.
so what is does is that it set the contact to not be coulomb, but an independent from the normal.

at this time, I just use the values from previous frames, therefore there are some inaccuracies.
However, there is also the inverse dynamics solver, that can be used iteratively to calculate the precises normal forces on each step.
I wanted to do that, but I saw that is does not makes big improvements are compared to using the previous normal forces.

Now I am trying to add stability control, and for that I am trying to get with heuristic, but it seems that
controlling the side lip angle is good case for using the inverse dynamics solver.

anyway, the point is that you can try doing some similar to manipulate the contacts to behave the way you want.

[JoeJ]

Sounds great. And i had tried this alrady, but with the ndModel. Not sure if articulated model already existed back then.
And i could modify the friction forces for example, but i could not move or disable the contacts, nor affecting their normal forces. My changes were ignored, it seemed.
Will try again...

Haha, now my ragdoll is dancing! I mean almost ofc., it's by accident. But it looks like dancing. Low frequency oscillation build up, and the outcome often looks kinda hot. Well, the skeleton originally comes from a female model, and the movements look like belly dance or bouncy, and constantly twisting around itself. It looks hot, yes.
It does not look robotic, or like those silly old Google AI locomotion simulations. It looks much more humkan than anything else. It's an accident, but i can see the potential already now.
This is going to work...

[Julio Jerez]

Haha, now my ragdoll is dancing! I mean almost ofc., it's by accident. But it looks like dancing

Are you able to make a video of it?

I’ve resumed training the robots now that I’ve refactored the SDK and restored almost all the rigid body demos.
Currently, I’m reviewing the reinforcement learning demos.
I also fixed a few bugs in my policy gradient trainer and plan to test it across all demos.
So far, it’s working well on the cart pole problem—the "Hello World" of RL.

It does not look robotic, or like those silly old Google AI locomotion simulations.

I totally get what you mean about RL sometimes producing unrealistic solutions.
If an agent is trained well enough with a powerful algorithm,
it can find ways to succeed in flawed or pathological environments, essentially validating incorrect reasoning.

By the way, are you experimenting with machine learning as well?

[JoeJ]

I've tried to clip the contact positions naively to a plane using the method you've proposed, and it works.
It's not perfect yet, likely because i do not yet adjust normals and penetrations so it's actually solvable, but i'm very optimistic.

Are you able to make a video of it?

Not worth the upload. The dancing is much more bug than feature.

By the way, are you experimenting with machine learning as well?

No, i feel to old to learn it. I'm happy if i can finish what i've started.
There is one application where i would really need it: Texture synthesis, to amplify details using given material samples. Before ML, there were no good results on this problem. Now generative AI can do much more than just that.
But i have an idea for an algorithm that is hopefully good enough.

Well, maybe it's not bad if there are some guys left who still work on something else than AI.

[Julio Jerez]

No, i feel to old to learn it. I'm happy if i can finish what i've started.
There is one application where i would really need it: Texture synthesis, to amplify details using given material samples. Before ML, there were no good results on this problem. Now generative AI can do much more than just that.
But i have an idea for an algorithm that is hopefully good enough.

You are most certainly probably right, my friend.

I like to think that I’m not using AI in the same way the majority of people are.
In a way, the big tech companies have realized the dystopian nightmares portrayed in movies over the past century—stories where a massive computer or robot eventually malfunctions, disagrees with its creators and takes over the world.

Films like Metropolis, 2001 A Space Odyssey, Blade Runner, I Robot, The Matrix, Terminator, Minority Report—and those are just the good ones—warned us about this kind of future.
The saddest part? People don’t even realize it’s happening, but instead of protesting, we are willing participants.

We used to fear that computers would physically harm us, track us with social security numbers, inject us with microchips, or something along those lines. But no one expected the reality we’re living in now:
AI isn’t here to kill you—it’s here to manipulate you into watching fake videos so that big tech can sell more ads.

What’s happening is far worse than what The Matrix or Terminator portrayed.
AI isn’t draining your blood—it’s draining your humanity, your money, your intelligence, and your creativity.

Do a search on any web browser, and the top five or six results are AI-generated.
More often than not, they contain errors, simply because when the Language model does not have at least one data point that is close to the question, it extrapolates, essentially it makes stuff up.

But worse than that, they subtly steer users toward the perspectives of big tech companies.

[Julio Jerez]

After that rant,
I am a hypocrite, I am too jumping on the reinforcement leaning ban wagon.

I spent more time than I expected making my own small deep neural network library and implemented few reinforcement learning algorithm.

What I want to do is to see if I can code a self balancing character.
All my life i have tried this using the zero moment point methods, that's the method used but old companies like Honda making Asimov. But it has always failed for my.

I realized that ZMP only works with models that satisfy linearity. Basically you can only predict the action to correct the next step.
Also ZMP is quite limiting, it only valid for contact.

After so much trials and fail, I gave up.
But then reading about RL, I can see how it seems quite powerfully when it works, and that's a big if when you are doing everything yourself.
But the good point is that RL provides a model less way of training, as opposed to ZMP methods, that count on the physics of the model.

in a nut shelve that's my interest in machine learning.
I thought that maybe I can combine both, if just use the RL.

So far the results leave a lot to be desired.

[Julio Jerez]

Oh BTW.
if you sync, you will see that I have a stable spider walking. but it is quite precarious.

I still have to add some more set up before going to do AI training.

basically, the goal of the training will be to adapt the animation to walk on uneven terrain
shifting the top body.
this is a much better model that I made in blender.

[JoeJ]

What’s happening is far worse than what The Matrix or Terminator portrayed.
AI isn’t draining your blood—it’s draining your humanity, your money, your intelligence, and your creativity.

Yep. It does that, if you are stupid enough. AI might completely undermine our education effords. And future generations of slop humans won't progress anymore at all. They will only consume and recycle from the vaguely preserved reminders of former generations.

But don't worry. It won't be our end. Our end will come much sooner. Becasue we are stupid enough already now, we manage to extract the most promising idots to become our leaders, and then we - billions of us - follow the dictatorship of those leaders without doubt, accepting their old man nostaligia as a proper ideology to follow, and as reason enough to hate and kill each other.

It would be our end. But nature has a mechanism to prevent such endings: Life and death, which not only affects individuals, but nations just as much. Before they can do too much harm, they vanish. And while the process of decay can't be stopped, it can be accelerated at the cost of unneeded suffer, as we see.

What was great won't become great again. What has broken up won't unite again. It's broken and won't become whole again.
But there's always hope for new greatness waiting for Don, and new unions for Vlad.
So who knows. Maybe we wake up tomorrow, surrounded by thorium reactors made in China, soon to be replaced by better fusion reactors. And AI robots will do all our work. We will just do arts or play videogames all day, and we will be healthy all our life. There is no money, and no need for kids to care for us, so no more overpopulation either. We'll have all time in the world to figure out how to get to Andromeda.

Anything is possible. Just not with such a$$holes on top, centralizing all power and ideology in just one hole.

That's my best attempt of saying something positive, to spend some consolidation among two aging men, who eventually have some problems with how quickly the world - be it politics or tech - changes into something different, while we fall with constant acceleration towards death.
Maybe we are a bit like Vlad and Don. Just a little bit of course, since we still live in the real world, not in an imaginary nostalgia of John Wayne movies or Zar tanks.

So, we won't make games great again either. Since that's not possible.
But we can make great games once more! \ /

if you sync, you will see that I have a stable spider walking. but it is quite precarious.

Ah, i'm still remembering the spider video shown here before my time. Always wanted to work on this too.
So i tried to make my data structures modular, to compose 6 or 8 legged creatures as needed.
But then there's always the point where i get inpatient, ditching the higher ideals in favor of quick results.
So i have again hardcoded the higher level stuff to the human skeleton, and the scorpion enemy has to wait...

Will check it out...

[Julio Jerez]

hey joe
check this our
https://www.youtube.com/watch?v=L2S23pouu9U


It's not much, but it means a lot to me.
I’ve spent so much time on this, and honestly it's nothing but frustrations.

I’ve been working hard to get a reliable Proximal Policy Gradient (PPO) trainer up and running,
but I’ve never been able to get it to converge as explained in the papers, and the openai pseudo code.
In fact, it doesn’t seem to perform any better than the vanilla policy, and I have not seen any logical explanation for the method, must people just copy the code.
I have serious doubts about the PPO heuristic—I believe its premises are not just ineffective, but possibly outright flawed and logically unsound, also the explanations they give for the heuristics does not hold any logical sense.
But that’s just my take.

Instead, I decided to move on. Based on what I’ve learned, studying the policy gradient theorem, and the proximal policy gradient, I started developing my own version, that I think I can improve upon.
That’s the only trainer that’s actually managed to learn to control the model effectively.

Anyway, with those results in hand, I’m now moving on to training the quad spider.
The goal is to see if I can get it to learn a policy that imitates an animation while still maintaining its balance.

[JoeJ]

Ha, that's good progress. It's those simple things that make most of the work, so i expect rapid competition from the AI camp from now on...

I should go back to graphics, but still working on the ragdoll for some days more. I hope for basic walking.
But i'm confused about chicken and egg problems. Should i place the next foot step from where i'm currently falling towards, or should i fall towards the already planned next step?
Neither feels right, but i'll see...

Anyway, the simulation is good now. really good. After all those years, i finally have Newton under control. And although what i have does not yet work, i can see it feels real. Animation has never achieved such impressions. This is the future, no doubt. \ /

[Julio Jerez]

Simple models like the cart pole converge so fast that is very misleading.
This model is complex enough to learn few tricks of the trade to make the RL algorithms converge.

once you get a trained model, there's a lot you can learn—and the good news is, it all aligns with the laws of physics.

One of the first things you’ll notice is how the top box gains angular momentum as the ball continuously applies torque to prevent it from falling.

This happens because, by design, the top box is much more massive than the rest of the structure and swings freely from the bar pivot.

As it gains angular momentum, it wants to conserve it—there’s nothing in the system to naturally dissipate it. The only way to reduce that momentum is for the ball to apply a counter-torque. This creates a classic frictionless oscillatory system.

You can clearly see this behavior toward the end of the simulation.

Now, I could redesign the model to make it easier to stabilize, but that’s not the point.
The real test is whether the neural network controller is powerful enough to discover the solution on its own, even in a highly unstable equilibrium.

Next, I’ll try adding some drag to the top box to see if it lasts longer.

I’ve done this before, and what I noticed is that the learning rate slows down significantly.
I suspect that’s because adding drag not only slows down the top box, but also introduces a counter-torque on the ball.
That means the controller now has to learn how to compensate for that additional effect.

One of the simplest solutions would be to place the pivot point of the top box very close to its center of mass. That way, the drag wouldn’t generate a counter-torque.

That’s a perfectly valid design choice if the goal is to build a physical robot—but if I did that just to make the controller converge, I’d be cheating myself.

In any case, I’ll try a few more variations before moving on to the walking spider robot.

[Julio Jerez]

I just added angular drag to the top box and re train it.
For some reason, like everything is RL, it is counter intuitive.
When adding drag to the top box, the algorithm converges to a solution about three times faster.
The model in the video train for 100 million sim steps.
The new one does it about 30 million.

But there is one drawback, adding drag is a lot more noisy,
The learn curve has wild up and down swings.

The new model, when box swing, now it does slow down without falling. No every time, only for reasonable angles.
So that good.

Well anyway, I think I have every thing I need to know with this model.