First impression of Newton

[rogerdahl]

Today is the first time I had heard about Newton Dynamics and I was thinking it might be useful for people that have been involved in the project for a long time to see the first impression that I, as a potential new user, got of the library and the project as a whole.

I've been wanting to do a small hobby project that involves some physics simulation for a while and found myself with some spare time this afternoon. I'm on Linux and knew that I would want to use an open source physics engine but didn't know what was out there. After some general searches, I had a list of a handful of libraries that seemed promising. After adding a few "this engine vs that engine" searches to further narrow things down, I found several very positive mentions of Newton, including a paper called, "An Evaluation of Open Source Physics Engines for Use in Virtual Reality Assembly Simulations" in which Newton came out as the most accurate. So, even though things can change quickly in the open source landscape and the paper was from 2012, I decided to give Newton a try.

Plugging "Newton Dynamics" into Google brought me directly to the home page, which looked nice and professional. I immediately noticed the centrally located and helpful sounding "Links to demos, tutorial, FAQ, etc" and, wanting to see some code examples, clicked the link. Focusing directly on the "Main Contents" section of the new page, I failed to notice the "currently migrating" disclaimer at the top and proceeded, only to find that every single link on the page was broken. Whenever I get to a project that has a bunch of broken links, I generally think, "abandoned project".

After some futile clicking, I returned to the home page thinking that I would just install the dev version of the library and look for examples there. A quick scan for install instructions on the home page yielded only the "Download and Fork at GitHub" link at the very top. That's odd, I thought. No packaged versions and no Linux distribution specific info? Well, I figured, maybe they're disciplined about doing development work in separate branches. So I clicked my way to GitHub. Hm, no information in the README.md except 3 links, one to the page full of broken links, one back to the home page and one to the forums. Where was the build instructions and where was the overview of dependencies? None of this fit with what I had read, that indicated that Newton was already a mature project years ago.

A quick look at the stats on GitHub showed both that there was ongoing coding being done and that a lot of hard work has gone into the project. 10 million code changes spread over 2500 commits, with essentially all work having been done by a single person after the project was created on GitHub in 2013. So far, so good! But when I took a look at the branches, I found that there was only a single branch and that development was happing directly there. Hm, new users are being referred directly to a development branch of the library? Would it build? So, after a "git clone", I ran cmake with the Makefile target. But, not surprisingly, cmake errored out due to missing dependencies. I hadn't found an overview of those. There was no INSTALL file either.

Next step was to repeatedly run cmake and install a new dependency each time cmake errored out. Fortunately, I have enough experience that I could see what's missing from the errors, but that's probably not the case for everyone. After the dependencies were in place, cmake ran ok and I got the makefiles. But now, the compile errored out. It was due to a __debugbreak() left in the code. The function is Windows / Visual Studio specific. So, even though Newton is described as a cross-platform library in the first sentence on the home page, the current branch has not been tested on Linux.

So, I, as a new user, was directed to download a development branch of the library directly from GitHub. I was given no build instructions and no overview of the dependencies. And the build had not been tested on Linux.

After taking out the __debugbreak(), I got the library to build. I then looked around in the source for examples and found the demosSandbox app. It looked promising, but it seemed like a bit of an odd example, everything having been rolled into a huge monolithic app. The size of the binary was almost 30MB. I found that some of the demos in there crashed, which was not surprising given that it was a dev branch. But before diving into the code, I decided to take another look for online documentation. This time, I tried the wiki and found working links, where one was for the API. Nice! But wait, there were links for version 1.35 and 2.0 but the source I had looked like it was version 3. If there was any documentation for the version 3 API on the wiki, I couldn't find it.

But, I thought, maybe the changes between version 2 and 3 are all internal, so the API is either unchanged or backwards compatible. So I decided to give things a try with the 2.0 API docs. After skimming through the "getting started" and API docs, I found that the first thing I needed to do was to create a NewtonWorld object with NewtonCreate(), which takes two function pointers that can be set to NULL. I wrote up the call, but, nope, "error: too many arguments". It looked like I would be writing against an API for which there was no current documentation.

So, it was time to hit the forums. General Discussion seemed like the most natural place to start and I clicked my way in there. Ok, it started out with an announcement section and the top announcement had a funny sounding title, "As it turned out, the emperors had not cloth." I opened it, and ended up reading through what looked like some kind of ongoing bitter feud. Apparently, the main developer feels that Newton's user base is diminishing because Newton is being maliciously undermined by competitors.

Mulling that over and thinking about the first impression of Newton that I had gained over the last couple of hours, I realized that, if it's indeed the case that people are moving away from Newton, it seemed unlikely to me that it's because of FUD that may be directed against Newton by competing projects. It seemed much more likely that the main reason would be that potential users are, like me, unaware of any FUD. Instead, they try using Newton, encounter the type of issues that I had just encountered and simply move on to a library with better documentation and better packaging. If the other project makes it easier to get up and running, it won't matter to most people if it's technically less accomplished. And if problems come up with the other library, which is now familiar to them, they'll probably try to fix things there instead of porting their project over to another library, so they're gone forever.

It's also worth noting that, to the new user that encounters threads such as "As it turned out, the emperors had not cloth", it won't matter much who's right or wrong, and they certainly won't dive into years of history to understand what's going on. Instead, it will just make them wonder if they're investing their time in the best library.

I see that a huge amount of work has gone into the Newton code, and stuff online indicates that it's an excellent library. But it's clear that other parts of the project need some TLC. Of course, that's pretty common with open source projects, the main reason being that it's more fun to write code than documentation. For myself, I'm going to keep trying to use Newton for my small project.

[d.l.i.w]

@Julio please let's not start this discussion again - it's the least relevant topic for a new user.

@ rogerdahl

Most of what you mentioned has been discussed on this forum many times and - from my observation - most of us using Newton are aware of these shortcomings.

Newton once was a closed source library (it has been open sourced 5 years ago) and so - traditionally - most of the activity happens in this forum (ask for help, bug reports, feature requests, ...).

There was an initiative to move the existing wiki (which is getting more and more outdated) to Github, but no one actually did the work.

If you want to use the "latest stable" version, you can download Newton 3.14 from the release section of Github, but generally it is not a bad idea to use the latest version.

AFAIK Julio mainly works on Windows, so sometimes thinks break on Linux - but as soon as someone notices, things get fixed quickly.
Usually I regularly check the Linux build as far as my time permits.

To build Newton for Linux you can use the "official" makefiles in coreLibrary_300/projects/ (these are probably broken right now) or CMake. CMake is "maintained" by me - so it's me to blame if things don't work as expected.

That being said - how to start?

Actually the tutorials on the wiki are still good to learn the basics of how to use Newton and most of the information still is valid. The API did change a bit though - as you noticed.

The most up to date documentation are the comments in coreLibrary_300/source/newton/Newton.cpp.

Besides this forum is the best way to get more information.

[d.l.i.w]

@Julio
__debugbreak(); in dBezierSpline.cpp, line 516 breaks the Linux build

[Julio Jerez]

I can see __debugbreak()
but how does dBezierSpline.cpp does it? do you have a line error?

I use __debugBreak in windows because is a goo way to have the code set breakpoint is release mode, so that it track some time hard to find Bug.
as a matter of fact In the file dBezierSpline.cpp I inserted a debugbreak like so that I can catch a divide by zero that some time happen on a line that should not.
The problem is that in the moment I place the line the compiler generates different code, and the divide by zero does no happen. It seems like a visual studio problem because I analyzed the code and there is not way that should happen, is the __debugBreak that you are talking about

[d.l.i.w]

Yes, it is. The error is

dBezierSpline.cpp:516:17: error: ‘__debugbreak’ was not declared in this scope

GCC does not know __debugbreak(). If you still need it, can't you put it in a #ifdef _MSC_VER?

[Julio Jerez]

ok I commented it out.

But I do not think the bug is fixed, I think that me inserting that line is what cause the bug to go away. Basically in eh code below by definition den can never be zero. but for some reason in release mode when a point is so close to the origin of a spline segment even if he value is no zero, rounding error can make be zero. but if I add any line there to check when it happens, the compiler rearrange the code and save the variable to memory causing a truncation to 64 bit.
I can fix if by adding a check that is den is too small then the point is at the beginning of a segment.
But I like to know what this is happing because it should not.

Code: Select all

      dBigVector dist (closestControlPoint - point);
      dFloat64 num = derivatives[1] % dist;
      dFloat64 den = derivatives[2] % dist + derivatives[1] % derivatives[1];
//      if (dAbs (den) < 1.0e-6f)
//         __debugbreak();

[d.l.i.w]

Thanks, the Linux build works fine again.

As for the crash, I have no idea. At least I can't remember the demo sandbox to crash because of division by zero.

[Julio Jerez]

I think this is a visual studio 32 bit release mode thong.
All modern compiler no longer use x87 code, the use sse registers, which do a automatic truncation to the precision, visual studio in 32 bit unless you set the precise mode option, in 32 bit will use x87 code and will never save intermediate values to memory.
This cause lot of rounding off bugs, you can have a value that is numerical zero, by in register it is not, whoever if you divide by that value you get a divide by zero, because the result will not fix in the float format.
is nothing to worry about this is just Visual studio 32 bit nonsence, after all these years.

[Julio Jerez]

Ok I am going to try to answer some of your questions.

Hm, no information in the README.md except 3 links, one to the page full of broken links, one back to the home page and one to the forums. Where was the build instructions and where was the overview of dependencies? None of this fit with what I had read, that indicated that Newton was already a mature project years ago.

The engine come with a normal Make file, and all the dependencies are in wit the engine.
The only real dependency is pthread which is part of eth archive for Widows system, and is standard for all others. The only one if wxWidgets but is only for buidling the demos.

I ran cmake with the Makefile target. But, not surprisingly, cmake errored out due to missing dependencies. I hadn't found an overview of those. There was no INSTALL file either.

I do no make the cmake script the is a Make file, but the C Make is more flexible, unfortunately I do not know much about Cmake, therefore is lag a little behind after I add new changes. I belie is was corrected now.

So, even though Newton is described as a cross-platform library in the first sentence on the home page, the current branch has not been tested on Linux.

My definition of cross platform is that I will compiled and run on any platform that is capable of compiling C++ iso standard.

So, I, as a new user, was directed to download a development branch of the library directly from GitHub. I was given no build instructions and no overview of the dependencies. And the build had not been tested on Linux.
After taking out the __debugbreak(), I got the library to build.

sorry you had a hard time, but it is a lot easier to build project now that we have Cmake script, the older days we needed to run dreaded config and the make file.

I then looked around in the source for examples and found the demosSandbox app. It looked promising, but it seemed like a bit of an odd example, everything having been rolled into a huge monolithic app. The size of the binary was almost 30MB. I found that some of the demos in there crashed, which was not surprising given that it was a dev branch.

is 30 mg because of wxwidget, which has nothing to do with engine, the entire engine is under 800 k in windows, My guess is similar size in other OS.
yes some demo will crash and will look odd because many of then are pieces of code that I have collected over the years from users that expose a particular problem. this is why some look odd and even repetitive.

I thought, maybe the changes between version 2 and 3 are all internal, so the API is either unchanged or backwards compatible. So I decided to give things a try with the 2.0 API docs. After skimming through the "getting started" and API docs, I found that the first thing I needed to do was to create a NewtonWorld object with NewtonCreate(), which takes two function pointers that can be set to NULL. I wrote up the call, but, nope, "error: too many arguments". It looked like I would be writing against an API for which there was no current documentation.

The api changes very little from release to release, but is does evolve.
That particular issue of passing two function pointer for memory allocation was fixed when people requested the ability to launch multiple worlds.

Apparently, the main developer feels that Newton's user base is diminishing because Newton is being maliciously undermined by competitors. Mulling that over and thinking about the first impression of Newton that I had gained over the last couple of hours, I realized that, if it's indeed the case that people are moving away from Newton, it seemed unlikely to me that it's because of FUD that may be directed against Newton by competing projects. It seemed much more likely that the main reason would be that potential users are, like me, unaware of any FUD. Instead, they try using Newton, encounter the type of issues that I had just encountered and simply move on to a library with better documentation and better packaging. If the other project makes it easier to get up and running, it won't matter to most people if it's technically less accomplished. And if problems come up with the other library, which is now familiar to them, they'll probably try to fix things there instead of porting their project over to another library, so they're gone forever.

I do yes. At least in the early day primarily from tow dishonest people Adrian Boing and Keeneth Bodin. At this point I do not really care of people the engine or not, I continue developing the engine for myself because I love what I do and I am too old to change career.

It's also worth noting that, to the new user that encounters threads such as "As it turned out, the emperors had not cloth", it won't matter much who's right or wrong, and they certainly won't dive into years of history to understand what's going on. Instead, it will just make them wonder if they're investing their time in the best library.

you are probably right in the end is does not matter and the truth is that a company can make a video game with bad library or with a good library. My point is that the people who are doing this appraisal stuff are extremely dishonest and know better that the are misleading and misinforming other people.

I see that a huge amount of work has gone into the Newton code, and stuff online indicates that it's an excellent library. But it's clear that other parts of the project need some TLC. Of course, that's pretty common with open source projects, the main reason being that it's more fun to write code than documentation. For myself, I'm going to keep trying to use Newton for my small project

well if it makes you fell better, it is OK to try other libraries first some are very good plus they all have more of the TLC stuff that you are looking forward, Newton is probably not going to have that anytime soon.

[rogerdahl]

All modern compiler no longer use x87 code, the use sse registers, which do a automatic truncation to the precision, visual studio in 32 bit unless you set the precise mode option, in 32 bit will use x87 code and will never save intermediate values to memory.

It looks like you're saying both that modern compilers don't use x87 and that they do use x87?

When compiling for x86 with Visual Studio, you can select how to handle floating point. You can use only x87 or you can use SSE, SSE2, AVX or AVX2:

https://msdn.microsoft.com/en-us/library/7t5yh4fd.aspx

If you select SSE or SSE2 (the default), the compiler will still use x87 instructions whenever it seems best for performance. However, by default Visual Studio also enables the precise FP option:

https://msdn.microsoft.com/en-us/library/e7s85ffb.aspx

Which causes FP operations to be truncated as you mention.

This cause lot of rounding off bugs, you can have a value that is numerical zero, by in register it is not, whoever if you divide by that value you get a divide by zero, because the result will not fix in the float format.
is nothing to worry about this is just Visual studio 32 bit nonsence, after all these years.

It seems like, with Visual Studio's default settings, you won't get the issues with inconsistent rounding.

[rogerdahl]

Thank you for the information, Julio and d.l.i.w. It seems like some fairly small changes to links and some lines of info could make a big difference in getting started. I'll try to find the time to do a pull request with some small tweaks.

[Julio Jerez]

It seems like, with Visual Studio's default settings, you won't get the issues with inconsistent rounding.

yes that's the problem, all newton project use Fast (/fp:fast) in visual studio.
this is done delivery on purpose, so that the code is robust for all settings.
I do not like the idea of achieving stability by using a compile setting.
All newton code is check by hand for floating point accuracy.
This bug we are talking about is just a utility function in the newton tool library, it is no part of the physic engine code, the divide by zero happen in 32 bit release, and I believe only in VS 2013, because I never had in 2011, or 2012 or any of the older versions, and I know how to fix it. what I want to do is try to catch the citation that causes it.
anyway, the point is that the engine is written to compile robust code even with relaxed numerical precision settings.

[Stucuk]

The main problem with documentation is that the people who did it in the past are no longer active and newer people don't want to do it. What would be good would be automatically generated documentation which is generated from the newton.cpp file. There are things like Doxygen which can generate documentation(I have never personally used any).

I have been working on a new layout for Newton which would make finding stable builds easier (Though it would need to be kept upto date). (I don't have FTP access so it would depend on Manny)

I am currently away from home currently so i can't do anything until the end of the week.

[rogerdahl]

I have been working on a new layout for Newton.

The new layout looks really promising! What's already there would have made a big difference the first time I got to the site.

[d.l.i.w]

Personally I don't see that much of a difference. Wouldn't be it easier to to improve the existing page?

I would split the link "Download and Fork at Github" into two:
a) Download
b) Source at Github

Or instead of b, add one of these https://github.com/blog/273-github-ribbons

Download would either link to a new page (similar to what Stucuk proposed) or to https://github.com/MADEAPPS/newton-dynamics/releases.

In case of a dedicated download page, I would recommend to generate the links to the release versions from the Github release page, if possible - otherwise it will be outdated very soon.