benchmark test: HaWe brickbench – for C# / Mono?

[Anders Søborg]

Hi there

Here is a screenshot of my first run. Is this what you need?

/Anders

Attachments:

You must be logged in to view attached files.

[Helmut Wunder]

thank you very much for your input!

to my surprise these new data differ a lot from previous test results (range shown in braces for the 1st to the 5th run):
your new data by 1st run show more or less identical values which had been performed previously not until far later runs, partially even faster than already the gpp C results for native executables which is actually more than surprising (even considering that for gpp C the Lego VM is still running simultaneously) – so this is now a much better performance for the start than before :

no. new (old 1st - 5th)
0     1   ( 15 - 1)
1    25   ( 35 - 26)
2   233   (212^- 155^) // renewed
3    12   ( 19 - 13)
4    40   (229 - 35 )
5   150   (299 - 179)
6   219   (395 - 268)
7   158   (475 - 185)

are you using a different compiler than before? in case yes: which version?
for completion of the list I’m also very curious about how the values would be for e.g., the 3rd and the 5th run additionally!

Thank you for your efforts!
Helmut

[Tcm0]

Anders, what did you do to your EV3? :O

[Helmut Wunder]

maybe pimped by a BBB ? 😛

[Anders Søborg]

Hi

I simply AOT compiled the program. But I think that the image that I showed in my last post was from the second run. Here are images from four runs.

Attachments:

You must be logged in to view attached files.

[Anders Søborg]

Hi and here is the image from the fifth run…

/Anders

Attachments:

You must be logged in to view attached files.

[Helmut Wunder]

ok – I’ll conclude:

no. new min/max (old 1st - 5th)
0     1 -   5   ( 15 - 1)
1    25 -  67   ( 35 - 26)
2   227 - 245   (212^- 155^) // renewed
3    12 -  18   ( 19 - 13)
4    40 -  98   (229 - 35 )
5   150 - 176   (299 - 179)
6   219 - 333   (395 - 268)
7   158 - 302   (475 - 185)

would this be correct ? (I#m a little confused, admittedly 😉 )

[Anders Søborg]

Hi again

I did a run without AOT compiling the program. As expected the first run is slow while the remaining runs are very fast. See attached images

Attachments:

You must be logged in to view attached files.

[Anders Søborg]

Hi and here is the image from the fifth run without AOT…

Attachments:

You must be logged in to view attached files.

[Helmut Wunder]

what is AOT and what is not-AOT ?
what is the common way ?

[Anders Søborg]

Hi there

AOT is described here.

what is the common way ?

Hmmm hard to tell. Pick the one that rates best 🙂

/Anders

[Helmut Wunder]

do I understand correctly:
AOT is precompiled,
and instead it would be like JIT for leJOS ?

so if it was precompiled – there shoudn’t be any remarkable runtime differences – correct ?

FTM, this would be the benchmarks from best to worst case (either if AOT or JIT, and by either run:

no. new min/max (old min/max)
0     1 -  16   (  1 -  15)
1    25 -  67   ( 26 -  35)
2   227 - 245   (155^ -212^) // renewed
3    12 -  18   ( 13 -  19)
4    40 -  98   ( 35 - 229 )
5   150 - 176   (179 - 299)
6   219 - 333   (268 - 395)
7   158 - 302   (185 - 475)

would you say this is correct ?
(tbh, a printed list would be a little more clearly arranged for evalutaion than just photos 😎 )

[Anders Søborg]

Hi Helmut

AOT is precompiled,
and instead it would be like JIT for leJOS ?

Correct.

so if it was precompiled – there shoudn’t be any remarkable runtime differences – correct ?

Actually it is not as simple as that.

Not AOT compiled

If it is not AOT compiled then you would see a high performance penalty the first time you run the test, this is actually also the case in my tests. The 2nd time you run the test the code was been JIT compiled and it will execute as fast as possible. This is also what you can see from the results.

AOT compiled
If the code has been AOT compiled there is no need for JIT compiling the code and you will not get a performance penalty on the first run. However since it is pre-compiled it is not optimized in any way (compared to JIT compile), so that is why the second AOT run is slightly slower than the second run of the Non AOT run.

So the conclusion is that if you can live with a performance penalty the first time some code is executed then don’t bother AOT compiling the code otherwise AOT compile the code.

would you say this is correct ?

The table seems to be correct – but did you use AOT or JIT

/Anders

[Helmut Wunder]

thank you for your explanation, now I see clear 😎

– I took the best and the worst runs of 1-5, of either way –
just observed:
1 current value had to be corrected, i.e. (4) for matrices.

The HaWe Brickbench Benchmark comparison Tables now had been updated!
(some questions marks remaining for daisy chaining, raw BT and raw Wifi etc.
if you should encounter issues, mistakes or bugs: please feel free to report, inputs are highly appreciated!):

http://www.mindstormsforum.de/viewtopic.php?f=71&t=8095&p=64772#p64772

thank you Anders for your input and your participation !
Over all really a great job! 😎

[Author]

Posts