benchmark test: HaWe brickbench – for C# / Mono?

[Helmut Wunder]

hey,
meanwhile I have assembled lots of source codes to compete benchmarks for NXT and EV3 on different software platforms:

http://www.mindstormsforum.de/viewtopic.php?f=71&t=8095&p=64772#p64772

Is someone able and interested in trancoding source codes to C#/Mono to compete with the other programming languages and firmwares?

ANSI C code for NXT (nxtOSEK) and EV3 (gpp CSLite) and LeJOS source codes for NXT and EV3 surely might be useful as coding patterns :
http://www.mindstormsforum.de/viewtopic.php?f=71&t=8095&p=64772#p64499

share and enjoy, and any participation would be appreciated!

[Helmut Wunder]

ps,
link to the source code libs seemed to work faulty – this one should work:
http://www.mindstormsforum.de/viewtopic.php?f=71&t=8095&start=60#p64499

[Vlad Ruzov]

I have ported the HW brickbench benchmark test to the Monobrick C# and then run RobotC and C# test on my EV3 brick.
The final results are:

• Integer add/subtr.
• Integer mult./div.
• Float operations
• Rnd.num.generation
• Matrix algebra
• Arrays shell sort.
• Toal milleseconds

More results can be found here:
https://docs.google.com/spreadsheets/d/1WpeffSpUEWKA_s7lOdseSbkDTSA1RQ_wzAI94Q_HLO0/edit?usp=sharing

Source C# code of that test is kept at these Github repositories:
https://github.com/vladru/MonoBrickBench

[Helmut Wunder]

hello,
thank you very much, I gladly take the Mono C# results up into my table.

Please notice that according to Copyright restrictions it is not allowed to publish your own tables:

© Helmut Wunder 2013,2014.
HaWe brickbench test Comparison tables/speadsheets and results are copyright protected and may not be freely created, copied and / or distributed. Additions and / or changes in the tables require written permission by the author.
HaWe brickbench test – Vergleichstabellen und Ergebnisse sind urheberrechtlich geschützt und dürfen nicht frei erstellt, kopiert und/oder verbreitet werden. Ergänzungen und/oder Veränderungen der Tabellen bedürfen der schriftlichen Genehmigung durch den Autor.
http://www.mindstormsforum.de/viewtopic.php?f=71&t=8095&start=60#p64772

I also obeserved that you did not include the Copyright rules for the code itself:

// hw brickbench
// benchmark test for NXT/EV3 and similar Micro Controllers
// PL: . . . . . . . . .
// Autor: (C) Helmut Wunder 2013,2014
// Ported to . . . . . by . . . . . .
// freie Verwendung für private Zwecke
// für kommerzielle Zwecke nur nach Genehmigung durch den Autor.
// protected under the friendly Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License
// http://creativecommons.org/licenses/by-nc-sa/3.0/
// version 1.08

I gladly publish both your C# code and your C# results in my results table, and of course you may link to my results tables in my Thread, but please delete your table which you have refered to in your own link.

Please understand that these precausions unfortunately had to be taken to prevent faulty and/or uncornfirmed data to be disseminated.

Please send me a link with all published and needed files in raw text code format (just blanks, no tabs), compressed in a zip folder. Github is far too confusing, additionally I need to have the access to the files in my own thread.

Last but not least a couple of questions:
– why do you have different “runs” – what does it mean? How have to be interpreted the divergent data ?
– what about the display bench mark? (As you might observe e.g., for RobotC and Java, grafic benchmark partially have a big influence on the overall benchmark)
– also some data are needed for memory specs provided by C#:
runt. code mem.,
variable mem.
BT raw master+slave
RS485/i2c raw master+slv.
BT, +MC I/O rem.protocol
RS485, +MC I/O rem.prot.
USB MC Daisy-Chain
WiFi MC I/O rem.protocol
feat. double=64bit float
feat.>=4D array/matrix oper.
Multitthreading
API complete (sensor+motor+screen)?

Thank you very much again for your input!

[Vlad Ruzov]

Hello Helmut,
Thank you for interest in my work.

1) I have added Copyright rules to the source code and have pushed changes to the GitHub repositories.

2) I have closed access to the google docs results table referenced above but I have no possibility to change previous post.

3) You can download compressed zip folder with sources here:
https://github.com/vladru/MonoBrickBench/archive/master.zip

Program.cs and RandM.cs are text files with C# code but all others files are needed for build solution.

3) The test was performed in a loop for show a difference between first and others runtimes.
The feature of Mono C# is that exe-file have no compilied executable code but have so-called P-Code which converted to machine-dependent code on the first call.
So the first run of any procedure or function takes longer runtime than the following.

4) The current version of MonoBrick EV3 firmware have no some graphical functions like circle and filled ellipse so I did not make the graphics tests.

5) Please indicate where the requirements, specifications or clarification for tests and data at clauses 3, 10-21 can be found.

I would like also to pay attention that RobotC HW brickbench test runned on my EV3 brick has better results than those that are published on your site.
But at any case, the MonoBrick C# performance about 50 times higher than the RobotC performance.

Regards, Vlad.

[Helmut Wunder]

Vlad,
thank you very much for your input and for your cooperation – I was afraid you might perceive my post as sort of rude, I apologize if you might have had this impression – but it was only meant to avoid obsolete data to misrepresent the real comparison at any later date which would be unfair to the other tests (not sure if I could express my intention correctly). Of course you are free to publish your own results which is in your own responsibility – and I certainly would appreciate if you’d publish them singularily without showing different platform data!

Indeed some changes meanwhile have been made in the RobotC VM which lead to a quicker runtime performance.
The code and the test have been written and performed by Xander Soldaat, and he wrote me that especially the sort algorithm is meanwhile replaced in an internal build by an intrinsic test with the speed comparable to Linux executables.
As soon as the new RC build has been released he surely will publish the updated results.
If you like, you certainly may perform the RC test and publish them, too (with a date time and a RC version stamp), and I then will gladly always update the newest results in the comparison chart. As I don’t have RC, I can’t do the tests by myself.
BTW, there are additional issues about the leJOS benchmarks and I’m waiting for Andy Shaw’s fixes, too, to include them as well, just post them please directly here in the forum again if you wish!

About Mono/C#:
Now I also understand the loop thing in your table. It appears to be something similar to the Java JIT (just-in-time) compiler. Actually it’s complicated to unite all those runtime data for each VM and each loop in my spreadsheet table – finally it would be bursting at the seams. Maybe just loops #1 + #3 would give a representative cross-section about this, I’ll add them ASAP.

Just to outline what the rest of the benchs is about (of course I know that you won’t know all the details without exception):
I would appreciate to get to know the matrix and the sort tests, possibly also the text_out. If there is no graphic lib, we’ll have to skip that thing (nxtOSEK also provides no graphic lib, but Martin Aumair wrote and published his own lib so that it could be performed).
About memory: is there a memavail() function where you can read this SRAM mem during runtime? Is it up to 64MByte like gpp C or even more, maybe up to all built-in 264 MByte?
Is the code size limited in either way?
About BT or daisy-chaining or multi-robot-communication and all that: Do you have an idea if direct I/O control is possible? Is there something written in the documentation? (The Lego FW provides 1+3 for USB daisy chaining and 1+7 for raw BT comm).
The multi-dim array thing is about how to manipulate multi-dim arrays, e.g. after beeing passed to functions. If you wish to compare the gpp C code to the RobotC or the nxtOSEK code you’ll recognize what I mean:
by gpp C, you can access both rows and columns individually/seperately, in other cases this is not possible, one has to multiply the rows by number of columns to access the cell in a virtual linearized array.
The first feature is the better one, of course, and this is what this point is about.
The last things about int, long, and float already have been answered in our German forum (you are welcome to join it!): http://www.mindstormsforum.de/viewtopic.php?f=71&t=8095&p=65299#p65297

The last line is about API completeness:
– can motors be commanded to run certain degrees by certain pwm? Is there a PID control to approach encoder targets?
– can all sensors of Lego EV3 and NXT kits be polled? Are there general i2c functions to poll 3rd party devices, too (even chained ones by reading/writing different single addr and data registers)?
– Screen API is complete AFAIK for writing text to any (x,y) position, graphics are not available as already mentioned.

Anyway, thanks again for your interest, your input and thanks for sharing your code! 🙂

[Helmut Wunder]

short addendum:
found your matrix and sort benchs, I missed them before! http://www.mindstormsforum.de/viewtopic.php?f=71&t=8095&start=75#p65290
Added ’em already to the table! http://www.mindstormsforum.de/viewtopic.php?f=71&t=8095&p=64772#p64772

[Vlad Ruzov]

Helmut,
please do not be alarmed – I was not embarrassed or offended by your post.

Regardin test looping:
My assumption that the first loop runtime is not very important.
When the program control a real device (robot etc.) it have repeat management algorithm in a loop.
First loop runtime simlpe gives a delay at the start, but does not affect the response time in the future.
So, perhaps in the test results will be reasonable to publish average runtime of several (at least 5) loops…

I will look to your questions and tasks and will try come back ASAP.

Thank you.
Vlad.

[Helmut Wunder]

Vlad,
about Mono/C# , I meanwhile updated the Mono/C# benchmarks acc. to your results:
http://www.mindstormsforum.de/viewtopic.php?f=71&t=8095&p=64772#p64772
Any news about the “?” positions?

about your RobotC tests: is it the same source which I have published?
http://www.mindstormsforum.de/viewtopic.php?f=71&t=8095&start=60#p64501

if not: could you please send me a link to the RobotC source code you have been using?
There meanwhile is a 4.25 pre-release but there is no test result output any more, surprisingly, as reported by a different user:
http://www.mindstormsforum.de/viewtopic.php?f=25&p=65395#p65382

kind regards,
Helmut

[Vlad Ruzov]

Hello,

I have made some memory tests under C#.
Memavail() is not available in .Net platform but there are many other memory functions.
Unfortunately most of them does not working in Mono/C#.
At the end I decided simply try to allocate as many memory as possible.
As a result, I found that the maximum memory szie available for code and data is 42 megabytes.
The source code of the test program can be found here:
https://github.com/vladru/BrickMemory/archive/master.zip

Regarding other points for Mono/C#:
12 – 17 : communication function are not implemented at present,
18: 64 bit.float – YES
19: >=4D arrays – YES
20: Recursions – YES
21: Multithreading – YES
22: API complete – ??
motors – YES
standard sensors (EV3 and NXT) – YES
some of HiTec sensors too (http://www.monobrick.dk/MonoBrickFirmwareDocumentation/annotated.html)
text screen – YES, graphic – whie NOT.
23: IDE – YES. Super powerful Xamarin Studio including debugger with breakpoints.

So, as I can see at Mono/C# firmware sources the graphic functions will be implemented in the next firmware release and then I will try to make graphics test.

Regarding RobotC:
I have used ‘benchmak EV3.c’ from the standard RobotC installation (v.4.10) (http://robotc.net/download/lego/)
I will see into new 4.25 robotc release soon…

/Vlad

[Vlad Ruzov]

I just tried RobotC 4.25 release candidate and very disappointed.
There is no one standard example that can be built successfully!
While I roll back to the old version…

[Helmut Wunder]

hi Vlad,
I’ll add what you reported to the spreadsheet!
(actually Xamarin is not quite “small and made-to-measure” though, IMO, compared to BricxCC, RobotC, or the Sketch IDE)

About RobotC, I’m curious where the code is to be found.
As I don’t use it – can you please post the benchmark sourcecode here please or link to a zip file instead?
I don’t have access to the file.
Is it still part of the 4.25 RobotC example files?

thanks for your input!
Helmut

[Vlad Ruzov]

Helmut,
The source of the HW brickbench benchmark test can be found after RobotC v.4.10 installation in the directory:
C:\Program Files\Robomatter Inc\ROBOTC Development Environment 4.X\Sample Programs\EV3\Miscellaneous
file ‘benchmark EV3.c’
I have upload it on the google drive:
https://drive.google.com/file/d/0B2AlHKJ1hLoDejktS3NEbHNfWUE/edit?usp=sharing

Unfortunately in the v.4.25RC this sample is missing.

Vlad.

[Helmut Wunder]

(edited)
thank you!
about the 4.25 samples files this is correct, it’s probably then on Xander’s homepage.
It actually was not meant to be distributed with RobotC releases because this is a commercial product.
Private distribution for private purposes of course is ok.

[Helmut Wunder]

the code distributed with RC 4.10 which you have linked to is probably the 3.62 NXT version – is this correct?

// hw brickbench
// benchmark test for NXT
// PL: ROBOTC 3.62 for NXT

The EV3 version should be for the 4.10 EV3 version instead.
Is there a 2nd benchmark file additionally?
The EV3 version is supposed to have a firmware-based shellsort or quicksort routine which might be up to 100x faster.

[Author]

Posts