[S5R3/R4] How to check Performance when Testing new Apps

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,522
Credit: 711,236,085
RAC: 857,401

Hi! RE: PS : shell

Message 77874 in response to message 77872

Hi!

Quote:

PS : shell output and stderr.txt of 2 runs =>

root@alex-Ubuntu:/var/lib/boinc-client/projects/refwusidetest# ./run.sh ./einstein_S5R3_4.24_i686-pc-linux-gnu
Now running: ./HierarchicalSearch --method=0 --Freq=1005.20242518 --FreqBand=0.0161398745876 --dFreq=6.71056161393e-06 --f1dot=-1.58548959919e-09 --f1dotBand=1.74403855911e-09 --df1dot=3.88447721545e-10 --skyGridFile=skygrid_1010Hz_S5R3.dat --DataFiles1='h1_1005.00_S5R3;l1_1005.00_S5R3;h1_1005.05_S5R3;l1_1005.05_S5R3;h1_1005.10_S5R3;l1_1005.10_S5R3;h1_1005.15_S5R3;l1_1005.15_S5R3;h1_1005.20_S5R3;l1_1005.20_S5R3;h1_1005.25_S5R3;l1_1005.25_S5R3;h1_1005.30_S5R3;l1_1005.30_S5R3;h1_1005.35_S5R3;l1_1005.35_S5R3' --tStack=90000 --nStacksMax=84 --pixelFactor=0.500 --nf1dotRes=1 --ephemE=earth_05_09 --ephemS=sun_05_09 --nCand1=10000 -o Hough.out --gridType=3 --useWeights=0 --printCand1 --semiCohToplist -d1
cp: ne peut évaluer `stderr.txt': Aucun fichier ou dossier de ce type
  adding: Hough.out (deflated 91%)

real 13m59.023s
user 12m54.980s
sys 0m6.256s

Stderr.txt =>
2008-05-26 19:12:29.8275 [normal]: Built at: Feb 21 2008 15:57:05

2008-05-26 19:12:29.8277 [normal]: Start of BOINC application './HierarchicalSearch'.
shmget: Invalid argument
Can't set up shared mem: -1
Will run in standalone mode.
2008-05-26 19:12:29.8287 [normal]: WARNING: Can't boinc-resolve result file 'Hough.out'
2008-05-26 19:12:29.8287 [normal]: WARNING: boinc-resolved result file "Hough.out" in local directory - will zip into "Hough.out.zip"
2008-05-26 19:12:29.8288 [debug]: Set up communication with graphics process.
2008-05-26 19:12:30.2650 [debug]: Reading SFTs and setting up stacks ... done
2008-05-26 19:12:49.4284 [normal]: INFO: Couldn't open checkpoint Hough.out.cpt
2008-05-26 19:12:49.4285 [debug]: Total skypoints = 31. Progress: 0,
$Revision: 1.115 $ OPT:3 SCV:9, SCTRIM:2, HLV:3, HP:7
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, c
11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, c
24, 25, 26, 27, 28, 29, 30, done.
FPU status flags: PRECISION
2008-05-26 19:26:26.8072 [normal]: done. calling boinc_finish(0).
called boinc_finish

----------

root@alex-Ubuntu:/var/lib/boinc-client/projects/refwusidetest# ./run.sh ./einstein_S5R3_4.24_i686-pc-linux-gnu
Now running: ./HierarchicalSearch --method=0 --Freq=1005.20242518 --FreqBand=0.0161398745876 --dFreq=6.71056161393e-06 --f1dot=-1.58548959919e-09 --f1dotBand=1.74403855911e-09 --df1dot=3.88447721545e-10 --skyGridFile=skygrid_1010Hz_S5R3.dat --DataFiles1='h1_1005.00_S5R3;l1_1005.00_S5R3;h1_1005.05_S5R3;l1_1005.05_S5R3;h1_1005.10_S5R3;l1_1005.10_S5R3;h1_1005.15_S5R3;l1_1005.15_S5R3;h1_1005.20_S5R3;l1_1005.20_S5R3;h1_1005.25_S5R3;l1_1005.25_S5R3;h1_1005.30_S5R3;l1_1005.30_S5R3;h1_1005.35_S5R3;l1_1005.35_S5R3' --tStack=90000 --nStacksMax=84 --pixelFactor=0.500 --nf1dotRes=1 --ephemE=earth_05_09 --ephemS=sun_05_09 --nCand1=10000 -o Hough.out --gridType=3 --useWeights=0 --printCand1 --semiCohToplist -d1
cp: ne peut évaluer `stderr.txt': Aucun fichier ou dossier de ce type
adding: Hough.out (deflated 91%)

real 13m29.012s
user 12m51.356s
sys 0m4.068s

EDIT : what is the best way to verify the outputs are correct when testing an app with the ref WU ?
I did some "file compare" with ought.out for example, but is seems it's timestamped so a direct ASCII/HEX/BIN compare will not help.
I tried with other files as well, but this led nowhere...
Is there a better way than just verify there's no errors in stderr.txt ?

Hmm... there's something strange here... The output in stderr.txt is NOT consistent with a 4.24 Linux app., which was released in January 2008, yet the stderr.txt contains this line:

2008-05-26 19:12:29.8275 [normal]: Built at: Feb 21 2008 15:57:05

CU
Bikeman

[AF>Futura Sciences]click
[AF>Futura Scie...
Joined: 12 Apr 05
Posts: 34
Credit: 1,923,040
RAC: 0

RE: Hmm... there's

Message 77875 in response to message 77874

Quote:


Hmm... there's something strange here... The output in stderr.txt is NOT consistent with a 4.24 Linux app., which was released in January 2008, yet the stderr.txt contains this line:

2008-05-26 19:12:29.8275 [normal]: Built at: Feb 21 2008 15:57:05

CU
Bikeman

HeHe, well done ;)
The app was 4.35 according built date and running time.
Makes it all clearer now :D

Sorry for that !
I don't know how i messed-up with the file name (as according to the log i didn't mess-up with the shell command line).
The file on the server is really the 4.24:
http://einstein.phys.uwm.edu/download/einstein_S5R3_4.24_i686-pc-linux-gnu

In order to find other versions that ones on the EAH download repository, i looked a bit everywhere, including P2P networks... Maybe that's it.
Should have double checked the name according to the build date.
Thanks for the tip, will know from now.

The run time is now matching well.
Will update the plot later today.

Cheers,
A lame Alex.

God created a few good looking guys.. and for the rest he put hairs on top..

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,522
Credit: 711,236,085
RAC: 857,401

Ahh...I see. I guess when

Ahh...I see.

I guess when you prepare a new chart, Gary would be delighted to see one with a logarithmic runtime-axis, if possible :-)

CU
Bikeman

[AF>Futura Sciences]click
[AF>Futura Scie...
Joined: 12 Apr 05
Posts: 34
Credit: 1,923,040
RAC: 0

Hi, I feel like I'm still

Hi,

I feel like I'm still missing a little something.

- With 4.07, I had very constant results.
4 runs within the same 4-5 sec in high priority.
But i felt it was a bit too slow compare to 4.01 (even if its not surprising 4.07 being a bit slower than 4.01)
So i made another batch of runs. Again, very same results for each run.
But, not matching the first batch results ! Not by much, around 5%.
Still confusing anyway, as I changed nothing in the testing protocol, the app exe is the same...

- With 4.49_0 (the not SSE one).
The result was surprisingly fast from the beginning. Something felt wrong.
So again, I went for another run yesterday.
As for the windows 4.07, same file, same protocol (no background BOINC, high priority, no many stuffs in memory, leave the OS alone during test... )
And now the results feel a lot more right. Non-SSE is now a lot slower than 4.49 SSE or SSE2 as it should be.
Again, results from each test run are really close to each other from the same batch.

For all the other apps i re-ran yesterday, new results match well the previous ones.

So no really answers here, just open questions.
At least, now the results seem pretty consistent.

Datas:

Graph:

Graph with logarithmic runtime-axis:

Well, we'll see if others running ref WU, will have the same experience.
Sorry for the mislead on these 3 apps results (4.07, 4.24, 4.49_0) in my initial post.

Cheers,
Alex.

God created a few good looking guys.. and for the rest he put hairs on top..

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,310
Credit: 250,088,165
RAC: 34,818

For the curious: The

For the curious: The reference workunit should still work with S5R4 Apps.

BM

BM

[AF>Futura Sciences]click
[AF>Futura Scie...
Joined: 12 Apr 05
Posts: 34
Credit: 1,923,040
RAC: 0

RE: For the curious: The

Message 77879 in response to message 77878

Quote:

For the curious: The reference workunit should still work with S5R4 Apps.

BM

Thx for the info Bernd !
Will try to run it ASAP, for the same ref computer, on Windows and Linux to add it to the previous logs.

Cheers,
Alex.

God created a few good looking guys.. and for the rest he put hairs on top..

[AF>Futura Sciences]click
[AF>Futura Scie...
Joined: 12 Apr 05
Posts: 34
Credit: 1,923,040
RAC: 0

So, as expected, no changes

So, as expected, no changes in crunch time with S5R4 apps:

Really slight variations only, if any.
Even if i ran the WU more than once for each app and "average", there is still little glitches in measurements. That should explain the slight variations you might see.
Maybe not for Linux 6.02 SSE one, witch appears to be 3% faster, a little more than the common error between to runs with a same app.
Maybe a slight change in compilers options, or something else... or nothing :D
BernD will probably tell you more on this if there's anything to say.

Cheers,
Alex.

God created a few good looking guys.. and for the rest he put hairs on top..

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 2,143
Credit: 2,948,977,183
RAC: 688,737

And yet, having just reported

And yet, having just reported my first S5R4 result with host 889490, the real-world WU timing is much longer than S5R3.

I'm comparing Windows 4.36 on Intel (between 23 and 25.5 Ksec) with 6.04 - which should be a functionally equivalent Einstein app, but registered over 48 Ksec.

It strikes me that with the DCF re-adjustment, fast hosts will over-cache when they first latch on to S5R4, but will then go quiet as soon as they return their first task and find out what they've let themselves in for. But the initial loading is likely to be nice and consecutive, as this one was.

Anyone like me to try for a 'wiggle and swoop' run with one of my standard quaddies, like we did last time?

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,522
Credit: 711,236,085
RAC: 857,401

RE: And yet, having just

Message 77882 in response to message 77881

Quote:

And yet, having just reported my first S5R4 result with host 889490, the real-world WU timing is much longer than S5R3.

I'm comparing Windows 4.36 on Intel (between 23 and 25.5 Ksec) with 6.04 - which should be a functionally equivalent Einstein app, but registered over 48 Ksec.

It strikes me that with the DCF re-adjustment, fast hosts will over-cache when they first latch on to S5R4, but will then go quiet as soon as they return their first task and find out what they've let themselves in for. But the initial loading is likely to be nice and consecutive, as this one was.

Anyone like me to try for a 'wiggle and swoop' run with one of my standard quaddies, like we did last time?

No quaddie here :-(, but that should be instructive.

As to CPU time per Workunit. the WUs seem to contain "more science" per unit. As I understand this, the stretch of time (a segment of the S5 LIGO science run) that is covered in S5R4 is longer than in S5R3, and basically every workunit looks at the complete stretch of time, but only for a small frequency window.

CU
Bikeman

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,522
Credit: 711,236,085
RAC: 857,401

RE: And yet, having just

Message 77883 in response to message 77881

Quote:

And yet, having just reported my first S5R4 result with host 889490, the real-world WU timing is much longer than S5R3.

I'm comparing Windows 4.36 on Intel (between 23 and 25.5 Ksec) with 6.04 - which should be a functionally equivalent Einstein app, but registered over 48 Ksec.

It strikes me that with the DCF re-adjustment, fast hosts will over-cache when they first latch on to S5R4, but will then go quiet as soon as they return their first task and find out what they've let themselves in for. But the initial loading is likely to be nice and consecutive, as this one was.

Anyone like me to try for a 'wiggle and swoop' run with one of my standard quaddies, like we did last time?

No quaddie here :-(, but that should be instructive.

As to CPU time per Workunit: the WUs seem to contain "more science" per unit. As I understand this, the stretch of observation time (a segment of the S5 LIGO science run) that is covered in S5R4 is longer than in S5R3, and basically every workunit looks at the complete stretch of time, but only for a small frequency window.

CU
Bikeman

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.