There is some progress, having converted all the skygrid files to ecliptic co-ordinates.
Now on my machine the client_state.xml file indicates that for my current WU's ( 650 - 660Hz band ) the cycle length is 293 ( numSkyPartitions ). So I take the ecliptic gridfile for that frequency and slice it, by ecliptic latitude, into 293 latitude bands. Then count how many ecliptic longitude points are in each band:
Does that look familiar or what? Flip it over -
Now that, Ladies and Gentlemen, is The Wiggles !!! :-) :-)
Now we're not the whole nine yards, yet. Not by a long way. This is the general character, not a specific traverse of the points of the work unit ( = slice ?? ). But what it means is that if one or a few consecutive slices/WU's have fewer points than expected as per a pure sinusoid, then the next few will 'catch up' and include more than expected as per a pure sinusoid. And vice versa. This must happen for isotropy to hold. So the wiggles oscillate up and down but with the main sinusoid as the mean behaviour.
Think of it like workers in a field picking some vegetables. Suppose the field was constructed in regular rows with each plant at a regular spaces along the rows. It is a flat field. An obvious picking procedure is to allot one row to each worker. This way, more or less, the work will be equal.
Suppose, perversely, we get the workers to still walk and pick in straight lines - however not aligned with any row, but at some 'odd' but constant angle like, say, 23.5 degrees to the rows. They will each then step across the field, each missing some vegetables or not at a given row depending on the geometry. Some will do more or less work than others, but still it averages out to the total veggies divided by total workers.
Now here we have curved fields/geometry, and the ecliptic tilt is not a 'neat' number or fraction of something else in the problem ...... so it's a bit more complicated. ;-)
To be continued ...... :-)
Cheers, Mike
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
Hmmm....I don't quite get the connction between the runtime and the wiggles in the graph.
Quote:
There is some progress, having converted all the skygrid files to ecliptic co-ordinates.
Now on my machine the client_state.xml file indicates that for my current WU's ( 650 - 660Hz band ) the cycle length is 293 ( numSkyPartitions ). So I take the ecliptic gridfile for that frequency and slice it, by ecliptic latitude, into 293 latitude bands. Then count how many ecliptic longitude points are in each band:
That's where I get lost. How do you slice the points? Using this egg-cutting tool using 292 wires with a uniform grid-width? Hmmm.... but the way E@H does the slices is (I think) that an almost equal number of points (ca 121) are in each slice (=result). So the "wires" are not spaced uniformly. They are spaced so that the outward surface area of the egg slices are uniform.
Hi Mike!
Hmmm....I don't quite get the connction between the runtime and the wiggles in the graph.
There isn't - yet. That's why :
Quote:
This is the general character, not a specific traverse of the points of the work unit ( = slice ?? ).
I was just demonstrating that the origin of the wiggles ( I think ) is the oscillation in the ( sine ) of the latitude of the points with respect to the ecliptic equator when you have a grid aligned neatly in celestial co-ordinates. The amount of oscillation waxes and wanes about the sinusoid. The WU will grab a series of points from the skygrid file and then effectively transform to ecliptic co-ordinates - to generate the sine for the Hough business.
Quote:
That's where I get lost. How do you slice the points? Using this egg-cutting tool using 292 wires with a uniform grid-width? Hmmm.... but the way E@H does the slices is (I think) that an almost equal number of points (ca 121) are in each slice (=result). So the "wires" are not spaced uniformly. They are spaced so that the outward surface area of the egg slices are uniform.
I haven't yet deduced the point traverse/selection for a WU. Indeed if you cut the egg with a number other than 293 ( in this example ) you still get the wiggles but you won't get the period of the main sinusoid right. The 'numSkyPartitions' ( from the client_state.xml file ) is 293 but the number of distinct declinations in the skygrid file is 162. So there is another criteria in play here ....... the point selection/partition algorithm. This may not be deducible for us, mind you, and I wouldn't be surprised if the egg slice didn't have equal spacings. In any case, under the ecliptic transform equal areas doesn't imply equal number of points within each ecliptic latitude based slice- some may get more or less than their fair share due to the 'odd' alignment with the celestial system and the grid granularity - but it averages out OK over the sphere despite some slice to slice variation.
Or put another way, while one can select an equal number of points per partition : that doesn't give equal runtimes as the sines ( with respect to the ecliptic ) of those points will cause variation, and the oscillations about it.
Please note that none of what I have done maybe at all correct - this is reverse engineering. :-) :-)
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
Ah...yes...so basically the wiggles would be "quantization noise", kind of, caused by the discretization of the sky-grid ?
CU
Bikeman
Bingo!!
The mind bender is that we are on a curved surface and the co-ordinates have been effectively tilted as well. So a neat, equal spaced, rectangular grid will never work :-(
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
Along the lines of the skygrid point traversal, I'd like to look at what might be deduced from the client_state.xml file as regards this particular area of entries, eg:
[pre]
.... stuff ....
.... various .....
--skyGridFile=skygrid_0550Hz_S5R5.dat --numSkyPartitions=204 --partitionIndex=41
--tStack=90000 --nStacksMax=121 --pixelFactor=0.500 --nf1dotRes=1 --ephemE=earth
--ephemS=sun --nCand1=10000 -o Hough.out --gridType=3 --useWeights=1
.... other various ....
.... other stuff ....
[/pre]I would be much obliged if anyone would be so kind as to email to me any client_state.xml files from their machine(s)?? As few, or as many, as you like ....
Just attach it/them to an email, zipped if you prefer, and perhaps with the subject line 'client state' so my email junk dog won't bite it, and send to :
hewsmike-[at]-optusnet.com.au
I have a number of hypotheses I'd like to play with, and I need a wide spread of such samples to examine .... :-)
Also if anyone has, or can recall, any particular knowledge of the behaviour of the apps with regard to any of the specific command line settings mentioned in the client_state file then please share your thoughts here. :-)
Cheers, Mike
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
Along the lines of the skygrid point traversal, I'd like to look at what might be deduced from the client_state.xml file ....
Just a few things you might like to think about in case they make what you receive potentially more useful to you.
As you describe it, the state file will contain ... blocks with one such block for every task cached on the machine. If a person had a 0.5 day cache he might have a small number of cached tasks but if he temporarily increased his cache to 5.0 days, he would have a much larger dataset to send to you. Afterwards, he could set the cache back to 0.5 (or whatever) and the machine could work off the excess over the coming several days with no adverse consequences - as long as he didn't go beserk with the temporary increase.
The second point is that is it just the command line flags you are interested in or is it also some performance criteria after the task is crunched? Before crunching starts, there are both the and blocks present but the latter is just a template waiting to be filled in. Here is an actual example
Once crunching has finished, a lot of extra data including all the command line flags, as well as the performance data gets inserted into the result template. I suspect that it may be these filled out result templates that are most useful to you. Again here is an actual example with excess stuff snipped
Detected CPU type 1
2009-02-03 23:12:36.2208 [normal]: This program is published under the GNU General Public License, version 2
2009-02-03 23:12:36.2209 [normal]: For details see http://einstein.phys.uwm.edu/license.php
2009-02-03 23:12:36.2209 [normal]: This Einstein@home App was built at: Dec 2 2008 15:07:40
2009-02-03 23:12:36.2209 [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_S5R5_1.01_i686-pc-linux-gnu_1'.
command line: ../../projects/einstein.phys.uwm.edu/einstein_S5R5_1.01_i686-pc-linux-gnu_1 --method=0 --Freq=937.946848716 --FreqBand=0.0207926846095 --dFreq=6.71056161393e-06 --f1dot=-1.98186199899e-09 --f1dotBand=2.18004819888e-09 --df1dot=1.22837955198e-10 --skyGridFile=skygrid_0940Hz_S5R5.dat --numSkyPartitions=594 --partitionIndex=560 --tStack=90000 --nStacksMax=121 --pixelFactor=0.500 --nf1dotRes=1 --ephemE=../../projects/einstein.phys.uwm.edu/earth_05_09 --ephemS=../../projects/einstein.phys.uwm.edu/sun_05_09 --nCand1=10000 -o ../../projects/einstein.phys.uwm.edu/h1_0937.75_S5R4__1748_S5R5a_1_0 --gridType=3 --useWeights=1 --printCand1 --semiCohToplist --semiCohPatchX=0.013197 --semiCohPatchY=0.013197 -d1 --DataFiles1=../../projects/einstein.phys.uwm.edu/h1_0937.75_S5R4;../../projects/einstein.phys.uwm.edu/l1_0937.75_S5R4;../../projects/einstein.phys.uwm.edu/h1_0937.80_S5R4;../../projects/einstein.phys.uwm.edu/l1_0937.80_S5R4;../../projects/einstein.phys.uwm.edu/h1_0937.85_S5R4;../../projects/einstein.phys.uwm.edu/l1_0937.85_S5R4;../../projects/einstein.phys.uwm.edu/h1_0937.90_S5R4;../../projects/einstein.phys.uwm.edu/l1_0937.90_S5R4;../../projects/einstein.phys.uwm.edu/h1_0937.95_S5R4;../../projects/einstein.phys.uwm.edu/l1_0937.95_S5R4;../../projects/einstein.phys.uwm.edu/h1_0938.00_S5R4;../../projects/einstein.phys.uwm.edu/l1_0938.00_S5R4;../../projects/einstein.phys.uwm.edu/h1_0938.05_S5R4;../../projects/einstein.phys.uwm.edu/l1_0938.05_S5R4;../../projects/einstein.phys.uwm.edu/h1_0938.10_S5R4;../../projects/einstein.phys.uwm.edu/l1_0938.10_S5R4
2009-02-03 23:12:36.2537 [debug]: Set up communication with graphics process.
2009-02-03 23:12:36.4036 [debug]: Reading SFTs and setting up stacks ... done
2009-02-03 23:12:48.8456 [normal]: INFO: Couldn't open checkpoint h1_0937.75_S5R4__1748_S5R5a_1_0.cpt
2009-02-03 23:12:48.8457 [debug]: Total skypoints = 121. Progress: 0,
$Revision: 1.131 $ REV:$Revision, OPT:6, SCVAR:9, SCTRIM:2, HOTVAR:4, HOTDIV:0, HGHPRE:7, HGHBAT:2
c
1, c
2, c
3, c
....
118, c
119, c
120, c
done.
Writing output ...
done.
FPU status flags: PRECISION
2009-02-04 09:09:40.4264 [normal]: done. calling boinc_finish(0).
called boinc_finish
If that is the case, it's a fairly easy task for each volunteer to get you this information. The steps would be (using local preferences):-
* Increase the current cache size locally by say 5 days to stock up with tasks.
* When fully cached up, set the cache size back to normal locally (or just clear local preferences).
* Suspend network activity on the host to prevent uploading and reporting of finished tasks.
* After about 5 days when the excess tasks are all crunched, capture and mail a copy of the state file
* Resume network activity to allow all the crunched tasks to be uploaded and reported.
Maybe you don't need the performance information, in which case please ignore the above comments.
Just a few things you might like ....... maybe you don't need the performance information, in which case please ignore the above comments.
Thank you Gary, indeed I would be interested in the results/performance. So if users would like to act as Gary advises, to then please send me such data too. :-)
Indeed the client_state, plus the other xml files for that matter, are quite informative and extensive documents!! Never really looked at them much before ...
Cheers, Mike.
( edit ) and there is a 'client_state_prev.xml' time stamped about a minute earlier on my machine. So it'd write over the older state file, and fiddle/rename to get a leap-frogging series ....
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
There is some progress,
)
There is some progress, having converted all the skygrid files to ecliptic co-ordinates.
Now on my machine the client_state.xml file indicates that for my current WU's ( 650 - 660Hz band ) the cycle length is 293 ( numSkyPartitions ). So I take the ecliptic gridfile for that frequency and slice it, by ecliptic latitude, into 293 latitude bands. Then count how many ecliptic longitude points are in each band:
Does that look familiar or what? Flip it over -
Now that, Ladies and Gentlemen, is The Wiggles !!! :-) :-)
Now we're not the whole nine yards, yet. Not by a long way. This is the general character, not a specific traverse of the points of the work unit ( = slice ?? ). But what it means is that if one or a few consecutive slices/WU's have fewer points than expected as per a pure sinusoid, then the next few will 'catch up' and include more than expected as per a pure sinusoid. And vice versa. This must happen for isotropy to hold. So the wiggles oscillate up and down but with the main sinusoid as the mean behaviour.
Think of it like workers in a field picking some vegetables. Suppose the field was constructed in regular rows with each plant at a regular spaces along the rows. It is a flat field. An obvious picking procedure is to allot one row to each worker. This way, more or less, the work will be equal.
Suppose, perversely, we get the workers to still walk and pick in straight lines - however not aligned with any row, but at some 'odd' but constant angle like, say, 23.5 degrees to the rows. They will each then step across the field, each missing some vegetables or not at a given row depending on the geometry. Some will do more or less work than others, but still it averages out to the total veggies divided by total workers.
Now here we have curved fields/geometry, and the ecliptic tilt is not a 'neat' number or fraction of something else in the problem ...... so it's a bit more complicated. ;-)
To be continued ...... :-)
Cheers, Mike
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
Hi Mike! Hmmm....I don't
)
Hi Mike!
Hmmm....I don't quite get the connction between the runtime and the wiggles in the graph.
That's where I get lost. How do you slice the points? Using this egg-cutting tool using 292 wires with a uniform grid-width? Hmmm.... but the way E@H does the slices is (I think) that an almost equal number of points (ca 121) are in each slice (=result). So the "wires" are not spaced uniformly. They are spaced so that the outward surface area of the egg slices are uniform.
CU
Bikeman
RE: Hi Mike! Hmmm....I
)
There isn't - yet. That's why :
I was just demonstrating that the origin of the wiggles ( I think ) is the oscillation in the ( sine ) of the latitude of the points with respect to the ecliptic equator when you have a grid aligned neatly in celestial co-ordinates. The amount of oscillation waxes and wanes about the sinusoid. The WU will grab a series of points from the skygrid file and then effectively transform to ecliptic co-ordinates - to generate the sine for the Hough business.
I haven't yet deduced the point traverse/selection for a WU. Indeed if you cut the egg with a number other than 293 ( in this example ) you still get the wiggles but you won't get the period of the main sinusoid right. The 'numSkyPartitions' ( from the client_state.xml file ) is 293 but the number of distinct declinations in the skygrid file is 162. So there is another criteria in play here ....... the point selection/partition algorithm. This may not be deducible for us, mind you, and I wouldn't be surprised if the egg slice didn't have equal spacings. In any case, under the ecliptic transform equal areas doesn't imply equal number of points within each ecliptic latitude based slice- some may get more or less than their fair share due to the 'odd' alignment with the celestial system and the grid granularity - but it averages out OK over the sphere despite some slice to slice variation.
Or put another way, while one can select an equal number of points per partition : that doesn't give equal runtimes as the sines ( with respect to the ecliptic ) of those points will cause variation, and the oscillations about it.
Please note that none of what I have done maybe at all correct - this is reverse engineering. :-) :-)
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
Ah...yes...so basically the
)
Ah...yes...so basically the wiggles would be "quantization noise", kind of, caused by the discretization of the sky-grid ?
CU
Bikeman
RE: Ah...yes...so basically
)
Bingo!!
The mind bender is that we are on a curved surface and the co-ordinates have been effectively tilted as well. So a neat, equal spaced, rectangular grid will never work :-(
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
Along the lines of the
)
Along the lines of the skygrid point traversal, I'd like to look at what might be deduced from the client_state.xml file as regards this particular area of entries, eg:
[pre]
.... stuff ....
.... various .....
--skyGridFile=skygrid_0550Hz_S5R5.dat --numSkyPartitions=204 --partitionIndex=41
--tStack=90000 --nStacksMax=121 --pixelFactor=0.500 --nf1dotRes=1 --ephemE=earth
--ephemS=sun --nCand1=10000 -o Hough.out --gridType=3 --useWeights=1
.... other various ....
.... other stuff ....
[/pre]I would be much obliged if anyone would be so kind as to email to me any client_state.xml files from their machine(s)?? As few, or as many, as you like ....
Just attach it/them to an email, zipped if you prefer, and perhaps with the subject line 'client state' so my email junk dog won't bite it, and send to :
hewsmike-[at]-optusnet.com.au
I have a number of hypotheses I'd like to play with, and I need a wide spread of such samples to examine .... :-)
Also if anyone has, or can recall, any particular knowledge of the behaviour of the apps with regard to any of the specific command line settings mentioned in the client_state file then please share your thoughts here. :-)
Cheers, Mike
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
@Mike, one client_state file
)
@Mike,
one client_state file sent.
RE: @Mike, one client_state
)
Thanks! :-)
Some more joy - I have found that MATLAB has xml parsing into DOM ( Document Object Model ) features! :-)
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
RE: Along the lines of the
)
Just a few things you might like to think about in case they make what you receive potentially more useful to you.
As you describe it, the state file will contain ... blocks with one such block for every task cached on the machine. If a person had a 0.5 day cache he might have a small number of cached tasks but if he temporarily increased his cache to 5.0 days, he would have a much larger dataset to send to you. Afterwards, he could set the cache back to 0.5 (or whatever) and the machine could work off the excess over the coming several days with no adverse consequences - as long as he didn't go beserk with the temporary increase.
The second point is that is it just the command line flags you are interested in or is it also some performance criteria after the task is crunched? Before crunching starts, there are both the and blocks present but the latter is just a template waiting to be filled in. Here is an actual example
h1_0937.75_S5R4__1747_S5R5a_1 0.000000 0 2 i686-pc-linux-gnu 101 16876120000000.000000 h1_0937.75_S5R4__1747_S5R5a 1234630710.000000 h1_0937.75_S5R4__1747_S5R5a_1_0 Hough.outOnce crunching has finished, a lot of extra data including all the command line flags, as well as the performance data gets inserted into the result template. I suspect that it may be these filled out result templates that are most useful to you. Again here is an actual example with excess stuff snipped
h1_0937.75_S5R4__1748_S5R5a_1 35804.880000 0 5 i686-pc-linux-gnu 101 48723500000000.000000Detected CPU type 1
2009-02-03 23:12:36.2208 [normal]: This program is published under the GNU General Public License, version 2
2009-02-03 23:12:36.2209 [normal]: For details see http://einstein.phys.uwm.edu/license.php
2009-02-03 23:12:36.2209 [normal]: This Einstein@home App was built at: Dec 2 2008 15:07:40
2009-02-03 23:12:36.2209 [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_S5R5_1.01_i686-pc-linux-gnu_1'.
command line: ../../projects/einstein.phys.uwm.edu/einstein_S5R5_1.01_i686-pc-linux-gnu_1 --method=0 --Freq=937.946848716 --FreqBand=0.0207926846095 --dFreq=6.71056161393e-06 --f1dot=-1.98186199899e-09 --f1dotBand=2.18004819888e-09 --df1dot=1.22837955198e-10 --skyGridFile=skygrid_0940Hz_S5R5.dat --numSkyPartitions=594 --partitionIndex=560 --tStack=90000 --nStacksMax=121 --pixelFactor=0.500 --nf1dotRes=1 --ephemE=../../projects/einstein.phys.uwm.edu/earth_05_09 --ephemS=../../projects/einstein.phys.uwm.edu/sun_05_09 --nCand1=10000 -o ../../projects/einstein.phys.uwm.edu/h1_0937.75_S5R4__1748_S5R5a_1_0 --gridType=3 --useWeights=1 --printCand1 --semiCohToplist --semiCohPatchX=0.013197 --semiCohPatchY=0.013197 -d1 --DataFiles1=../../projects/einstein.phys.uwm.edu/h1_0937.75_S5R4;../../projects/einstein.phys.uwm.edu/l1_0937.75_S5R4;../../projects/einstein.phys.uwm.edu/h1_0937.80_S5R4;../../projects/einstein.phys.uwm.edu/l1_0937.80_S5R4;../../projects/einstein.phys.uwm.edu/h1_0937.85_S5R4;../../projects/einstein.phys.uwm.edu/l1_0937.85_S5R4;../../projects/einstein.phys.uwm.edu/h1_0937.90_S5R4;../../projects/einstein.phys.uwm.edu/l1_0937.90_S5R4;../../projects/einstein.phys.uwm.edu/h1_0937.95_S5R4;../../projects/einstein.phys.uwm.edu/l1_0937.95_S5R4;../../projects/einstein.phys.uwm.edu/h1_0938.00_S5R4;../../projects/einstein.phys.uwm.edu/l1_0938.00_S5R4;../../projects/einstein.phys.uwm.edu/h1_0938.05_S5R4;../../projects/einstein.phys.uwm.edu/l1_0938.05_S5R4;../../projects/einstein.phys.uwm.edu/h1_0938.10_S5R4;../../projects/einstein.phys.uwm.edu/l1_0938.10_S5R4
2009-02-03 23:12:36.2537 [debug]: Set up communication with graphics process.
2009-02-03 23:12:36.4036 [debug]: Reading SFTs and setting up stacks ... done
2009-02-03 23:12:48.8456 [normal]: INFO: Couldn't open checkpoint h1_0937.75_S5R4__1748_S5R5a_1_0.cpt
2009-02-03 23:12:48.8457 [debug]: Total skypoints = 121. Progress: 0,
$Revision: 1.131 $ REV:$Revision, OPT:6, SCVAR:9, SCTRIM:2, HOTVAR:4, HOTDIV:0, HGHPRE:7, HGHBAT:2
c
1, c
2, c
3, c
....
118, c
119, c
120, c
done.
Writing output ...
done.
FPU status flags: PRECISION
2009-02-04 09:09:40.4264 [normal]: done. calling boinc_finish(0).
called boinc_finish
]]>
1233702638.024615
h1_0937.75_S5R4__1748_S5R5a
1234618319.000000
h1_0937.75_S5R4__1748_S5R5a_1_0
Hough.out
If that is the case, it's a fairly easy task for each volunteer to get you this information. The steps would be (using local preferences):-
* When fully cached up, set the cache size back to normal locally (or just clear local preferences).
* Suspend network activity on the host to prevent uploading and reporting of finished tasks.
* After about 5 days when the excess tasks are all crunched, capture and mail a copy of the state file
* Resume network activity to allow all the crunched tasks to be uploaded and reported.
Maybe you don't need the performance information, in which case please ignore the above comments.
Cheers,
Gary.
RE: Just a few things you
)
Thank you Gary, indeed I would be interested in the results/performance. So if users would like to act as Gary advises, to then please send me such data too. :-)
Indeed the client_state, plus the other xml files for that matter, are quite informative and extensive documents!! Never really looked at them much before ...
Cheers, Mike.
( edit ) and there is a 'client_state_prev.xml' time stamped about a minute earlier on my machine. So it'd write over the older state file, and fiddle/rename to get a leap-frogging series ....
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal