Skygrid file analysis
30 Jan 2009 5:13:59 UTC
Topic 194172
(moderation:
I thought I'd fork of this line of thinking from 'S5R5 Performance Analysis' as a distinct thread.
I think I've been able to obtain all the skygrid files for S5R5, from 60Hz through to 1300Hz inclusive. If anyone has, or knows of, any outside this range then please PM me.
Quote:
I found these scattered amongst various non-self-evident [ don't ask! :-):-) ] directory structures from the root at
http://einstein.phys.uwm.edu/download/
each file being time stamped ~ 20-Nov-2008 21:43
My client_state.xml file has the following example entry which contains hints that suggested this location ( plus mirror sites ):
[pre]
skygrid_0550Hz_S5R5.dat
175984.000000
0.000000
7a3ed62280da49c24574b14fb3b45ace
1
http://einstein.ligo.caltech.edu/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.phys.uwm.edu/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.aei.mpg.de/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.astro.gla.ac.uk/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.ligo.caltech.edu/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.phys.uwm.edu/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.aei.mpg.de/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.astro.gla.ac.uk/download/fd/skygrid_0550Hz_S5R5.dat
[/pre]Yes, it did take a while to round them all up. :-)
I've uploaded these to my website, 125 in all to date. Anyone is welcome to download any file of interest by invoking, say with your browser, the web address:
http://downloads.mikehewson.id.au/skygrid/zips/skygrid_XXXXHz_S5R5.dat.zip
where XXXX runs from 0060 to 1300 ( note the 4 digit positions with left sided zero padding ), in steps of 10 ( ie. 0060 0070 0080 ..... 1280 1290 1300 ).
Quote:
They are the same compressed files that BOINC downloads by the names 'skygrid_XXXXHz_S5R5.dat', I've just added the '.zip' ending for the sake of file association in Windows.
Each will unzip to produce a non-compressed ASCII file by the name of 'skygrid_XXXXHz_S5R5.dat'.
Quote:
Thus, for instance :
http://downloads.mikehewson.id.au/skygrid/zips/skygrid_1110Hz_S5R5.dat.zip
can be downloaded and un-zips to
skygrid_1110Hz_S5R5.dat
The files have one sky position/point per line, each containing two ASCII strings separated by whitespace representing the numerical entries right_ascension and declination ( ie. on celestial sphere ) in radian measure. The file sizes are from ~ 10KB @ 60Hz to ~ 4.5MB @ 1300Hz.
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
Skygrid file analysis
)
Now, what to do with the files? For good background I might suggest reading this very helpful post of Bernd's. I'll quote a key part ( my emphasis in bold ):
Note the sky-grid files are in celestial co-ordinates, but the interesting behaviour for our work units is in ecliptic co-ordinates. Bikeman has already noted that the peaks of WU times are at/near the ecliptic poles, with notable minima at/near zero ecliptic latitude.
What this means is that one can project a line that goes from the south pole of the Sun through to it's north pole, right out to the celestial sphere.
The north ecliptic pole is somewhere in Draco with :
celestial right ascension ~= 18 hours ( or 3 * PI / 2 radians, or 270 degrees )
celestial declination = PI /2 - epsilon
where epsilon ~ 23.5 degrees ( Earth's 'tilt' )
but has ecliptic co-ordinates:
ecliptic longitude = 0
ecliptic latitude = PI / 2 ( or plus 90 degrees .... )
Whereas the south ecliptic pole is somewhere nearby the Large Magellenic cloud with :
celestial right ascension ~= 6 hours ( or PI / 2 radians, or 90 degrees )
celestial declination = - PI /2 + epsilon
but has ecliptic co-ordinates:
ecliptic longitude = 0
ecliptic latitude = - PI / 2 ( or minus 90 degrees .... )
Both celestial right ascension and ecliptic longitude are measured as increasing to the east from the vernal/spring equinox ( called first point of Aries, but it's actually in Pisces due to another effect, hence the '~' when quoting some right ascensions ).
So I'll now focus on writing MATLAB code to process all the skygrid files with a view to :
- conversion to ecliptic co-ordinates from celestial
- sort the resultant point set file by ecliptic latitude ( most negative to most positive )
- with any points at equal ecliptic latitude are sorted further by increasing ecliptic longitude
The rough/hopeful idea here is that I can then 'unwrap the orange peel' ( sequence the point set ), in ecliptic co-ordinates now, from the south ecliptic pole upwards to the north in a spiral like fashion.
If anyone has followed me thus far ( ??phew!! ), one could deduce that while the original skygrid files have been 'politely' ordered by celestial declination ( and secondarily by celestial right ascension ) to make one spiral turn per celestial declination value - there is no guarantee that such an ordering will be suitably revealed in the ecliptic latitude ( and secondarily by ecliptic longitude ).
That is can one unpeel the orange easily by starting at an ecliptic pole? If so then : the cyclic sinusoidal behaviour already deduced will be reproduced, while the 'wiggles' will also come out as they are an artifact of the transform and the method of production of these discrete point sets/grids.
Time for a lie down .... :-)
Cheers, Mike.
( edit ) I'm not implying I can necessarily reproduce any given WU's actual grid traversal but reproduce the overall 'as if' behaviour.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
Hi! Hey, this will be fun!
)
Hi!
Hey, this will be fun! Developing the "next generation runtime predictor" (New: now with wiggles !!).
If you want to play around with RA-dec to ecliptic coordinate transformations, TOPCAT (once more :-) ) is a very useful tool.
I've got version 3.4 which allows to add new columns to a data table containing coordinates in a transformed coordinate system. Click on the "Display column metadata" icon in the main window after you've loaded data (e.g a skygrid file) and then click on "add new sky-coordinate columns". Choose "Input coordinates" and "output coordinates" and then new columns will be generated. You can save the resulting file so you'll get the skygrid in both coordinate systems.
Now it gets really interesting !!!
TOPCAT allows you to add computed columns (or "Synthetic column", use the same "column-metadata" window to do this), so ... hmmm..... let's add a column that computes the sine of the angle between the skypoint and the ecliptic plane.
While we are at it, create another synthetic column with the expression "$0" which will give us the index of the row in the dataset, so we can plot the skypoints in their original order.
So let's see what do we have now (all done just with TOPCAT!!):
"enriched" Skygrid table:
Column / meaning
RA RA in original coordinate system
DEC dec in original coordinate system
LONG "longitude" in ecliptic coordinates, computed by TOPCAT
LAT "latitude" in ecliptic coordinates, computed by TOPCAT
abs(sin(LAT)) sine of the angle between the skypoint and the ecliptic plane.
ind index of row => the sequence of traversing the sky in E@H
Now let's plot ind vs. abs(sin(LAT)) and you get .... (to be continued)
(edited for typos)
....we see ...wiggles
)
....we see ...wiggles :-)
(you see the skygrid file being traversed in it's original order from left to right (pole to pole)).
I might have said this before...I love TOPCAT :-).
CU
H-B
RE: Hi! Hey, this will be
)
Thanks heavens! Now there's two people that understand what I'm saying! :-):-)
We don't actually have to get/mimic the point traversal/sequence right for this to work. What would be sufficient is to establish how many points lie within each suitable interval of ecliptic latitude. As that number of points ( per latitude interval ) varies with latitude - from one ecliptic pole to the ecliptic equator to the other ecliptic pole - then that ought reveal the sinusoid/wiggles we seek.
Cheers, Mike.
( edit ) You're too quick for me - heck, slam dunk!! :-) :-) :-)
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
For a really excellent
)
For a really excellent prediction of runtime however, this is not enough. Our first approximation was that runtime depends mainly on declination and that causes some wiggles. We can make a better assumption by assuming that runtime depends on latitude in the ecliptic coordinate system, but that's not quite true as well.
Here are plots of runtime per skypoint in the original and ecliptic skycoordinates.
What we see are those two runtime minima that introduce a dependency on longitudes as well.
It's better seen in a 3D plot (runtime axis flipped to get better visibility):
This "Twin Peaks anomaly" is much harder to grasp with an explicit formula, and I wonder whether some 2D interpolation from experimental runtime data would be in order.
But one step after another...I guess "Twin Peaks" can wait.
CU
Bikeman
Ah, that's beautiful work
)
Ah, that's beautiful work right there!! :-)
Now, why there is some ecliptic longitude dependence is that the original files had within them series of points with increasing celestial right ascension BUT with precisely the same celestial declination - as the files were constructed to be so.
So the peeling of the orange in celestial co-ordinates is not a spiral in the perfect smooth sense, but 'straight' segments with steps between - the steps occurring between the last right ascension value in a series for a given declination AND the first right ascension value of a series for the next declination.
Under transform to ecliptic co-ordinates however, there are probably-not/unlikely-to-be such series of points with precisely the same ecliptic latitude in common, that then can be ordered by increase in ecliptic longitude. So you wouldn't get straights/steps nor a spiral.
Yeah, it's a head spin all right .... yeah I'll sleep on it.
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
RE: Now, why there is some
)
Hmmm... not sure I get this right but the "Twin Peaks" anomaly is completely independent of the skygrids. For those diagrams, I made up my own skygrids (500 of them) each with only one skypoint and then measured the runtime. So "Twin Peaks" is not a skygrid artifact, it's a "real" dependency of runtime. There is some astronomical meaning for the coordinates of those "peaks", iirc it's about the mean velocity vector of Earth during the observation time or some such.
But again, I guess the "Twin Peaks" should be dealt with after perfecting a simpler prediction scheme that takes into account the tilt between the celestrial and ecliptic coordinate system.
CU
Bikeman
RE: Hmmm... not sure I get
)
Sorry, I didn't mean the existence of the peaks ( note to others : these are actually runtime troughs ), but their exact position in ecliptic longitude. Or put another way, does any skygrid ( I'm thinking of the actual grid files now ) have any points precisely on the ecliptic equator, giving the lowest possible sine ( ie. zero )? For that matter does any sky grid ( when viewed in either celestial or ecliptic systems ) contain a point(s) which hit the equinoxes right on the mark? To be investigated ......
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
Hi! Just to provide some
)
Hi!
Just to provide some visual evidence to explain why the irregularities ("wiggles") in runtime are bigger in S5R5 than in S5R4, I made these two diagrams of result files, the first from S5R4, the second from S5R5, both for the 720 Hz skygrid
As was announced earlier, there are now fewer skypoints per result, but more work is done per skypoint.
Because the skypoints now cover a smaller interval of right ascension values, some runtime irregularities do not average out as nicely as they did in S5R4.
Nothing new, we already knew this, but still nice to get a visual check on this.
For a really accurate runtime model, we would have to give good estimates for the actually covered sky-area, not just the (mean) declination value. I guess there is no other way as to more or less exactly replicate the partitioning algorithm used by the client or otherwise small errors in the model will add up quickly if not done right.
CU
Bikeman
Ok, winding back to the
)
Ok, winding back to the grinding .... trawling through the original skygrid files ( ie. celestial co-ordinates ), seeing what's what .... :-)
I'll start with some simple results. For each frequency :
(A) Count the total number of grid points. Not surprisingly this is quadratic ( because we have two dimensions to play with and we contract the spacings in each equally, due to isotropy ).
which best fits to:
point count = 0.081662 * (frequency^2) + 0.12789 * frequency + 0.086998
Thus we have a more spotty sphere as the frequency goes up ie. a finer grid. Hmmmmmm .... a 'skygrid density constant' in there somewhere??? [ 'complete-the-square' on the quadratic ]
(B) Count the total number of distinct declination values.
which fits to:
declination count = 0.25328 * frequency + 0.48377
Although spiral/winding isn't quite right, more like slices produced by those wire egg slicers :
Meaning there are more declination values ( wires in the slicer ) from pole -> equator -> pole as frequency increases.
(C) Count the total number of right_ascension values for each declination value. Here's an example for 1000Hz :
Yeah, it's a sinusoid. The equator at declination zero has the most number of points around, lessening to either pole. Reminds me of an old joke about the difference between circumnavigating the Earth at the equator vs just walking around the flag at the South Pole.
Now to simultaneously plot all the frequency bands together:
Showing that the skygrid becomes more dense with frequency, and hence there are more right ascension values for each slice of declination - but retaining the sinusoidal variation from pole -> equator -> pole.
Now, none of this is much new, but I just thought I'd show you all how it works. :-)
So I move on to the ecliptic conversion and see if I can apply much the same analysis ..... will the wiggles be there?? Are they an artifact of the transform ...... stay tuned .... :-)
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal