Skygrid file analysis

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6534
Credit: 284700169
RAC: 113726
Topic 194172

I thought I'd fork of this line of thinking from 'S5R5 Performance Analysis' as a distinct thread.

I think I've been able to obtain all the skygrid files for S5R5, from 60Hz through to 1300Hz inclusive. If anyone has, or knows of, any outside this range then please PM me.

Quote:

I found these scattered amongst various non-self-evident [ don't ask! :-):-) ] directory structures from the root at

http://einstein.phys.uwm.edu/download/

each file being time stamped ~ 20-Nov-2008 21:43

My client_state.xml file has the following example entry which contains hints that suggested this location ( plus mirror sites ):
[pre]
skygrid_0550Hz_S5R5.dat
175984.000000
0.000000
7a3ed62280da49c24574b14fb3b45ace
1

http://einstein.ligo.caltech.edu/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.phys.uwm.edu/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.aei.mpg.de/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.astro.gla.ac.uk/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.ligo.caltech.edu/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.phys.uwm.edu/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.aei.mpg.de/download/fd/skygrid_0550Hz_S5R5.dat
http://einstein.astro.gla.ac.uk/download/fd/skygrid_0550Hz_S5R5.dat
[/pre]Yes, it did take a while to round them all up. :-)


I've uploaded these to my website, 125 in all to date. Anyone is welcome to download any file of interest by invoking, say with your browser, the web address:

http://downloads.mikehewson.id.au/skygrid/zips/skygrid_XXXXHz_S5R5.dat.zip

where XXXX runs from 0060 to 1300 ( note the 4 digit positions with left sided zero padding ), in steps of 10 ( ie. 0060 0070 0080 ..... 1280 1290 1300 ).

Quote:
They are the same compressed files that BOINC downloads by the names 'skygrid_XXXXHz_S5R5.dat', I've just added the '.zip' ending for the sake of file association in Windows.


Each will unzip to produce a non-compressed ASCII file by the name of 'skygrid_XXXXHz_S5R5.dat'.

Quote:

Thus, for instance :

http://downloads.mikehewson.id.au/skygrid/zips/skygrid_1110Hz_S5R5.dat.zip

can be downloaded and un-zips to

skygrid_1110Hz_S5R5.dat


The files have one sky position/point per line, each containing two ASCII strings separated by whitespace representing the numerical entries right_ascension and declination ( ie. on celestial sphere ) in radian measure. The file sizes are from ~ 10KB @ 60Hz to ~ 4.5MB @ 1300Hz.

Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6534
Credit: 284700169
RAC: 113726

Skygrid file analysis

Now, what to do with the files? For good background I might suggest reading this very helpful post of Bernd's. I'll quote a key part ( my emphasis in bold ):

Quote:
The time needed for the Fstat part is constant for every template, i.e. independent of the actual sky position and frequency. This isn't true for the Hough step. The size of a fundamental data structure in the Hough part depends on the declination of the current sky position (actually the (abs() or sqr() - I don't remember from the top of my head) sine of the angle between the sky position and the ecliptic plane), and so does the time of every computation that involves it (This is very much simplified, but if you neglect everything that averages out over a task even in S5R3, it boils down to this).


Note the sky-grid files are in celestial co-ordinates, but the interesting behaviour for our work units is in ecliptic co-ordinates. Bikeman has already noted that the peaks of WU times are at/near the ecliptic poles, with notable minima at/near zero ecliptic latitude.

What this means is that one can project a line that goes from the south pole of the Sun through to it's north pole, right out to the celestial sphere.

Quote:
The celestial sphere has north and south poles projected from Earth's north and south poles respectively, and an equator similiarly concentric with Earth's equator.


The north ecliptic pole is somewhere in Draco with :

celestial right ascension ~= 18 hours ( or 3 * PI / 2 radians, or 270 degrees )
celestial declination = PI /2 - epsilon

where epsilon ~ 23.5 degrees ( Earth's 'tilt' )

but has ecliptic co-ordinates:

ecliptic longitude = 0
ecliptic latitude = PI / 2 ( or plus 90 degrees .... )

Whereas the south ecliptic pole is somewhere nearby the Large Magellenic cloud with :

celestial right ascension ~= 6 hours ( or PI / 2 radians, or 90 degrees )
celestial declination = - PI /2 + epsilon

but has ecliptic co-ordinates:

ecliptic longitude = 0
ecliptic latitude = - PI / 2 ( or minus 90 degrees .... )

Both celestial right ascension and ecliptic longitude are measured as increasing to the east from the vernal/spring equinox ( called first point of Aries, but it's actually in Pisces due to another effect, hence the '~' when quoting some right ascensions ).

Quote:

For the math's masochists the transform is as follows:
D = celestial declination
A = celestial right ascension
B = ecliptic latitude
L = ecliptic longitude
e = obliquity of ecliptic ~ 0.4091 radians ( Earth's 'tilt' )[pre]sin(B) = sin(D)cos(e) - cos(D)sin(A)sin(e) (1)
cos(L)cos(B) = cos(A)cos(D) (2)[/pre]Leading to the algorithm

- given D and A
- determine sin(B) using (1)
- obtain B
- obtain cos(B)
- determine cos(L) using (2)
- obtain L


So I'll now focus on writing MATLAB code to process all the skygrid files with a view to :

- conversion to ecliptic co-ordinates from celestial
- sort the resultant point set file by ecliptic latitude ( most negative to most positive )
- with any points at equal ecliptic latitude are sorted further by increasing ecliptic longitude

The rough/hopeful idea here is that I can then 'unwrap the orange peel' ( sequence the point set ), in ecliptic co-ordinates now, from the south ecliptic pole upwards to the north in a spiral like fashion.

If anyone has followed me thus far ( ??phew!! ), one could deduce that while the original skygrid files have been 'politely' ordered by celestial declination ( and secondarily by celestial right ascension ) to make one spiral turn per celestial declination value - there is no guarantee that such an ordering will be suitably revealed in the ecliptic latitude ( and secondarily by ecliptic longitude ).

That is can one unpeel the orange easily by starting at an ecliptic pole? If so then : the cyclic sinusoidal behaviour already deduced will be reproduced, while the 'wiggles' will also come out as they are an artifact of the transform and the method of production of these discrete point sets/grids.

Time for a lie down .... :-)

Cheers, Mike.

( edit ) I'm not implying I can necessarily reproduce any given WU's actual grid traversal but reproduce the overall 'as if' behaviour.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3522
Credit: 686042913
RAC: 589795

Hi! Hey, this will be fun!

Hi!

Hey, this will be fun! Developing the "next generation runtime predictor" (New: now with wiggles !!).

If you want to play around with RA-dec to ecliptic coordinate transformations, TOPCAT (once more :-) ) is a very useful tool.

I've got version 3.4 which allows to add new columns to a data table containing coordinates in a transformed coordinate system. Click on the "Display column metadata" icon in the main window after you've loaded data (e.g a skygrid file) and then click on "add new sky-coordinate columns". Choose "Input coordinates" and "output coordinates" and then new columns will be generated. You can save the resulting file so you'll get the skygrid in both coordinate systems.

Now it gets really interesting !!!

TOPCAT allows you to add computed columns (or "Synthetic column", use the same "column-metadata" window to do this), so ... hmmm..... let's add a column that computes the sine of the angle between the skypoint and the ecliptic plane.

While we are at it, create another synthetic column with the expression "$0" which will give us the index of the row in the dataset, so we can plot the skypoints in their original order.

So let's see what do we have now (all done just with TOPCAT!!):

"enriched" Skygrid table:

Column / meaning
RA RA in original coordinate system
DEC dec in original coordinate system
LONG "longitude" in ecliptic coordinates, computed by TOPCAT
LAT "latitude" in ecliptic coordinates, computed by TOPCAT
abs(sin(LAT)) sine of the angle between the skypoint and the ecliptic plane.
ind index of row => the sequence of traversing the sky in E@H

Now let's plot ind vs. abs(sin(LAT)) and you get .... (to be continued)

(edited for typos)

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3522
Credit: 686042913
RAC: 589795

....we see ...wiggles

....we see ...wiggles :-)

(you see the skygrid file being traversed in it's original order from left to right (pole to pole)).

I might have said this before...I love TOPCAT :-).

CU
H-B

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6534
Credit: 284700169
RAC: 113726

RE: Hi! Hey, this will be

Message 90342 in response to message 90340

Quote:

Hi!

Hey, this will be fun! Developing the "next generation runtime predictor" (New: now with wiggles !!).

If you want to play around with RA-dec to ecliptic coordinate transformations, TOPCAT (once more :-) ) is a very useful too.

I've got version 3.4 which allows to add new columns to a data table containing coordinates in a transformed coordinate system. Click on the "Display column metadata" icon in the main window after you've loaded data (e.g a skygrid file) and then clock on "add new sky-coordinate columns". Choose "Input coordinates" and "output coordinates" and then new columns will be generated. You can save the resulting file so you'll get the skygrid in both coordinate systems.

Now it get's really interesting !!!

TOPCAT allows you to add computed columns (or "Synthetic column", use the same "column-metadata" window to do this), so ... hmmm..... let's add a column that computes the sine of the angle between the skypoint and the ecliptic plane.

While we are at it, create another synthetic column with the expression "$0" which will give us the index of the row in the dataset, so we can plot the skypoints in their original order.

So let's see what do we have now (all done just with TOPCAT!!):

"enriched" Skygrid table:

Column / meaning
RA RA in original coordinate system
DEC dec in original coordinate system
LONG "longitude" in ecliptic coordinates, COputed by TOPCAT
LAT "latitude" in ecliptic coordinates, computed by TOPCAT
abs(sin(LAT)) sine of the angle between the skypoint and the ecliptic plane.
ind index of row => the sequence of traversing the sky in E@H

Now let's plot ind vs. abs(sin(LAT)) and you get .... (to be continued)


Thanks heavens! Now there's two people that understand what I'm saying! :-):-)

We don't actually have to get/mimic the point traversal/sequence right for this to work. What would be sufficient is to establish how many points lie within each suitable interval of ecliptic latitude. As that number of points ( per latitude interval ) varies with latitude - from one ecliptic pole to the ecliptic equator to the other ecliptic pole - then that ought reveal the sinusoid/wiggles we seek.

Cheers, Mike.

( edit ) You're too quick for me - heck, slam dunk!! :-) :-) :-)

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3522
Credit: 686042913
RAC: 589795

For a really excellent

For a really excellent prediction of runtime however, this is not enough. Our first approximation was that runtime depends mainly on declination and that causes some wiggles. We can make a better assumption by assuming that runtime depends on latitude in the ecliptic coordinate system, but that's not quite true as well.

Here are plots of runtime per skypoint in the original and ecliptic skycoordinates.

What we see are those two runtime minima that introduce a dependency on longitudes as well.

It's better seen in a 3D plot (runtime axis flipped to get better visibility):

This "Twin Peaks anomaly" is much harder to grasp with an explicit formula, and I wonder whether some 2D interpolation from experimental runtime data would be in order.

But one step after another...I guess "Twin Peaks" can wait.

CU
Bikeman

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6534
Credit: 284700169
RAC: 113726

Ah, that's beautiful work

Message 90344 in response to message 90343

Ah, that's beautiful work right there!! :-)

Now, why there is some ecliptic longitude dependence is that the original files had within them series of points with increasing celestial right ascension BUT with precisely the same celestial declination - as the files were constructed to be so.

So the peeling of the orange in celestial co-ordinates is not a spiral in the perfect smooth sense, but 'straight' segments with steps between - the steps occurring between the last right ascension value in a series for a given declination AND the first right ascension value of a series for the next declination.

Under transform to ecliptic co-ordinates however, there are probably-not/unlikely-to-be such series of points with precisely the same ecliptic latitude in common, that then can be ordered by increase in ecliptic longitude. So you wouldn't get straights/steps nor a spiral.

Quote:

[musing on the side]( In my mind's eye ) I foresee ecliptic latitude bands which can be ordered ( pole -> equator -> pole ), with increasing ecliptic longitudes within such bands. The variation of wiggles thus would depend on the choice of ecliptic latitude band width ...... OR ...... alternately some selection criteria/algorithm for jumping from one point to the next ( but always increasing in ecliptic longitude ), starting from a pole and winding/peeling around and around, never missing a point ......

Yes, jump by taking the next - not yet selected/sequenced - point which has
- a no-lesser ( modulo 360 degrees ) ecliptic longitude in some already chosen winding sense, BUT
- an ecliptic latitude closer to the pole from which you started on.
[/musing on the side]

Yeah, it's a head spin all right .... yeah I'll sleep on it.

Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3522
Credit: 686042913
RAC: 589795

RE: Now, why there is some

Message 90345 in response to message 90344

Quote:

Now, why there is some ecliptic longitude dependence is that the original files had within them series of points with increasing celestial right ascension BUT with precisely the same celestial declination - as the files were constructed to be so.

Hmmm... not sure I get this right but the "Twin Peaks" anomaly is completely independent of the skygrids. For those diagrams, I made up my own skygrids (500 of them) each with only one skypoint and then measured the runtime. So "Twin Peaks" is not a skygrid artifact, it's a "real" dependency of runtime. There is some astronomical meaning for the coordinates of those "peaks", iirc it's about the mean velocity vector of Earth during the observation time or some such.

But again, I guess the "Twin Peaks" should be dealt with after perfecting a simpler prediction scheme that takes into account the tilt between the celestrial and ecliptic coordinate system.

CU
Bikeman

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6534
Credit: 284700169
RAC: 113726

RE: Hmmm... not sure I get

Message 90346 in response to message 90345

Quote:

Hmmm... not sure I get this right but the "Twin Peaks" anomaly is completely independent of the skygrids. For those diagrams, I made up my own skygrids (500 of them) each with only one skypoint and then measured the runtime. So "Twin Peaks" is not a skygrid artifact, it's a "real" dependency of runtime. There is some astronomical meaning for the coordinates of those "peaks", iirc it's about the mean velocity vector of Earth during the observation time or some such.

But again, I guess the "Twin Peaks" should be dealt with after perfecting a simpler prediction scheme that takes into account the tilt between the celestrial and ecliptic coordinate system.


Sorry, I didn't mean the existence of the peaks ( note to others : these are actually runtime troughs ), but their exact position in ecliptic longitude. Or put another way, does any skygrid ( I'm thinking of the actual grid files now ) have any points precisely on the ecliptic equator, giving the lowest possible sine ( ie. zero )? For that matter does any sky grid ( when viewed in either celestial or ecliptic systems ) contain a point(s) which hit the equinoxes right on the mark? To be investigated ......

Quote:
[aside]
The equinoxes are the only two points that don't change under celestial ecliptic transform. They have identical values for declination as latitude, and for right-ascension as longitude. So the vernal is [0,0] and the autumnal [0,PI] either way ( radian measure ). You can envision a line running through both equinoxes as being an axle around which the co-ordinates transform/rotate by the Earth's 'tilt'.
[/aside]


Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3522
Credit: 686042913
RAC: 589795

Hi! Just to provide some

Hi!

Just to provide some visual evidence to explain why the irregularities ("wiggles") in runtime are bigger in S5R5 than in S5R4, I made these two diagrams of result files, the first from S5R4, the second from S5R5, both for the 720 Hz skygrid

As was announced earlier, there are now fewer skypoints per result, but more work is done per skypoint.

Because the skypoints now cover a smaller interval of right ascension values, some runtime irregularities do not average out as nicely as they did in S5R4.

Nothing new, we already knew this, but still nice to get a visual check on this.

For a really accurate runtime model, we would have to give good estimates for the actually covered sky-area, not just the (mean) declination value. I guess there is no other way as to more or less exactly replicate the partitioning algorithm used by the client or otherwise small errors in the model will add up quickly if not done right.

CU
Bikeman

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6534
Credit: 284700169
RAC: 113726

Ok, winding back to the

Ok, winding back to the grinding .... trawling through the original skygrid files ( ie. celestial co-ordinates ), seeing what's what .... :-)

I'll start with some simple results. For each frequency :

(A) Count the total number of grid points. Not surprisingly this is quadratic ( because we have two dimensions to play with and we contract the spacings in each equally, due to isotropy ).

which best fits to:

point count = 0.081662 * (frequency^2) + 0.12789 * frequency + 0.086998

Thus we have a more spotty sphere as the frequency goes up ie. a finer grid. Hmmmmmm .... a 'skygrid density constant' in there somewhere??? [ 'complete-the-square' on the quadratic ]

(B) Count the total number of distinct declination values.

which fits to:

declination count = 0.25328 * frequency + 0.48377

Although spiral/winding isn't quite right, more like slices produced by those wire egg slicers :

Meaning there are more declination values ( wires in the slicer ) from pole -> equator -> pole as frequency increases.

(C) Count the total number of right_ascension values for each declination value. Here's an example for 1000Hz :

Yeah, it's a sinusoid. The equator at declination zero has the most number of points around, lessening to either pole. Reminds me of an old joke about the difference between circumnavigating the Earth at the equator vs just walking around the flag at the South Pole.

Now to simultaneously plot all the frequency bands together:

Showing that the skygrid becomes more dense with frequency, and hence there are more right ascension values for each slice of declination - but retaining the sinusoidal variation from pole -> equator -> pole.

Now, none of this is much new, but I just thought I'd show you all how it works. :-)

So I move on to the ecliptic conversion and see if I can apply much the same analysis ..... will the wiggles be there?? Are they an artifact of the transform ...... stay tuned .... :-)

Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.