Fermi LAT Gamma-ray pulsar search "FGRP2" - shorter tasks

Hi Bernd,

I notice we have now gone above the LATeah0030 you mentioned, so I guess there must be more needed than you originally thought. Can you please give an indication of the new upper limit for these "_RERUN" jobs?

As you may recall, we had a discussion about the file sizes of all the skygrids - particularly the very large ones at the time that LATeah0023 and LATeah0024 data was being crunched initially. Recently, there have been a lot of "_RERUN" tasks for these. Those started tapering off a little some days ago and I'm now seeing reruns for the range 0026 to 0029 as well. I thought that rerun tasks might be just about finished. However, this morning, I'm seeing reruns for 0031 and 0032 so I'm interested to find out how many more of these there are and where the new upper limit for LATeah00xx might be?

I should hasten to add that I'm in no way concerned by any of this. I'm also not in need of any "sticky" files mechanism as was mentioned in the previous discussion. These days, I'm using a rather involved shell script that ensures that all participating hosts always have a full complement of the files they are likely to need. This script automates the saving and deploying of all such files using rsync. It also allows for files to be deleted from the cache once I think they are finished with. It's just that I didn't know that files above 0030 were going to be 'needed' again :-). I'd kept them all from when those were first issued but hadn't been redeploying them. I've hastily added them back into the mix now :-).

I've added a few bells and whistles to this script which allow me to do a number of things from managing the cached files to recording useful stats (for me anyway) of exactly what is happening on each host that participates in the scheme. I find it very useful to have these hosts do all their downloading in my ISP's 'off-peak' period where I have a much larger allowance. Each morning I can review the log to see what new data files were downloaded, how many potential downloads were skipped because the files were already available, how many tasks each host requested, etc.

As an example, I've included below, a copy of this morning's log in full. The 'off-peak' time starts at 1.00am local time, so you will notice that each host fills its work cache sometime after 1.00am. There are 20 hosts in this little group. Each host in turn syncs itself with the cache and then makes a request for new work. If any one host downloads a 'new' data file, it will be pushed back to the cache and so be immediately available to all subsequent hosts. That way, any such file is only ever downloaded once. Notice that host 192.168.0.116 downloaded the new file skygrid_LATeah0045U_0720.0.dat and that host 192.168.0.125 (2 hosts later) subsequently acquired this file from the cache and so was able to skip the download - as did quite a few other hosts later on as well. However the thing that really caught my attention was host 192.168.0.162 downloading file skygrid_LATeah0032U_0304.0.dat. "I thought these were supposed to finish at 0030, what the @#$%$#@ is going on ...", sez he, muttering to himself ... :-).

[pre]New run to get more tasks for hosts listed below -- started at Sat Oct 26 01:13:04 EST 2013.
Options - cleanstr=, datastr=*LATeah*, Days=5.2, Pause=380, Hosts=tchosts, Verbose=yes
===========================================================================================
TIME IP ADDRESS NEW TASKS STATS OR FILES DOWNLOADED / SKIPPED
---- ---------- ---------------------------------------------
01:13:04: 192.168.0.112 --> 17 new tasks received made up of 5 FGRP2 and 12 BRP5 (PAS)
--> No new data files downloaded but 1 was skipped.
--> skipped skygrid_LATeah0045U_0688.0.dat

01:20:14: 192.168.0.116 --> 14 new tasks received made up of 4 FGRP2 and 10 BRP5 (PAS)
--> 1 new data file(s) downloaded and 1 skipped.
--> dloaded skygrid_LATeah0045U_0720.0.dat
--> skipped skygrid_LATeah0045U_0688.0.dat

01:27:25: 192.168.0.117 --> 16 new tasks received made up of 3 FGRP2 and 13 BRP5 (PAS)
--> No new data files downloaded but 1 was skipped.
--> skipped skygrid_LATeah0045U_0688.0.dat

01:34:37: 192.168.0.125 --> 20 new tasks received made up of 10 FGRP2 and 10 BRP5 (PAS)
--> 2 new data file(s) downloaded and 3 skipped.
--> dloaded skygrid_LATeah0044U_0336.0.dat
--> dloaded skygrid_LATeah0045U_0656.0.dat
--> skipped skygrid_LATeah0045U_0688.0.dat
--> skipped LATeah0044U.dat
--> skipped skygrid_LATeah0045U_0720.0.dat

01:41:49: 192.168.0.155 --> 22 new tasks received made up of 8 FGRP2 and 14 BRP5 (PAS)
--> No new data files downloaded but 5 were skipped.
--> skipped skygrid_LATeah0045U_0688.0.dat
--> skipped skygrid_LATeah0045U_0720.0.dat
--> skipped skygrid_LATeah0024U_1136.0.dat
--> skipped skygrid_LATeah0045U_0656.0.dat
--> skipped skygrid_LATeah0045U_0592.0.dat

01:48:57: 192.168.0.162 --> 27 new tasks received made up of 4 FGRP2 and 23 BRP5 (PAS)
--> 1 new data file(s) downloaded and 5 skipped.
--> dloaded skygrid_LATeah0032U_0304.0.dat
--> skipped skygrid_LATeah0045U_0688.0.dat
--> skipped LATeah0032U.dat
--> skipped LATeah0029U.dat
--> skipped skygrid_LATeah0029U_0912.0.dat
--> skipped skygrid_LATeah0045U_0656.0.dat

01:56:17: 192.168.0.168 --> 8 new tasks received made up of 8 FGRP2 and 0 BRP5 (PAS)
--> 1 new data file(s) downloaded and 5 skipped.
--> dloaded skygrid_LATeah0032U_0624.0.dat
--> skipped skygrid_LATeah0045U_0688.0.dat
--> skipped skygrid_LATeah0030U_1392.0.dat
--> skipped skygrid_LATeah0030U_1360.0.dat
--> skipped LATeah0032U.dat
--> skipped skygrid_LATeah0030U_1424.0.dat

02:03:16: 192.168.0.183 --> 14 new tasks received made up of 5 FGRP2 and 9 BRP5 (PAS)
--> No new data files downloaded but 2 were skipped.
--> skipped skygrid_LATeah0045U_0688.0.dat
--> skipped skygrid_LATeah0024U_1136.0.dat

02:10:22: 192.168.0.184 --> 13 new tasks received made up of 5 FGRP2 and 8 BRP5 (PAS)
--> No new data files downloaded but 3 were skipped.
--> skipped skygrid_LATeah0045U_0688.0.dat
--> skipped skygrid_LATeah0024U_1040.0.dat
--> skipped skygrid_LATeah0045U_0656.0.dat

02:17:31: 192.168.0.186 --> 13 new tasks received made up of 3 FGRP2 and 10 BRP5 (PAS)
--> No new data files downloaded but 1 was skipped.
--> skipped skygrid_LATeah0045U_0688.0.dat

02:24:40: 192.168.0.187 --> 5 new tasks received made up of 5 FGRP2 and 0 BRP5 (PAS)
--> No new data files downloaded but 2 were skipped.
--> skipped skygrid_LATeah0045U_0656.0.dat
--> skipped skygrid_LATeah0045U_0688.0.dat

02:32:02: 192.168.0.188 --> 12 new tasks received made up of 4 FGRP2 and 8 BRP5 (PAS)
--> 2 new data file(s) downloaded and 2 skipped.
--> dloaded skygrid_LATeah0032U_0784.0.dat
--> dloaded skygrid_LATeah0032U_0720.0.dat
--> skipped skygrid_LATeah0045U_0688.0.dat
--> skipped LATeah0032U.dat

02:39:05: 192.168.0.189 --> 14 new tasks received made up of 5 FGRP2 and 9 BRP5 (PAS)
--> No new data files downloaded but 3 were skipped.
--> skipped skygrid_LATeah0045U_0688.0.dat
--> skipped skygrid_LATeah0024U_0048.0.dat
--> skipped skygrid_LATeah0024U_1136.0.dat

02:46:14: 192.168.0.190 --> 35 new tasks received made up of 19 FGRP2 and 16 BRP5 (PAS)
--> No new data files downloaded but 5 were skipped.
--> skipped LATeah0044U.dat
--> skipped skygrid_LATeah0044U_0432.0.dat
--> skipped skygrid_LATeah0045U_0688.0.dat
--> skipped skygrid_LATeah0027U_1104.0.dat
--> skipped skygrid_LATeah0044U_0656.0.dat

02:53:17: 192.168.0.191 --> 16 new tasks received made up of 6 FGRP2 and 10 BRP5 (PAS)
--> No new data files downloaded but 3 were skipped.
--> skipped skygrid_LATeah0024U_1168.0.dat
--> skipped skygrid_LATeah0045U_0688.0.dat
--> skipped skygrid_LATeah0045U_0656.0.dat

03:00:22: 192.168.0.192 --> 11 new tasks received made up of 4 FGRP2 and 7 BRP5 (PAS)
--> 1 new data file(s) downloaded and 3 skipped.
--> dloaded skygrid_LATeah0032U_0400.0.dat
--> skipped LATeah0032U.dat
--> skipped skygrid_LATeah0045U_0624.0.dat
--> skipped skygrid_LATeah0045U_0656.0.dat

03:07:24: 192.168.0.193 --> 21 new tasks received made up of 10 FGRP2 and 11 BRP5 (PAS)
--> No new data files downloaded but 4 were skipped.
--> skipped skygrid_LATeah0045U_0720.0.dat
--> skipped skygrid_LATeah0045U_0688.0.dat
--> skipped LATeah0031U.dat
--> skipped skygrid_LATeah0031U_0816.0.dat

03:14:26: 192.168.0.196 --> 23 new tasks received made up of 8 FGRP2 and 15 BRP5 (PAS)
--> No new data files downloaded but 2 were skipped.
--> skipped skygrid_LATeah0045U_0656.0.dat
--> skipped skygrid_LATeah0045U_0688.0.dat

03:21:26: 192.168.0.197 --> 19 new tasks received made up of 8 FGRP2 and 11 BRP5 (PAS)
--> No new data files downloaded but 2 were skipped.
--> skipped skygrid_LATeah0045U_0688.0.dat
--> skipped skygrid_LATeah0045U_0656.0.dat

03:28:26: 192.168.0.199 --> 12 new tasks received made up of 4 FGRP2 and 8 BRP5 (PAS)
--> No new data files downloaded but 2 were skipped.
--> skipped skygrid_LATeah0045U_0656.0.dat
--> skipped skygrid_LATeah0045U_0720.0.dat

===========================================================================================

Total of 332 new tasks overall - 128 FGRP2 tasks and 204 BRP5 (PAS) - for 20 active hosts.
A total of 8 new data files were downloaded and 55 potential downloads were skipped.
Run finished for 20 active host(s) and 0 skipped host(s) on Sat Oct 26 03:35:26 EST 2013.
###########################################################################################[/pre]

One peculiar thing I've noticed is that some of the above hosts actually don't report fully all the skipped downloads. I haven't investigated it thoroughly but it seems that hosts running a V7 BOINC don't fully record in stdoutdae.txt (which is what I'm using to extract info about what gets downloaded and what gets skipped) the details of every file whose download was skipped. It doesn't affect anything and I'm not bothered by it. It just means that there were probably even more than the total of 55 skipped downloads reported above. Over the last couple of weeks I seem to be avoiding the download of between 50 and 100 of these files every day so I consider it well worth the effort of working out a plan of how to do it and then writing a script that actually worked correctly :-).