The "cleanup" for the S5GC1HF run

I'm not sure what you mean by this. What exactly are you using to edit your state file? You need a plain text editor, preferably one that can handle regular expressions. You will find it easiest to construct a small script to make the whole process as automatic as possible. After you have been going for a while and you accumulate a wide range of blocks in your state file, it would be far too onerous to edit them out one at a time manually.

I'm using Linux for my hosts and the standard unix tools make it very easy to maintain the state file. I've never written any Windows batch files but if I was forced too, this is what I would do. Someone who knows how to write scripts for Windows would be able to do a much better job. What is below is just off the top of my head.

For me (with my limitations) I'd grab Notepad2 for any manual editing I needed to do. For script work, I'd use grep.exe, sed.exe, etc, so I could pretend I was using Unix :-). Google the names - I'm sure they exist. Then when I needed to remove all tags in the state file, I'd just run this (uncommented) script:-

grep.exe marked client_state.xml
pause
..\BOINC\boinccmd --quit
pause
ren client_state.xml client_state.bak
sed.exe /^.*marked.*/d client_state.bak > client_state.xml
del client_state.bak

Firstly, (assuming the script is being run from the BOINC_Data directory), grep is run to see if there are actually any tags in the file. The pause will allow you to bail out if there aren't.

Secondly, if you proceed from this point, boinccmd is used to shut boinc down. Be aware that it can take a few seconds for boinc to terminate and write out the state file. The second pause is simply to allow you to wait a few seconds for this to happen. I don't know how you could do this properly in Windows but hopefully there would be a way.

Thirdly, make a backup of the state file and run it through a stream editor (sed.exe) to search for and remove all lines that contain the given regular expression (the stuff between the slashes). The 'd' is the delete command. Write the output as a new state file. If you are happy that this will all work, you can delete the backup state file. If you wanted to be cautious you could omit that step until you had restarted boinc and all was well.

Finally, depending on how you normally launch boinc, you could include that at the end of the script to make the whole thing pretty automatic.

You will always get work for the existing frequency, or perhaps one above, if the existing is depleted (temporarily or otherwise), and if there are no current resends available. Your chances of getting a resend therefore very much depend on having as wide a range as possible of old frequencies in your project directory and documented by blocks in your state file. If you can double the old frequencies you have saved you will double your chances of getting a resend when you ask. You must take control of when you ask (using NNT). If you wait say around 4-8 hours before allowing a work request, there is more chance that some may have accumulated in the interim.

However the two most important things you should do are these:-

1. Use the technique I listed in my previous response to seed your state file with only the highest frequency you currently have. You can achieve this by temporarily turning off all the lower ones using manually inserted tags - just copy and paste an existing one in the appropriate places. This will get you a substantial number of new contiguous LIGO files and blocks.

2. Progressively expand the size of your work cache. You need one that's several times larger than your current one. The main reason for this is that when you start getting resends for older data, you need those tasks to stick around for several days so that the LIGO files wont be deleted if/when they become . If it's 6 or 8 days before these tasks get to the top of the queue, there is much more opportunity to get more of them and so lessen the risk of having to manually insert the LIGO data into the project directory and, more importantly, the blocks back into your state file. When properly set up with enough resends in your cache to properly protect your full range of data, it's basically a 10 second operation to run a script to remove (potentially hundreds of) tags immediately prior to doing a work fetch run.

Before you attempt to make a big increase in your work cache, take the opportunity as described above to expand your range of saved frequencies. Modifying your state file with temporary tags is a bit tedious so you should take the opportunity to do it easily with your second host - see below.

You really should take the opportunity of this second host to get a wider range of data for your first host's current frequency and not a totally different frequency. Here's how to do it.

• * Set NNT on the AMD and let the current tasks complete and be reported.
  * Stop BOINC and open the state file for editing with a good text editor.
  * Remove every single block for both LIGO data and skygrid files and remove the files themselves from the project directory.
  * Seed the state file with 5 blocks (4 LIGO files and 1 skygrid file) for the topmost frequency you currently have on your primary host and then save and close.
  * Place copies of the 4 LIGO files and 1 skygrid file (for this same frequency) in your project directory.
  * Restart BOINC, set your cache settings in the manager to 0.0/0.1 and then unset NNT.
  

You should get a single task for the particular frequency you have chosen and a total of 44 LIGO files and blocks for the 11 frequency bands immediately above your chosen frequency. Your intention is to harvest all of these for use on your primary host. You would be wasting the opportunity to get the maximum benefit if you didn't start with the topmost frequency that you currently had. Unless you wanted to, there is no need to keep running the noisy AMD once it had finished that single task that it got. Just set NNT to prevent more tasks.

Once you have seeded the project directory on your primary host with the new LIGO data you can make a step increase in your cache there. There is no need to seed the state file with the new blocks you have harvested - just save them somewhere for future use. Increase the "extra days" setting by say 0.5 - 0.8 days. That should get you a number of tasks that will probably cover several frequency bands. If you have other projects that you don't want to disturb, make sure you set NNT for them until you eventually revert to your usual cache size. Your life will be more difficult and you'll need to think carefully about each move if you don't want to disturb other projects. My comments from here on will assume that you don't have the complications of multiple active projects

If you got say 10 new E@H tasks covering say 4 higher frequency bands, BOINC will notice that the 16 new LIGO files are already in place and will automatically insert the blocks in your state file and will issue "download skipped" messages for each LIGO file you already have. The next time around you can make a bigger cache size increment, say 1.0 - 1.5 extra days which will give a larger number of tasks and the opportunity to get more blocks into your state file. Eventually you could even start drawing completely new frequencies above what you had harvested on the AMD. When you have a nice total cache (say 6 - 8 days) you can set NNT and run a script to get rid of all delete tags you might have accumulated along the way. That only takes a few seconds to run and afterwards you should glance through the state file and the project directory to make sure there are no "holes" in your continuous data range. If there are you should fill them from what you've saved.

You should wait a decent time as mentioned previously, and then you can unset NNT. If you had waited 8 hours, BOINC will try to get 0.33 days work unless you modify your cache settings at the same time. The search for new tasks on the server will start at your lowest advertised frequency and will move progressively upwards through the bands collecting any available resends. If there were plenty, your entire work request could be filled with resends. If there were none, you will get only primary tasks. In either case you will get plenty of "server request to delete file ..." messages. So you then just set NNT, run your script to remove the tags, fill any gaps, and wait another 8 hours or so and repeat the process.

I realise that this is all quite complicated. There are further subtleties that I haven't even touched on so you have to be prepared to work things out as you go. If you have any specific questions, please ask.

Latest update on the vacuum cleaners.

As previously discussed, I have 4 quads working the 1430Hz - 1440Hz frequency range and existing on a diet of essentially resends only. They have 10 day work caches and I've been able to keep them full - well up to now at least. There are certainly signs that resends for this frequency range are getting more difficult to find now but, on past experience, that will change when the new run kicks in and many hosts currently sucking up everything in sight get shifted away from the old run. At the moment I would guess the scheduler is sending resends plus the necessary LIGO data to many hosts needing a frequency change. When a host has transitioned to the new run, it will get tasks there and not be given old run resends unless it continues to "advertise" old run data availability.

I have quite a number of other hosts that are not specifically looking for resends. These hosts have been chewing on the remaining primary tasks at the very upper end of this same frequency range (which now extends through to 1440.60Hz). There are extremely few primary tasks left (for that range) and I was interested to note that there are no primary tasks (seemingly) just above 1439.95Hz and into the lower part of the 1440.xx range. A task at 1439.95 requires data from 1439.95 to 1440.55. A day ago I wasn't having any trouble getting such tasks, including 1440.00 tasks (which is why I have data up to 1440.60. This morning, the available sequence numbers for 1439.95 tasks were around _15 and diminishing rapidly. The remaining few tasks would have disappeared in a very short time after that.

For two machines, in trying to get tasks of 1440.00 and above, there were server delete requests for all bands from 1440.00 to my upper limit (1440.60). These machines are now pulling new tasks in the 149x.xx range, a planned change where there are still available tasks (sequence#s around _500 or thereabouts). So I'll need to shift all the rest of my hosts in the next few days - well before their caches run dry. A good job for the Easter break.

As previously described, it's quite simple to set up the transition. Just drop the LIGO files into the project directory and drop (at least a subset of) the blocks into the state file. The thing that takes the most time is a couple of controlled work request cycles to top up the work cache whilst observing that the correct frequency range is being used for the new tasks. It's all going smoothly so far.

In case anybody is wondering why I want all hosts feeding in the same frequency range, it's to do with the bandwidth used in downloading all the different LIGO data for randomly selected frequency ranges. A few months ago, I was using 250+ GB a month. Since Christmas, I've been using less than 30GB per month.

The volume of downloads dramatically increases as the end of a run approaches (eg like right now). This is because the scheduler has to shift clients to different frequency bands much more frequently as the small number of remaining primary tasks rapidly vanishes. As explained above, when the bulk of hosts has transitioned to the new run, those hosts specifically set up to advertise a specific old frequency range will have the field to themselves. I've done this at the end of past runs and kept a few hosts going on old data up to a month or more after a run transition occurs. I expect to be able to do the same again this time. All I have to do is keep enough work on hand to last a few days into the new run. At that time, the bulk of my hosts will be allowed to transition to the new run and the number left will depend on resend availability. That number may even be more than the current 4 doing resends.

Here's a short one for a change :-)

The 1430Hz to 1440Hz range is completely devoid of primary tasks (and has been that way for a day or two now) but continues to supply lots of resends. The 4 quads working in this range continue to be able to maintain full (or overfull) caches essentially on resends alone. I have set up the 1490.xxHz frequency set on these 4 machines so that if/when they ask for work and a 1430-1440 resend is not available, they will get new tasks from the 1490.xxHz and above range. i've tested this and it works as expected.