possible Einstein@Home (BRP) cut-backs in January

We're currently using the cooling system for the current Atlas cluster that has been designed to cool its extension, you haven't seen that yet. The rented cooling you saw (IIRC) was really a temporary solution. And even the current intermediate solution doesn't work as planned. I think roughly every 5 weeks on average we had an unscheduled outage of the cluster due to cooling issues, many of these causing hardware losses (mostly HDs and power supplies).

Einstein@Home shouldn't be completely off-line at all.

The server migration is a bit tricky to handle under full load, but we have already done this before without anyone noticing at all (did anyone miss any URLs referring to einstein-abp1?). What you may notice is just some daemons on the server status page changing from "einstein-wug" to "einstein4", and being disabled or completely de-configured (i.e. vanished) occasionally in transition. BRP4 validation will be delayed a bit, that's all.

Planning for the Atlas outage is effectively out of our hands. A couple of companies need to fix a couple of problems in their installations, and the consulting engineers are trying to squeeze all the different necessary work into this one week. We don't even know the detailed plans yet, we just hope that we could power up at least a few racks of nodes at the end of the week (or Monday next).

The core Einstein@Home machines will be kept running, cooled only by the air in the room. They need to be connected to a different power (UPS) system, but as all have redundant power supplies this should go seamless. Even if we would need to shut them down for a few minutes or even hours, this shouldn't be much of a problem for the clients.

The only thing that won't work during the outage of Atlas is the pre-processing of Arecibo data. We'll try to fill every free byte on the download servers with pre-processed datasets to generate BRP workunits from before Atlas is shut down. Currently this would mean ~1600 beams, which should last about a week. This adds to our standard buffer of 450 beams, which already helped us a few times to survive (usually unplanned) weekend outages. We may also disable the CPU Apps for BRP4 during that time, to leave the BRP4 tasks for the GPUs. There will be plenty of other work for CPUs that won't be as demanding for our infrastructure as BRP4 is. Depending on how the work goes on Atlas and the demand of BRP4 work on Einstein@Home, there might be a shortage of BRP4 tasks at the end of that week. But still I wouldn't call the project "offline" then.

BM

Update:

BRP4 validators started, server status display should work for BRP4.

BRP4 assimilation and file deletion should start shortly.

FCGI_file_upload_handler still not working, duct tape solution for file upload made (more) permanent.

BM

Another update:

- Almost everything is working again. Validators finished crunching through the backlog.

- FCGI FUH is still not working. We're currently using a wrapper that calls multiple instances of the CGI version. Seems to be good enough, though somehow contradicting the whole purpose of FCGI. I'll make another attempt to fix the FCGI version when I have more time.

- The only thing that is still not working is the BRP4 "post-processing" - collecting the individual result files for each beam, producing plots etc. For this to work there is still some data ("incomplete beams") to be copied from the old to the new server. Until this is finished, post-processing is done based on data on the old server. As there are no results arriving there anymore, the "Incoming canonical result rate today" on the server status page will stay at zero until post-processing is switched over. Copying of the old results will happen in the background with low priority, I expect this to take another few more days.

BM