Linux Computation Error: Once one goes, they all go.

I have a machine that does this too. It's a 3570K based host, overclocked to 4.2HGz, with a GTX650Ti. It's running GW tasks on all cores and x2 BRP4 tasks on the GPU. About every week or so, the client will just quit. No tasks have errors - it's just like the host gets sick of the stress and tells the client to get lost for a while :-). It has all new components and an after-market CPU cooler and it's in a relatively cool environment. I just restart the client and it continues happily from where it left off. It's running PCLinuxOS, fully updated. The machine actually hosts a local repository for PCLinuxOS packages which I update using rsync once a week from one of the official PCLinuxOS mirrors. I keep the local repo on a USB external hard drive and use it to update all my PCLinuxOS hosts. Updating is lightning fast from a local repo like this and I like the 'download once, deploy to many' philosophy which saves heaps of bandwidth :-). Apart from the BOINC client deciding to quit every once in a while, the machine is running perfectly smoothly.

Which is completely to be expected if BOINC thinks your copy of the app is corrupt.

I've seen this very behaviour (or slight variants of it) many times over the years, so welcome to the club :-). The thing that occurs to me is that BOINC (having decided that a vital component is corrupt) perhaps should try to get a fresh copy of that component rather than just trashing all the tasks. You would think things could be put on hold for a bit, at least while an attempt was made. However I've never seen BOINC do anything other than trash the entire cache.

Most of the examples I've seen have been with GW tasks with one of the large data files suddenly being declared corrupt. Because of locality scheduling, it's quite likely that most if not all tasks in the cache could depend on a single large data file. I've also seen examples of either the 'sun' or 'earth' or 'skygrid' files suddenly failing the MD5 checksum check. A long time ago, when I saw my first examples of this, I got suspicious as to why files in continuous use should suddenly be corrupt. I decided to check and to my surprise, on many occasions, the files in question were not actually corrupt at all. I could use a separate program to generate a checksum and it agreed perfectly with what was stored in the state file.

I'm sure that some of these failures were due to transient hardware issues under the high temperature, high stress conditions of crunching. I'm sure that flaky RAM has caused quite a few. Also some were due to old disks developing bad sectors, etc. In these cases, the files were actually bad. I don't delete any files that are bad. I simply rename them with a .BAD extension and then get a fresh copy.

When one of these problems occurs, BOINC seems to trash the affected tasks and go into a 24hr backoff, with all the errored tasks just sitting there in the tasks tab and the clock counting down. You could just 'update' to force BOINC to report them all but, with a bit of experimenting, I think there is a perfectly viable recovery procedure. You need to be familiar with the contents of the state file (client_state.xml) and you need to be prepared to edit it after stopping BOINC.

In summary, what I do is

• * Check/replace any files that BOINC thinks are corrupt.
  * Find and fix all negative values in blocks.
  * Find and remove any blocks that have been added by BOINC to blocks.
  * Fix (or completely remove) all the blocks that have been trashed by BOINC.
  * Restart BOINC.
  

This is just a summary of what I do - there are quite a few pesky little details I've glossed over :-) I would experience at least one of these failures per month and, so far, I've always been able to recover the full cache of trashed tasks as long as I notice the problem before the 24 hr backoff expires. Obviously it's too late if BOINC has already reported the damage.

The trickiest bit is handling the blocks appropriately. If there are successfully completed results, you don't (usually) need to touch those but you certainly want to keep them so they can be reported. It appears that BOINC often decides that something is corrupt just as one task is finishing and a new one is starting. This seems (occasionally) to cause the just finished task to be marked as an error. So I usually check and fix that if necessary. The needs to be zero and the needs to be 5.

not started or not completed should have an of zero and a of 2. Trashed will have three distinguishing characteristics. The will be -185, the will be 3, and there will be an block of several lines followed by a tag and a value. All these lines need to be removed because (with the 'corrupt' file checked or replaced) there is no error condition, the result is not ready to report and a completed time has not yet been determined. If you have lots of results in your cache, it can be a bit tedious to fix every last one of them like this.

So, with ~100 tasks to recover, I usually fix a few (a couple per core) and then completely delete all the remaining blocks from the state file. That way, there is work to commence when BOINC is restarted and when I allow the client to contact the server, the server will notice what is missing and send them all back to the client (in batches of 12 per contact) as 'resend lost results'.

I'm not suggesting that this will be viable for the average volunteer. You really do need to be completely comfortable with editing the state file and you need a good understanding of what , , , etc, blocks should look like. And you need a lot of patience :-). These days I'm quite comfortable with all this and it usually only takes me around 15 minutes to recover from one of these 'events', once I've noticed it :-).

I used to think that too but, if you think about it, the project knows quickly and can resend the tasks so it's no biggie. For me, the biggie is the inconvenience of the 1 task per day per core limit until good results are returned and the extra management to get things back on track. It's worthwhile for me to spend 15 mins and then be immediately back to full speed with a full cache of work and an unsullied tasks per day limit :-).

I'm using 6.12.34.

Thanks for this information. I don't have time to follow BOINC development so I wasn't aware of this particular bug in BOINC. The Linux distro I use is very slow with upgrading BOINC packages (they're still on 6.10.x) so I normally grab a version from the BOINC website and install the components manually. I check out the library dependencies with ldd and then find any needed extra library packages and install them. I wanted to go to V7 BOINC on this machine but when I did the ldd check, I found the client depends on version 2.15 of glibc and the latest version in the repo is 2.13 so at that point I put the transition to V7 BOINC on hold. If the FD leak was fixed in a V6 BOINC, I'll give 6.12.43 a spin and hope that the glibc 2.15 requirement didn't come in until V7 :-).

I have a number of machines with GTX650 GPUs that are on 6.12.34 but the fastest one is the only one so far where I've noticed this problem. Looks like I'd better attend to this rather quickly :-). Thanks again for the info.

Hmmmm...

Interesting. I've been plowing through all my project message boards to get back up to speed with what is and has been going on since I took my sabbatical.

This is the first one I've come across discussing 'cascade' failures with BOINC (other than ones where I mentioned it).

I noted you talked about it in terms of the effect on (and remedies for) EAH.

So the question I have for you is when they happen are they just killing EAH or does it get any other projects you have running as well. The reason I ask is when I have one on Windows I can't remember a single time it didn't take out everything the host had in the cache at the time the cascade started.

Of course, that could just be I never seem to be in front a host when the cascade starts. Even if I had been though, when I examine the post mortem wreckage and see how fast it progressed through the cache, I'm not sure I could have done anything to stop it anyway. ;-)

Just as a side note to your remarks about BOINC versions. At least on Windows, I never had one occur until I was running a CC in the late 6x series. I want to say that it wasn't until they moved to 7 as the official and I upgraded to 6.12.34 since that was the last 'recommended' version (IIRC), but I just don't remember for sure. I haven't had one on the 7x host I have (yet). :-D

Something similar seems to have just happened to me; a whole bunch of BRP4 tasks all just errored out like this within a few seconds of starting.

Though poking about the results, quite a few of the tasks have also failed for my wingmen (who all seems to running Windows). Could it be that some (or even one?) of the iffy workunits has somehow caused a cascade failure? Or is there a bad batch of jobs? Something else?

Failures on the affected host here

A few sample tasks that also errored for others 1 2 3 4

I did find a few jobs that a wingman managed to complete 1 2 3

As yet, none of the failed jobs have validated ok for anyone else as far as I can see.