Pending Credit

One factor that probably modulates resend time delays is the locality scheduling which Einstein employs to reduce network traffic. When this works nicely, many GW WUs are sliced out of one set of biggish files transferred from the servers to a particular host.

I imagine resends go much more quickly if a suitable host which already has the requisite files aboard requests additional work of the right type soon after the resend window opens up. Contrarywise, it may be considerably delayed if there is a lack of a host with the requisite files already onboard requesting more work.

Generally this locality thing is good--for users who pay by the byte, it gives very substantial savings in download network traffic charges, lowers outgoing load from the servers, and probably somewhere even saves a few grams of carbon emissions.

Here is a writeup on locality scheduling by Oliver Bock of the Einstein staff, which may have more detail than you like, but might give more of an impression of how this works and how it can create delays (and also how it means your set of quorum partners at any given moment is not at all a random sample of the entire Einstein community).

Of course, it is also possible that something has gone wrong here, but I'd not bet on it.

Nothing like this has happened in this case. It was not caused by a BOINC bug with hostIDs. It's most likely simply due to the idiosyncrasies of locality scheduling used for the current LV run. (Please note that the 4th example quoted by Campion that was issued "within 2 hours" was a FGRP task and so was not subject to locality scheduling).

It was just coincidence that these three LV tasks had three separate 1st wingmen, all of whom failed to return those tasks. The three hosts in question all had quite different physical hardware so it couldn't have been an existing host being issued with new IDs.

In each of these particular three cases, there was quite a delay between the issue of the first and second tasks - several days in fact. That in itself (whilst not common) is not all that unusual - I've seen it happen before. Once the three second tasks were issued (over a 6 second interval to 3 consecutive new hostIDs - 5020956, 5020957 and 5020958) all three lingered around for a further 14 days unreturned so that third tasks were needed in each case.

It's not unusual for tasks never to be returned. It happens in a surprisingly large number of cases so seeing three in a row like this doesn't necessarily mean that something is wrong. The unusual thing is that it took around a further 10 days after each deadline miss for a suitable host to be found and the third task to be issued. In fact, a single host eventually took all three resends and has since returned them all so the saga is now at an end.

As others have mentioned, a delay in issuing tasks can be caused by a shortage of hosts who already have the necessary group of large data files on board. The scheduler seems to prefer to wait until a suitably endowed host requests work. Perhaps it is waiting too long before deciding to issue new data files to a requesting host.

A natural way to get rid of resends without the extra data download penalty is to issue them to new hosts which need a data download anyway. In the past I've seen many newly added hosts get resend tasks as their very first work issue.

With different hardware in each case, but all with exactly the same operating system: Linux 3.0.0-15-virtual

Could this be characteristic of a cluster or hardware burn-in test starting up? Especially since each host fetched precisely one task for precisely one CPU on multicore hardware.

Whilst I know the Atlas and similar clusters contribute a great deal to Einstein's overall scientific research, maybe their interaction with the locality scheduler needs a second look if this is they way they interact with 'regular' users - and in this case, they didn't do any useful science, either.

OK, I'll leave it in your capable hands, then.

From what I read, the current clients handle 'allow_multile_clients' OK, but there's an outstanding bug in the manager that makes it hard to control them. I don't imagine the clusters have very much use for BOINC Manager, though! One workround which has been found to work is to use third-party applications like BoincTasks for any management that might be necessary - I believe you're in private contact with somebody who can advise on that.