High number of task timeouts

First, some background information needed to fully understand things.

The BOINC framework needs to provide reliable communications between client and server over an inherently unreliable medium - the internet.  On each request from a client, the server checks information from the client against what it has in its own database.  It confirms that that the two sides are in agreement and it then has to source suitable tasks to fill a client request.  If the server detects a discrepancy, it will try to do something about it.

Just one of the possible safeguards is the ability to do something about previously allocated work that appears, for whatever reason, to have become 'lost' at the client end.  The server can attempt to 'resend lost tasks'.  One of the probably many ways that tasks can become lost is through unreliable comms.  The server sent a bunch of work but the client didn't receive the details.  Obviously another way is if the client's own database (the state file, client_state.xml) had the information but then got damaged in some way.

Another bit of important background in your case is that Einstein uses a 'home grown' (and older) version of the BOINC server code in order to handle locality scheduling which is vital for the efficient deploying of the very large numbers of huge data files required for GW type searches.  That code uses duration correction factor (DCF) to 'correct' the time it will take to crunch a task.  Each task contains an 'estimate' of the work content.  Even for a standard work content, different hardware configured and operated differently by each individual owner and with different operating systems, is liable to take a different amount of time.  DCF was designed to adjust the estimate progressively (for each volunteer computer) so that, irrespective of its state of configuration, the crunch time estimate could be made to match reality.

The problem with DCF is that there is only one of them per project and not per individual search being used at that project.  For DCF to have a chance of working properly here, there would need to be one per search unless somehow, the work content could be more accurately defined in the first place.  As an example of what I'm talking about, for the FGRPB1G GPU search, all my hosts need a DCF of somewhere around 0.25 to 0.35.  In other words, for the conditions I'm using, the initial estimate is around 3 times larger than what the task will actually take.

For the GW search at the conditions I'm testing (RX 570 GPU running tasks 4x) the DCF is close to 5.  Admittedly, because I'm running 4x (which pretty much doubles the daily output) the crunch time is around twice what a single task would take.  So for single tasks, the DCF would be close to 2.5 which is still close to an order of magnitude larger than what I need for FGRPB1G.  Estimated crunch times are going to act in a very crazy fashion if I try to run both searches simultaneously.

There are several ways to cope with this.  The easiest way is to choose one of the GPU searches and run it only on an individual machine.   A second way would be to alternate between searches using a suitable time span - eg. a week or a month.  You could run one search for a month unattended, set NNT at the end and return all the completed work and then reconfigure for the other search and set it going.  The changeover could be fairly quick to do once a month say.

A third way would be to allow both searches but set a very low work cache size, bearing in mind the deadline is different for each currently - 7 days for GW and 14 days for FGRPB1G last time I noticed.  If you set no more than say 0.2 days total cache size, the maximum actual work with wildly wrong estimates driven by a DCF an order of magnitude lower than it needed to be, would only be 2 days.  At the other end of the scale with a DCF 10 times too high, a full cache of the affected tasks would only be 0.02 days worth of true crunch time.  Neither of these two extremes should cause BOINC to panic so should be relatively safe for (close to) unattended operation.  You just need to convince yourself that wildly fluctuating estimates are not a concern.  The potential concern is the project unable to supply work, allowing your host to run out quite quickly.

Now, the analysis of your situation.  All dates and times are UTC as listed in the errors list.  Your machine has been running both FGRPB1G tasks and O2MDF (GW) tasks over a period.  Your GW errors list shows the "Timed out - no response" issue for close to two weeks.  Undoubtedly many older tasks have been completed by other hosts and removed from the online database over the period.  There is clear evidence of big numbers of these failures from Nov 20.

It's still happening now.  I looked yesterday and it would have been before midnight on Nov 25.  The latest date I have in my notes is a task that was timed out at 08:01:11 on Nov 25.  It had been allocated at 07:56:09.   The earliest task in that complete series that timed out at much the same time (08:01:10) was allocated earlier at 07:50:46.  It was 11 pages of errors (about 220 individual items) earlier and allocated nearly 6 mins earlier in time.  In other words, the scheduler took nearly six minutes to assemble a list of over 200 tasks to send as a batch in response to a work request.  The client had probably given up waiting - there's bound to be some sort of timeout on waiting for scheduler responses.

If you really want to know what exactly happened, go back through your event log (stdoutdae.txt) or its predecessor (stdoutdae.old) if necessary and look for event log entries just earlier than 07:50 UTC for Nov 25.  Your times will be local times, probably.  Just convert if necessary.  You should be able to see the client request and anything that came back from the server.

When it was ready to send the response, and with the client perhaps no longer paying attention, I have no idea what the scheduler does these days.  In the past, the scheduler would hold them as 'lost tasks' and would send them in batches of 12 at a time in response to further requests from the client.  Because the list is already assembled, I imagine that could happen relatively quickly without keeping the client waiting for such a long time.  So the 'resend lost tasks' feature was probably quite useful in that context.

These days it would appear that the scheduler drops the whole bundle, marks them as errors with a status of "timed out - no response".  This undoubtedly 'scares' the average volunteer into thinking it's entirely their problem.  As a complete side note, if a project is trying to 'encourage' the volunteers to keep contributing, it's sure hard to understand how 'claiming' (through the offensive terminology used) that the 'errors' are the sole responsibility of the client is the proper thing to do.  Why not a status of "Unable to allocate" or something like that.  And don't include them in the client error list if they are not client errors.

When I looked again today, there are nearly 200 more fresh "Timed out - no response" errors.  The situation with these tasks is very similar.  Assemble the full list over quite a long period then drop the lot soon after trying to send them.  Here are the actual details of two tasks, one at the start and one at the end a full batch.

Current error list page 152  -  Nov 26 - allocated at 00:20:19 and dropped at 00:27:14
Current error list page 161  -  Nov 26 - allocated at 00:26:02 and dropped at 00:27:15

Once again, 6 mins to assemble the full list and around a minute later decide to drop the lot.  Notice (as was common on many previous days) this all happened just after midnight UTC which is exactly when a "new day" for daily allocation limits kicks in.  It looks very much like your client had been 'backed-off' (ie. used up its daily limit) for some time and once a new day kicked in and it made a huge request, the scheduler wasn't able to handle the request in a timely manner.  Unfortunately, you need to work out how to break this cycle if you want to get back to normal operations.  Setting a low work cache size is the obvious first step.

I haven't looked at anything other than the GW errors list.  I have no idea how and when you get other tasks to crunch.  I have no idea of the reasonableness or otherwise of your work cache settings.  Hopefully, with some of the background I've gone through, you should be able to work out suitable settings for your situation.  If you have specific questions, please ask.  Sorry it took so long to get a response back to you.

For any of the staff happening to read, if this assessment is even roughly correct, the real problem would seem to be how long it takes for the scheduler to assemble enough tasks to meet a big request.  There seems to be a reasonable solution.  Do what is done for resending lost tasks.  Put a limit on the number that can be sent as one batch, eg. 12.  Send the 12 quickly and force the client to ask again if it really does need more.

I also recently had this problem around midnight UTC 28--29 November after upgrading my graphics card to a Radeon VII.

https://einsteinathome.org/host/12796547/tasks/6/0

At that time, I think I was in the middle of dialing in an undervolt for the card. At some point during the day I set maximum job cache to be something very low to avoid having to abort so many tasks if I had to reinstall the OS after messing up a driver installation, but I don't recall if that change was before or after these errors.

At the time I was also running all possible tasks. The following day I disabled O2MD tasks since they didn't seem to be utilizing the Radeon VII well (running them about as fast as my old 750ti).