Upload issues

I saw this issue a few weeks ago.

I tend to "stand pat" with the two systems I use to crunch with, so I only check on them every few days. At times like this, I switch the box lacking work over to another project, let it download some work, then select "no new work" and check again in a few hours. My fetch cache is small, so I'm not downloading days worth of work from the secondary project. It keeps my machines busy, without dumping a ton of completed tasks which are waiting for upload. By tomorrow morning, things should be back to normal.

Hope everyone has a great 2021!

I understand that some of us know when an abort will not lose the work, but most do not.  Just like BOINC warns, many times aborting the uploads -will- kill the work.  Therefor I feel it is important to remind people reading these posts that aborting the upload might kill the work done.

I am lucky that I can keep enough tasks in my cache to work for several days before I run out of work.  Most of the time the backed-off tasks eventually clear out and everything starts working again.

I only wish that I could afford fast enough machines to run out of work within a few hours.  But, to those of you who can afford them, "KEEP ON TRUCKIN!"  (for those unfamiliar with that old expression, it means, basically, "Hurray for the good work you are doing, and good luck continuing your work!"

;-)

So this is why Einstein uploads aren't happening.

Both my HP z210 SFF xeon with Quadro 600 and low powered generic Atom D525 with nvidia gt 710 are backlogged with completed Einstein jobs.

Temporary of course. 

I'll shut down my systems for now to save power & try again tomorrow.

There are far safer ways than this to clear the stuck uploads.

The opening message from Bernd said, "... on Sundays we get a pretty high load on our main upload server ..." in describing the cause of the problem.  The temporary solution was described very briefly as, "... for now we are limiting the uploads depending on the current load."

Here's my interpretation - I know nothing more than the bald statement itself.  If the load reaches a certain level, automatically disallow all uploads until the load drops.  When below some threshold value, turn the uploads back on again.  It's a Sunday, so it's likely just a script making the decisions automatically.

We can infer that there will be regular periods when uploads are ON before going OFF again.  The problem is that every time BOINC retries and is denied, the backoff grows longer.  So don't madly try updating all the time, because you'll just make things worse and you get a much longer backoff as a result.

Just realise that the cycles in the load swings will probably be rather regular above and below whatever the set point is.  If you judge the cycle right, you can get everything uploaded in that small window.  I don't really know what the cycle time is but I'm guessing it may be something like 15 to 30 mins.  If I experimented a bit It should be easy enough to find the sweet spot.

I don't have to worry about that because the control scripts I use to monitor all my hosts do notice when uploads are stuck and do force an update if necessary to correct that problem.  My timezone is UTC+10 so all of this is happening on Sunday night for me and when I check the logs on Monday morning, all of the hosts were stuck at various times but all were cleared by the script actions.  The logs show many pages where individual hosts were stuck but also show the problem regularly being cleared.  Purely by chance the script is taking action at just the right time to hit the sweet spot.

I'm sure, anybody being prevented from downloading new tasks could clear the stuck uploads with just a little bit of guessing as to when to try.  Of course, this is not a satisfactory long term fix.  I do trust that Bernd will sort this out ASAP.