Servers said to be down at present (2021-01-03)

Glenn Hawley, RASC Calgary
Glenn Hawley, R...
Joined: 6 Mar 05
Posts: 48
Credit: 892043034
RAC: 346505
Topic 224401

No communication has occurred for the last day or more, and I'm now run out of GPU tasks with only a few CPU tasks still running.

The Event Log says: "Error reported by file upload server: File uploads are temporarily unavailable"

Usually this sort of an outage lasts no more than a few hours, but is there a more serious problem here?

archae86
archae86
Joined: 6 Dec 05
Posts: 3157
Credit: 7221554931
RAC: 967170

Glenn Hawley, RASC Calgary

Glenn Hawley, RASC Calgary wrote:
The Event Log says: "Error reported by file upload server: File uploads are temporarily unavailable"

Uploads unavailable is not the same thing as server down.

Bernd has reported that the project has a throttling scheme when the upload requests exceed capacity, and that in recent times this has tended to activate during the weekends and largely to clear by end of the day Sunday.

Some uploads got through now and again here even at the peak congestion, and the success rate has gone way up in recent hours.I currently have just 13 pending uploads across three capable systems running GPU GRP tasks, where I had hundreds eight hours ago.  This is not the fruit of manual retries, just letting the automation proceed.

I suspect if you wait a couple more hours you'll find things back to normal.

 

mikey
mikey
Joined: 22 Jan 05
Posts: 12682
Credit: 1839087224
RAC: 3839

archae86 wrote: Glenn

archae86 wrote:

Glenn Hawley, RASC Calgary wrote:
The Event Log says: "Error reported by file upload server: File uploads are temporarily unavailable"

Uploads unavailable is not the same thing as server down.

Bernd has reported that the project has a throttling scheme when the upload requests exceed capacity, and that in recent times this has tended to activate during the weekends and largely to clear by end of the day Sunday.

Some uploads got through now and again here even at the peak congestion, and the success rate has gone way up in recent hours.I currently have just 13 pending uploads across three capable systems running GPU GRP tasks, where I had hundreds eight hours ago.  This is not the fruit of manual retries, just letting the automation proceed.

I suspect if you wait a couple more hours you'll find things back to normal. 

And just like that I clicked 'retry now' and my over a dozen uploads all went up in a flash.

Glenn Hawley, RASC Calgary
Glenn Hawley, R...
Joined: 6 Mar 05
Posts: 48
Credit: 892043034
RAC: 346505

What concerns me more is that

What concerns me more is that I'm getting no new tasks at all.

ie- no downloads either.

 

Glenn Hawley, RASC Calgary
Glenn Hawley, R...
Joined: 6 Mar 05
Posts: 48
Credit: 892043034
RAC: 346505

Ah... not 10 minutes after I

Ah... not 10 minutes after I sent the last post, half a dozen tasks for the GPU downloaded. 

So maybe things are back on track again.

 

mikey
mikey
Joined: 22 Jan 05
Posts: 12682
Credit: 1839087224
RAC: 3839

Glenn Hawley, RASC Calgary

Glenn Hawley, RASC Calgary wrote:

What concerns me more is that I'm getting no new tasks at all.

ie- no downloads either. 

I am getting both downloads and the finished tasks are being uploads. I am doing cpu tasks on this machine, I haven't checked my other machines though.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.