Unplanned project downtime 26 Aug 2022

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,136
Credit: 230,870,970
RAC: 17,261
Topic 228036

There was a campus-wide power outage yesterday afternoon, and the UPS that powers our network equipment stayed offline even after the power was restored, until some manual intervention this morning.

We'll continue to scan the systems for trouble caused by the outage, so far we didn't find more things broken than a few disks.

We'll do a detailed post-mortem analysis on Monday.

Sorry for the unplanned outage and the trouble it may have caused on your end.

BM

GWGeorge007
GWGeorge007
Joined: 8 Jan 18
Posts: 1,563
Credit: 2,767,716,175
RAC: 6,138,913

No problem Bernd, thank you

No problem Bernd, thank you for staying on top of it and keeping us informed.  I know that working on the weekend wasn't what you expected to do, but thanks again for staying on top of it.

Many of us had an alternate plan for the "just in case" scenario, so it really didn't matter that much.

 

George

Proud member of the Old Farts Association (OFA)

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 2,612
Credit: 21,434,736,034
RAC: 34,911,004

thanks Bernd. The only

thanks Bernd.

The only problem I see now is a lot of stuck uploads that wont go through. not sure if it's just because many people are trying to upload as well, or is there's a problem with the upload server.

_________________________________________________________________________

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 2,121
Credit: 1,919,982,458
RAC: 2,761,989

Upload error

Upload error log:

27/08/2022 13:18:59 | Einstein@Home | [http] [ID#14980] Info:  connect to 130.75.116.34 port 80 failed: No route to host
27/08/2022 13:18:59 | Einstein@Home | [http] [ID#14980] Info:  Failed to connect to einstein4.aei.uni-hannover.de port 80: No route to host
27/08/2022 13:18:59 | Einstein@Home | [http] [ID#14980] Info:  Closing connection 5909
27/08/2022 13:18:59 | Einstein@Home | [http] HTTP error: Couldn't connect to server
27/08/2022 13:19:00 | Einstein@Home | Temporarily failed upload of LATeah3012L12220819_892.0_0_0.0_20553435_0_0: connect() failed
 

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6,403
Credit: 208,249,855
RAC: 143,382

Uploading and downloading has

Uploading and downloading has stabilised for me at least, no more project requested back-offs. Thanks for the 'weekend servicing' Bernd.

Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

zombie67 [MM]
Joined: 10 Oct 06
Posts: 108
Credit: 356,887,099
RAC: 2,242

I still have tasks that will

I still have tasks that will not upload.

36653    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_550000_0_0    
36654    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_700000_0_0    
36655    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_750000_0_0    
36656    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_800000_0_0    
36657    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_600000_0_0    
36658    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_550000_0_0: connect() failed    
36659    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:31:42 on upload of M22_1_cfbf00002_segment_4_dms_200_185_550000_0_0    
36660    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_700000_0_0: connect() failed    
36661    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:25:34 on upload of M22_1_cfbf00002_segment_4_dms_200_185_700000_0_0    
36662    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_750000_0_0: connect() failed    
36663    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:05:11 on upload of M22_1_cfbf00002_segment_4_dms_200_185_750000_0_0    
36664    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_800000_0_0: connect() failed    
36665    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:06:17 on upload of M22_1_cfbf00002_segment_4_dms_200_185_800000_0_0    
36666    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_600000_0_0: connect() failed    
36667    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:06:11 on upload of M22_1_cfbf00002_segment_4_dms_200_185_600000_0_0    
36668            8/27/2022 6:02:20 AM    Project communication failed: attempting access to reference site    
36669            8/27/2022 6:02:22 AM    Internet access OK - project servers may be temporarily down.    
 

Reno, NV
Team: SETI.USA

Harri Liljeroos
Harri Liljeroos
Joined: 10 Dec 05
Posts: 2,367
Credit: 2,242,808,461
RAC: 1,178,177

The uploads are failing for

The uploads are failing for the Meerkat tasks. All FGRB uploads have been cleared.

tullio
tullio
Joined: 22 Jan 05
Posts: 2,108
Credit: 61,407,735
RAC: 73

I have 116 tasks failures,

I have 116 tasks failures, all uploaded.

Tullio

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 2,612
Credit: 21,434,736,034
RAC: 34,911,004

Harri Liljeroos wrote: The

Harri Liljeroos wrote:

The uploads are failing for the Meerkat tasks. All FGRB uploads have been cleared.

same, only BRP7 tasks stuck for me now.

_________________________________________________________________________

Bedrich Hajek
Bedrich Hajek
Joined: 9 Dec 05
Posts: 6
Credit: 1,135,659,211
RAC: 2,395,904

I am also unable to upload

I am also unable to upload BRP7s.

 

 

zombie67 [MM]
Joined: 10 Oct 06
Posts: 108
Credit: 356,887,099
RAC: 2,242

FWIW, I occasionally get

FWIW, I occasionally get more BRP7s to download.  But when completed they still will not upload.

Reno, NV
Team: SETI.USA

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.