Unplanned project downtime 26 Aug 2022

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4265
Credit: 244922206
RAC: 16811
Topic 228036

There was a campus-wide power outage yesterday afternoon, and the UPS that powers our network equipment stayed offline even after the power was restored, until some manual intervention this morning.

We'll continue to scan the systems for trouble caused by the outage, so far we didn't find more things broken than a few disks.

We'll do a detailed post-mortem analysis on Monday.

Sorry for the unplanned outage and the trouble it may have caused on your end.

BM

GWGeorge007
GWGeorge007
Joined: 8 Jan 18
Posts: 2767
Credit: 4540297819
RAC: 2145032

No problem Bernd, thank you

No problem Bernd, thank you for staying on top of it and keeping us informed.  I know that working on the weekend wasn't what you expected to do, but thanks again for staying on top of it.

Many of us had an alternate plan for the "just in case" scenario, so it really didn't matter that much.

 

George

Proud member of the Old Farts Association

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 3681
Credit: 33819734166
RAC: 37800505

thanks Bernd. The only

thanks Bernd.

The only problem I see now is a lot of stuck uploads that wont go through. not sure if it's just because many people are trying to upload as well, or is there's a problem with the upload server.

_________________________________________________________________________

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 2139
Credit: 2752648967
RAC: 1479594

Upload error

Upload error log:

27/08/2022 13:18:59 | Einstein@Home | [http] [ID#14980] Info:  connect to 130.75.116.34 port 80 failed: No route to host
27/08/2022 13:18:59 | Einstein@Home | [http] [ID#14980] Info:  Failed to connect to einstein4.aei.uni-hannover.de port 80: No route to host
27/08/2022 13:18:59 | Einstein@Home | [http] [ID#14980] Info:  Closing connection 5909
27/08/2022 13:18:59 | Einstein@Home | [http] HTTP error: Couldn't connect to server
27/08/2022 13:19:00 | Einstein@Home | Temporarily failed upload of LATeah3012L12220819_892.0_0_0.0_20553435_0_0: connect() failed
 

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6534
Credit: 284700169
RAC: 113726

Uploading and downloading has

Uploading and downloading has stabilised for me at least, no more project requested back-offs. Thanks for the 'weekend servicing' Bernd.

Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

zombie67 [MM]
Joined: 10 Oct 06
Posts: 110
Credit: 451529802
RAC: 10303

I still have tasks that will

I still have tasks that will not upload.

36653    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_550000_0_0    
36654    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_700000_0_0    
36655    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_750000_0_0    
36656    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_800000_0_0    
36657    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_600000_0_0    
36658    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_550000_0_0: connect() failed    
36659    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:31:42 on upload of M22_1_cfbf00002_segment_4_dms_200_185_550000_0_0    
36660    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_700000_0_0: connect() failed    
36661    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:25:34 on upload of M22_1_cfbf00002_segment_4_dms_200_185_700000_0_0    
36662    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_750000_0_0: connect() failed    
36663    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:05:11 on upload of M22_1_cfbf00002_segment_4_dms_200_185_750000_0_0    
36664    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_800000_0_0: connect() failed    
36665    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:06:17 on upload of M22_1_cfbf00002_segment_4_dms_200_185_800000_0_0    
36666    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_600000_0_0: connect() failed    
36667    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:06:11 on upload of M22_1_cfbf00002_segment_4_dms_200_185_600000_0_0    
36668            8/27/2022 6:02:20 AM    Project communication failed: attempting access to reference site    
36669            8/27/2022 6:02:22 AM    Internet access OK - project servers may be temporarily down.    
 

Reno, NV
Team: SETI.USA

Harri Liljeroos
Harri Liljeroos
Joined: 10 Dec 05
Posts: 3608
Credit: 2901435670
RAC: 1033627

The uploads are failing for

The uploads are failing for the Meerkat tasks. All FGRB uploads have been cleared.

tullio
tullio
Joined: 22 Jan 05
Posts: 2118
Credit: 61407735
RAC: 0

I have 116 tasks failures,

I have 116 tasks failures, all uploaded.

Tullio

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 3681
Credit: 33819734166
RAC: 37800505

Harri Liljeroos wrote: The

Harri Liljeroos wrote:

The uploads are failing for the Meerkat tasks. All FGRB uploads have been cleared.

same, only BRP7 tasks stuck for me now.

_________________________________________________________________________

Bedrich Hajek
Bedrich Hajek
Joined: 9 Dec 05
Posts: 7
Credit: 1644171288
RAC: 23610

I am also unable to upload

I am also unable to upload BRP7s.

 

 

zombie67 [MM]
Joined: 10 Oct 06
Posts: 110
Credit: 451529802
RAC: 10303

FWIW, I occasionally get

FWIW, I occasionally get more BRP7s to download.  But when completed they still will not upload.

Reno, NV
Team: SETI.USA

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.