Unplanned project downtime 26 Aug 2022

Forums › Technical News

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4332
Credit: 251716235
RAC: 35398
27 Aug 2022 11:15:37 UTC
Topic 228036
(moderation:
    )

    There was a campus-wide power outage yesterday afternoon, and the UPS that powers our network equipment stayed offline even after the power was restored, until some manual intervention this morning.

    We'll continue to scan the systems for trouble caused by the outage, so far we didn't find more things broken than a few disks.

    We'll do a detailed post-mortem analysis on Monday.

    Sorry for the unplanned outage and the trouble it may have caused on your end.

    BM

    GWGeorge007
    GWGeorge007
    Joined: 8 Jan 18
    Posts: 3119
    Credit: 5017593401
    RAC: 1861110

    No problem Bernd, thank you

    27 Aug 2022 11:27:04 UTC
    Message 200228
    (moderation:
      )

      No problem Bernd, thank you for staying on top of it and keeping us informed.  I know that working on the weekend wasn't what you expected to do, but thanks again for staying on top of it.

      Many of us had an alternate plan for the "just in case" scenario, so it really didn't matter that much.

       

      George

      Proud member of the Old Farts Association

      Ian&Steve C.
      Ian&Steve C.
      Joined: 19 Jan 20
      Posts: 4045
      Credit: 48133277066
      RAC: 33410965

      thanks Bernd. The only

      27 Aug 2022 11:44:56 UTC
      Message 200229
      (moderation:
        )

        thanks Bernd.

        The only problem I see now is a lot of stuck uploads that wont go through. not sure if it's just because many people are trying to upload as well, or is there's a problem with the upload server.

        _________________________________________________________________________

        Richard Haselgrove
        Richard Haselgrove
        Joined: 10 Dec 05
        Posts: 2143
        Credit: 2981917223
        RAC: 755191

        Upload error

        27 Aug 2022 12:21:16 UTC
        Message 200233
        (moderation:
          )

          Upload error log:

          27/08/2022 13:18:59 | Einstein@Home | [http] [ID#14980] Info:  connect to 130.75.116.34 port 80 failed: No route to host
          27/08/2022 13:18:59 | Einstein@Home | [http] [ID#14980] Info:  Failed to connect to einstein4.aei.uni-hannover.de port 80: No route to host
          27/08/2022 13:18:59 | Einstein@Home | [http] [ID#14980] Info:  Closing connection 5909
          27/08/2022 13:18:59 | Einstein@Home | [http] HTTP error: Couldn't connect to server
          27/08/2022 13:19:00 | Einstein@Home | Temporarily failed upload of LATeah3012L12220819_892.0_0_0.0_20553435_0_0: connect() failed
           

          Mike Hewson
          Mike Hewson
          Moderator
          Joined: 1 Dec 05
          Posts: 6591
          Credit: 325253344
          RAC: 210665

          Uploading and downloading has

          27 Aug 2022 12:58:00 UTC
          Message 200235
          (moderation:
            )

            Uploading and downloading has stabilised for me at least, no more project requested back-offs. Thanks for the 'weekend servicing' Bernd.

            Cheers, Mike.

            I have made this letter longer than usual because I lack the time to make it shorter ...

            ... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

            zombie67 [MM]
            Joined: 10 Oct 06
            Posts: 121
            Credit: 504037728
            RAC: 115265

            I still have tasks that will

            27 Aug 2022 13:07:25 UTC
            Message 200236
            (moderation:
              )

              I still have tasks that will not upload.

              36653    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_550000_0_0    
              36654    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_700000_0_0    
              36655    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_750000_0_0    
              36656    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_800000_0_0    
              36657    Einstein@Home    8/27/2022 6:01:57 AM    Started upload of M22_1_cfbf00002_segment_4_dms_200_185_600000_0_0    
              36658    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_550000_0_0: connect() failed    
              36659    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:31:42 on upload of M22_1_cfbf00002_segment_4_dms_200_185_550000_0_0    
              36660    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_700000_0_0: connect() failed    
              36661    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:25:34 on upload of M22_1_cfbf00002_segment_4_dms_200_185_700000_0_0    
              36662    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_750000_0_0: connect() failed    
              36663    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:05:11 on upload of M22_1_cfbf00002_segment_4_dms_200_185_750000_0_0    
              36664    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_800000_0_0: connect() failed    
              36665    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:06:17 on upload of M22_1_cfbf00002_segment_4_dms_200_185_800000_0_0    
              36666    Einstein@Home    8/27/2022 6:02:19 AM    Temporarily failed upload of M22_1_cfbf00002_segment_4_dms_200_185_600000_0_0: connect() failed    
              36667    Einstein@Home    8/27/2022 6:02:19 AM    Backing off 00:06:11 on upload of M22_1_cfbf00002_segment_4_dms_200_185_600000_0_0    
              36668            8/27/2022 6:02:20 AM    Project communication failed: attempting access to reference site    
              36669            8/27/2022 6:02:22 AM    Internet access OK - project servers may be temporarily down.    
               

              Reno, NV Team: SETI.USA

              Harri Liljeroos
              Harri Liljeroos
              Joined: 10 Dec 05
              Posts: 4469
              Credit: 3268865684
              RAC: 1968953

              The uploads are failing for

              27 Aug 2022 14:06:47 UTC
              Message 200241
              (moderation:
                )

                The uploads are failing for the Meerkat tasks. All FGRB uploads have been cleared.

                tullio
                tullio
                Joined: 22 Jan 05
                Posts: 2118
                Credit: 61407735
                RAC: 0

                I have 116 tasks failures,

                27 Aug 2022 16:04:38 UTC
                Message 200247
                (moderation:
                  )

                  I have 116 tasks failures, all uploaded.

                  Tullio

                  Ian&Steve C.
                  Ian&Steve C.
                  Joined: 19 Jan 20
                  Posts: 4045
                  Credit: 48133277066
                  RAC: 33410965

                  Harri Liljeroos wrote: The

                  27 Aug 2022 17:05:27 UTC
                  Message 200248 in response to message 200241
                  (moderation:
                    )

                    Harri Liljeroos wrote:

                    The uploads are failing for the Meerkat tasks. All FGRB uploads have been cleared.

                    same, only BRP7 tasks stuck for me now.

                    _________________________________________________________________________

                    Bedrich Hajek
                    Bedrich Hajek
                    Joined: 9 Dec 05
                    Posts: 7
                    Credit: 1651810524
                    RAC: 42039

                    I am also unable to upload

                    27 Aug 2022 19:45:02 UTC
                    Message 200254
                    (moderation:
                      )

                      I am also unable to upload BRP7s.

                       

                       

                      zombie67 [MM]
                      Joined: 10 Oct 06
                      Posts: 121
                      Credit: 504037728
                      RAC: 115265

                      FWIW, I occasionally get

                      28 Aug 2022 5:22:10 UTC
                      Message 200278
                      (moderation:
                        )

                        FWIW, I occasionally get more BRP7s to download.  But when completed they still will not upload.

                        Reno, NV Team: SETI.USA

                        Comment viewing options

                        Select your preferred way to display the comments and click "Save settings" to activate your changes.

                        Forums › Technical News