upload problems

Forums › Cruncher's Corner

Alex
Alex
Joined: 1 Mar 05
Posts: 451
Credit: 507758253
RAC: 80327
9 Feb 2015 12:41:49 UTC
Topic 197963
(moderation:
    )

    I see upload pending wu's on one of my pc's.
    Message says:
    09.02.2015 13:39:38 | Einstein@Home | Started upload of p2030.20131123.G176.72-00.59.N.b0s0g0.00000_1053_0_0
    09.02.2015 13:39:39 | Einstein@Home | Temporarily failed upload of p2030.20131123.G176.72-00.59.N.b0s0g0.00000_1053_0_0: transient HTTP error
    09.02.2015 13:39:39 | Einstein@Home | Backing off 00:02:39 on upload of p2030.20131123.G176.72-00.59.N.b0s0g0.00000_1053_0_0
    09.02.2015 13:39:42 | | Project communication failed: attempting access to reference site
    09.02.2015 13:39:44 | | Internet access OK - project servers may be temporarily down.

    Sebastian M. Bobrecki
    Sebastian M. Bo...
    Joined: 20 Feb 05
    Posts: 63
    Credit: 1529603535
    RAC: 108

    upload problems

    9 Feb 2015 12:42:41 UTC
    Message 129747
    (moderation:
      )

      Me too.

      Richard Haselgrove
      Richard Haselgrove
      Joined: 10 Dec 05
      Posts: 2143
      Credit: 2963929022
      RAC: 710516

      And here. Symptoms seem to be

      9 Feb 2015 12:55:30 UTC
      Message 129748
      (moderation:
        )

        And here. Symptoms seem to be similar to the ones which led to the upload server being taken offline last month - see Uploads disabled.

        In particular, I think it's a problem with the upload storage area behind the server, rather than the upload server itself.

        Bernd Machenschalk
        Bernd Machenschalk
        Moderator
        Administrator
        Joined: 15 Oct 04
        Posts: 4318
        Credit: 250813996
        RAC: 34370

        We enabled the deletion of

        9 Feb 2015 13:00:15 UTC
        Message 129749
        (moderation:
          )

          We enabled the deletion of BRP4 "tasks" (result files).

          This causes quite some I/O load for now, but should be over when the file deleter has finished its backlog.

          BM

          Edit: See Server status page: "Tasks waiting for file deletion".

          BM

          Rae Lockyer
          Rae Lockyer
          Joined: 8 Jun 05
          Posts: 3
          Credit: 6307827
          RAC: 0

          Files to be deleted doesn't

          9 Feb 2015 13:30:14 UTC
          Message 129750 in response to message 129749
          (moderation:
            )

            Files to be deleted doesn't seem to be dropping

            Laguna
            Laguna
            Joined: 22 Jan 05
            Posts: 1
            Credit: 14230044
            RAC: 0

            The numbers do drop. But they

            9 Feb 2015 13:56:34 UTC
            Message 129751
            (moderation:
              )

              The numbers do drop. But they drop slowly.
              It seems its going to take a few hours...

              Bernd Machenschalk
              Bernd Machenschalk
              Moderator
              Administrator
              Joined: 15 Oct 04
              Posts: 4318
              Credit: 250813996
              RAC: 34370

              Will drop even slower now,

              9 Feb 2015 14:33:02 UTC
              Message 129752
              (moderation:
                )

                Will drop even slower now, but the throtteling should leave more I/O for file uploads.

                BM

                BM

                Richard Haselgrove
                Richard Haselgrove
                Joined: 10 Dec 05
                Posts: 2143
                Credit: 2963929022
                RAC: 710516

                I've seen the backlog

                9 Feb 2015 15:01:06 UTC
                Message 129753
                (moderation:
                  )

                  I've seen the backlog dropping by about 100,000 in 40 minutes, so it should be finished by maybe 18:00 UTC - depending how much that last change slows it down.

                  But I still haven't seen a single upload going through. Many of the failures look like this:

                  Quote:
                  09/02/2015 14:51:20 | Einstein@Home | Started upload of p2030.20131123.G176.29-01.25.N.b1s0g0.00000_2264_1_0
                  09/02/2015 14:51:20 | Einstein@Home | [http] [ID#3323] Info: timeout on name lookup is not supported
                  09/02/2015 14:51:20 | Einstein@Home | [http] [ID#3323] Info: Hostname was found in DNS cache
                  09/02/2015 14:51:20 | Einstein@Home | [http] [ID#3323] Info: Hostname in DNS cache was stale, zapped
                  09/02/2015 14:51:20 | Einstein@Home | [http] [ID#3323] Info: Trying 130.75.116.34...
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Info: Connected to einstein4.aei.uni-hannover.de (130.75.116.34) port 80 (#5007)
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: POST /EinsteinAtHome/cgi-bin/file_upload_handler HTTP/1.1
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.4.36)
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Host: einstein4.aei.uni-hannover.de
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Accept: */*
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Accept-Encoding: deflate, gzip
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Content-Type: application/x-www-form-urlencoded
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Accept-Language: en_GB
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Content-Length: 4740
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Expect: 100-continue
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server:
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: HTTP/1.1 100 Continue
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: HTTP/1.1 404 Not Found
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: Server: nginx/1.2.1
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: Date: Mon, 09 Feb 2015 14:51:25 GMT
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: Content-Type: text/html
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: Content-Length: 168
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: Connection: close
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server:
                  09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Info: Closing connection 5007
                  09/02/2015 14:51:24 | Einstein@Home | Backing off 00:13:04 on upload of p2030.20131123.G176.29-01.25.N.b1s0g0.00000_2264_1_0


                  - the entire file is sent to the upload server, but it then gets that 404 error from (I presume) the upstream storage server. When that happens, BOINC (without the debug logging) simply shows

                  Quote:
                  09/02/2015 14:51:20 | Einstein@Home | Started upload of p2030.20131123.G176.29-01.25.N.b1s0g0.00000_2264_1_0
                  09/02/2015 14:51:24 | Einstein@Home | Backing off 00:13:04 on upload of p2030.20131123.G176.29-01.25.N.b1s0g0.00000_2264_1_0


                  Not even a transient error between 'starting' and 'backing off'.

                  Stef
                  Stef
                  Joined: 8 Mar 05
                  Posts: 206
                  Credit: 110568193
                  RAC: 0

                  Mon 09 Feb 2015 04:37:48 PM

                  9 Feb 2015 15:39:43 UTC
                  Message 129754 in response to message 129753
                  (moderation:
                    ) 

                    Mon 09 Feb 2015 04:37:48 PM CET | Einstein@Home | Not requesting tasks: too many uploads in progress


                    D'oh. Is there a workaround for this?

                    Richard Haselgrove
                    Richard Haselgrove
                    Joined: 10 Dec 05
                    Posts: 2143
                    Credit: 2963929022
                    RAC: 710516

                    RE: Mon 09 Feb 2015

                    9 Feb 2015 16:39:02 UTC
                    Message 129755 in response to message 129754
                    (moderation:
                      )

                      Quote:
                      Mon 09 Feb 2015 04:37:48 PM CET | Einstein@Home | Not requesting tasks: too many uploads in progress

                      D'oh. Is there a workaround for this?


                      No, and I don't think there should be. Getting more tasks at this stage would just add to the ever-growing backlog a few minutes later.....

                      It's a deliberate self-limiter.

                      Richard Haselgrove
                      Richard Haselgrove
                      Joined: 10 Dec 05
                      Posts: 2143
                      Credit: 2963929022
                      RAC: 710516

                      RE: We enabled the deletion

                      9 Feb 2015 16:59:12 UTC
                      Message 129756 in response to message 129749
                      (moderation:
                        )

                        Quote:

                        We enabled the deletion of BRP4 "tasks" (result files).

                        This causes quite some I/O load for now, but should be over when the file deleter has finished its backlog.

                        BM

                        Edit: See Server status page: "Tasks waiting for file deletion".


                        SSP is saying that the file deletion is complete (0 tasks backlog), but uploads don't seem to have re-started yet.

                        Edit: still getting that 404 Not Found error.

                        Comment viewing options

                        Select your preferred way to display the comments and click "Save settings" to activate your changes.

                        Forums › Cruncher's Corner