upload problems

Alex
Alex
Joined: 1 Mar 05
Posts: 449
Credit: 338,940,596
RAC: 8,290
Topic 197963

I see upload pending wu's on one of my pc's.
Message says:
09.02.2015 13:39:38 | Einstein@Home | Started upload of p2030.20131123.G176.72-00.59.N.b0s0g0.00000_1053_0_0
09.02.2015 13:39:39 | Einstein@Home | Temporarily failed upload of p2030.20131123.G176.72-00.59.N.b0s0g0.00000_1053_0_0: transient HTTP error
09.02.2015 13:39:39 | Einstein@Home | Backing off 00:02:39 on upload of p2030.20131123.G176.72-00.59.N.b0s0g0.00000_1053_0_0
09.02.2015 13:39:42 | | Project communication failed: attempting access to reference site
09.02.2015 13:39:44 | | Internet access OK - project servers may be temporarily down.

Sebastian M. Bobrecki
Sebastian M. Bo...
Joined: 20 Feb 05
Posts: 62
Credit: 1,043,257,333
RAC: 1,126,324

upload problems

Me too.

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 1,935
Credit: 270,564,967
RAC: 261,957

And here. Symptoms seem to be

And here. Symptoms seem to be similar to the ones which led to the upload server being taken offline last month - see Uploads disabled.

In particular, I think it's a problem with the upload storage area behind the server, rather than the upload server itself.

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 3,938
Credit: 200,113,756
RAC: 44,949

We enabled the deletion of

We enabled the deletion of BRP4 "tasks" (result files).

This causes quite some I/O load for now, but should be over when the file deleter has finished its backlog.

BM

Edit: See Server status page: "Tasks waiting for file deletion".

BM

Rae Lockyer
Rae Lockyer
Joined: 8 Jun 05
Posts: 3
Credit: 6,307,827
RAC: 0

Files to be deleted doesn't

Files to be deleted doesn't seem to be dropping

Laguna
Laguna
Joined: 22 Jan 05
Posts: 1
Credit: 11,257,867
RAC: 2,989

The numbers do drop. But they

The numbers do drop. But they drop slowly.
It seems its going to take a few hours...

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 3,938
Credit: 200,113,756
RAC: 44,949

Will drop even slower now,

Will drop even slower now, but the throtteling should leave more I/O for file uploads.

BM

BM

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 1,935
Credit: 270,564,967
RAC: 261,957

I've seen the backlog

I've seen the backlog dropping by about 100,000 in 40 minutes, so it should be finished by maybe 18:00 UTC - depending how much that last change slows it down.

But I still haven't seen a single upload going through. Many of the failures look like this:

Quote:
09/02/2015 14:51:20 | Einstein@Home | Started upload of p2030.20131123.G176.29-01.25.N.b1s0g0.00000_2264_1_0
09/02/2015 14:51:20 | Einstein@Home | [http] [ID#3323] Info: timeout on name lookup is not supported
09/02/2015 14:51:20 | Einstein@Home | [http] [ID#3323] Info: Hostname was found in DNS cache
09/02/2015 14:51:20 | Einstein@Home | [http] [ID#3323] Info: Hostname in DNS cache was stale, zapped
09/02/2015 14:51:20 | Einstein@Home | [http] [ID#3323] Info: Trying 130.75.116.34...
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Info: Connected to einstein4.aei.uni-hannover.de (130.75.116.34) port 80 (#5007)
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: POST /EinsteinAtHome/cgi-bin/file_upload_handler HTTP/1.1
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.4.36)
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Host: einstein4.aei.uni-hannover.de
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Accept: */*
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Accept-Encoding: deflate, gzip
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Content-Type: application/x-www-form-urlencoded
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Accept-Language: en_GB
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Content-Length: 4740
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server: Expect: 100-continue
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Sent header to server:
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: HTTP/1.1 100 Continue
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: HTTP/1.1 404 Not Found
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: Server: nginx/1.2.1
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: Date: Mon, 09 Feb 2015 14:51:25 GMT
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: Content-Type: text/html
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: Content-Length: 168
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server: Connection: close
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Received header from server:
09/02/2015 14:51:23 | Einstein@Home | [http] [ID#3323] Info: Closing connection 5007
09/02/2015 14:51:24 | Einstein@Home | Backing off 00:13:04 on upload of p2030.20131123.G176.29-01.25.N.b1s0g0.00000_2264_1_0


- the entire file is sent to the upload server, but it then gets that 404 error from (I presume) the upstream storage server. When that happens, BOINC (without the debug logging) simply shows

Quote:
09/02/2015 14:51:20 | Einstein@Home | Started upload of p2030.20131123.G176.29-01.25.N.b1s0g0.00000_2264_1_0
09/02/2015 14:51:24 | Einstein@Home | Backing off 00:13:04 on upload of p2030.20131123.G176.29-01.25.N.b1s0g0.00000_2264_1_0


Not even a transient error between 'starting' and 'backing off'.

Stef
Stef
Joined: 8 Mar 05
Posts: 206
Credit: 110,568,193
RAC: 1

Mon 09 Feb 2015 04:37:48 PM

Mon 09 Feb 2015 04:37:48 PM CET | Einstein@Home | Not requesting tasks: too many uploads in progress


D'oh. Is there a workaround for this?

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 1,935
Credit: 270,564,967
RAC: 261,957

RE: Mon 09 Feb 2015

Quote:
Mon 09 Feb 2015 04:37:48 PM CET | Einstein@Home | Not requesting tasks: too many uploads in progress

D'oh. Is there a workaround for this?


No, and I don't think there should be. Getting more tasks at this stage would just add to the ever-growing backlog a few minutes later.....

It's a deliberate self-limiter.

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 1,935
Credit: 270,564,967
RAC: 261,957

RE: We enabled the deletion

Quote:

We enabled the deletion of BRP4 "tasks" (result files).

This causes quite some I/O load for now, but should be over when the file deleter has finished its backlog.

BM

Edit: See Server status page: "Tasks waiting for file deletion".


SSP is saying that the file deletion is complete (0 tasks backlog), but uploads don't seem to have re-started yet.

Edit: still getting that 404 Not Found error.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.