upload pending forever

Richie
Richie
Joined: 7 Mar 14
Posts: 656
Credit: 1702989778
RAC: 0
Topic 203584

I've got two tasks stuck, upload: pending. I remember seeing this problem also earlier this year. At that time I ended up re-attaching the project on that host, because I didn't figure out how to get rid of those infinite pending uploads. Is there another way? Other Einstein-tasks upload and download fine. I've tried restarting computer and Boinc and network connection and tried "retry" for those uploads, but it does nothing.

Right now these files are stuck (Host # 12469906):

LATeah2003L_20.0_0_-5.7e-11_576_1_0

p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_3

p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_4

p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_5

p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_6

p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_7

Here's latest contact with server: https://einsteinathome.org/host/12469906/log

At the same time those files have status "Upload:pending" and at the task list they have 100% progress and status is "Uploading".

Christian Beer
Christian Beer
Joined: 9 Feb 05
Posts: 595
Credit: 118641056
RAC: 109758

Other tasks use the same

Other tasks use the same server and upload handler. I guess you already tried rebooting the whole machine? You could also try to activate some http_debug flags to see why the upload is not working.

Richie
Richie
Joined: 7 Mar 14
Posts: 656
Credit: 1702989778
RAC: 0

I activated http_debug flag,

I activated http_debug flag, restarted computer and started Boinc Manager. Then I resumed network activity. Here's some log:


 

Sat 10 Dec 2016 06:54:16 PM EET |  | Resuming network activity
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | [file_xfer] Couldn't start upload of LATeah2003L_20.0_0_-5.7e-11_576_1_0
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | [file_xfer] URL http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_medium: not found
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | Backing off 00:03:03 on upload of LATeah2003L_20.0_0_-5.7e-11_576_1_0
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | [file_xfer] Couldn't start upload of p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_3
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | [file_xfer] URL http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_medium: not found
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | Backing off 00:02:11 on upload of p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_3
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | [file_xfer] Couldn't start upload of p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_4
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | [file_xfer] URL http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_medium: not found
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | Backing off 00:02:37 on upload of p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_4
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | [file_xfer] Couldn't start upload of p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_5
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | [file_xfer] URL http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_medium: not found
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | Backing off 00:02:32 on upload of p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_5
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | [file_xfer] Couldn't start upload of p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_6
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | [file_xfer] URL http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_medium: not found
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | Backing off 00:03:45 on upload of p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_6
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | [file_xfer] Couldn't start upload of p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_7
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | [file_xfer] URL http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_medium: not found
Sat 10 Dec 2016 06:54:16 PM EET | Einstein@Home | Backing off 00:02:27 on upload of p2030.20141118.G192.45+00.71.C.b1s0g0.00000_960_0_7
Sat 10 Dec 2016 06:54:16 PM EET |  | [http] HTTP_OP::init_get(): https://albertathome.org/rss_main.php
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:    Trying 129.89.61.67...
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  Connected to albertathome.org (129.89.61.67) port 443 (#0)
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  ALPN, offering http/1.1
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  successfully set certificate verify locations:
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:    CAfile: ca-bundle.crt
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:    CApath: /etc/ssl/certs
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  TLSv1.2 (OUT), TLS header, Certificate Status (22):
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  TLSv1.2 (OUT), TLS handshake, Client hello (1):
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  TLSv1.0 (IN), TLS handshake, Server hello (2):
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  TLSv1.0 (IN), TLS handshake, Certificate (11):
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  TLSv1.0 (IN), TLS handshake, Server key exchange (12):
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  TLSv1.0 (IN), TLS handshake, Server finished (14):
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  TLSv1.0 (OUT), TLS handshake, Client key exchange (16):
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  TLSv1.0 (OUT), TLS change cipher, Client hello (1):
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  TLSv1.0 (OUT), TLS handshake, Finished (20):
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  TLSv1.0 (IN), TLS change cipher, Client hello (1):
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  TLSv1.0 (IN), TLS handshake, Finished (20):
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  SSL connection using TLSv1.0 / DHE-RSA-AES128-SHA
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  ALPN, server did not agree to a protocol
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  Server certificate:
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:       subject: C=US; ST=Wisconsin; L=Milwaukee; O=University of Wisconsin - Milwaukee; OU=IT; CN=www.albertathome.org
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:       start date: Oct 19 00:00:00 2015 GMT
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:       expire date: Oct 18 23:59:59 2018 GMT
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:       subjectAltName: albertathome.org matched
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:       issuer: C=US; O=thawte, Inc.; CN=thawte SSL CA - G2
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:       SSL certificate verify ok.
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Sent header to server: GET /rss_main.php HTTP/1.1
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Sent header to server: Host: albertathome.org
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 7.6.31)
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Sent header to server: Accept: */*
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Sent header to server: Accept-Encoding: deflate, gzip
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Sent header to server: Content-Type: application/x-www-form-urlencoded
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Sent header to server: Accept-Language: en_US
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Sent header to server:
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: HTTP/1.1 200 OK
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: Date: Sat, 10 Dec 2016 16:54:18 GMT
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: Server: Apache/2.2.3 (CentOS)
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: X-Frame-Options: SAMEORIGIN
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: X-Powered-By: PHP/5.3.3
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: Set-Cookie: SESS760ce8e7499ac68de5f1863e1164f56b=fhec7eqh034c3044hpu49k2213; expires=Mon, 02-Jan-2017 20:27:38 GMT; path=/; domain=.albertathome.org; secure
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: Last-Modified: Sat, 10 Dec 2016 16:00:41 GMT
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: ETag: "19dd8bab77a1e6aa907655214002b13e"
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: Expires: Sun, 19 Nov 1978 05:00:00 GMT
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: Cache-Control: must-revalidate
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: Content-Encoding: gzip
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: Content-Length: 1312
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server: Content-Type: application/rss+xml; charset=utf-8
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Received header from server:
Sat 10 Dec 2016 06:54:18 PM EET |  | [http] [ID#0] Info:  Connection #0 to host albertathome.org left intact
Sat 10 Dec 2016 06:54:19 PM EET |  | [http] HTTP_OP::init_get(): https://einsteinathome.org/rss_main.php
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:    Trying 129.89.61.70...
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  Connected to einsteinathome.org (129.89.61.70) port 443 (#1)
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  ALPN, offering http/1.1
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  successfully set certificate verify locations:
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:    CAfile: ca-bundle.crt
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:    CApath: /etc/ssl/certs
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  TLSv1.2 (OUT), TLS header, Certificate Status (22):
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  TLSv1.2 (OUT), TLS handshake, Client hello (1):
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  TLSv1.2 (IN), TLS handshake, Server hello (2):
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  TLSv1.2 (IN), TLS handshake, Certificate (11):
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  TLSv1.2 (IN), TLS handshake, Server key exchange (12):
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  TLSv1.2 (IN), TLS handshake, Server finished (14):
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  TLSv1.2 (OUT), TLS change cipher, Client hello (1):
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  TLSv1.2 (OUT), TLS handshake, Finished (20):
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  TLSv1.2 (IN), TLS change cipher, Client hello (1):
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  TLSv1.2 (IN), TLS handshake, Finished (20):
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  ALPN, server did not agree to a protocol
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:  Server certificate:
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:       subject: C=US; ST=Wisconsin; L=Milwaukee; O=University of Wisconsin-Milwaukee; OU=Physics; CN=www.einsteinathome.org
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:       start date: Dec 11 00:00:00 2015 GMT
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:       expire date: Dec 10 23:59:59 2018 GMT
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:       subjectAltName: einsteinathome.org matched
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:       issuer: C=US; O=thawte, Inc.; CN=thawte SSL CA - G2
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Info:       SSL certificate verify ok.
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Sent header to server: GET /rss_main.php HTTP/1.1
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Sent header to server: Host: einsteinathome.org
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 7.6.31)
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Sent header to server: Accept: */*
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Sent header to server: Accept-Encoding: deflate, gzip
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Sent header to server: Content-Type: application/x-www-form-urlencoded
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Sent header to server: Accept-Language: en_US
Sat 10 Dec 2016 06:54:20 PM EET |  | [http] [ID#0] Sent header to server:
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: HTTP/1.1 200 OK
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: Date: Sat, 10 Dec 2016 16:54:21 GMT
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: Server: Apache
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: Last-Modified: Sat, 10 Dec 2016 16:33:05 GMT
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: ETag: "38fe375a2fa5bbf2fb9233e81cbc6201-gunzip-gzip"
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: Expires: Sun, 19 Nov 1978 05:00:00 GMT
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: Cache-Control: must-revalidate
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: X-Content-Type-Options: nosniff
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: X-Frame-Options: sameorigin
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: Content-Type: application/rss+xml; charset=utf-8
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: Set-Cookie: -------**; expires=Mon, 02-Jan-2017 20:27:41 GMT; Max-Age=2000000; path=/; domain=einsteinathome.org; secure
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: Vary: Accept-Encoding
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: Content-Encoding: gzip
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server: Content-Length: 1348
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Received header from server:
Sat 10 Dec 2016 06:54:21 PM EET |  | [http] [ID#0] Info:  Connection #1 to host einsteinathome.org left intact


 ** I took away the "Set-Cookie" line because I wasn't sure if it was harmful to show.

Ps. At the same time this same host has a bunch of Albert-tasks ready to report, but haven't been able to report them or download new ones, because scheduler responds "HTTP internal error". This host is # 17746 there. Same situation also with another host (# 17617 in Albert). Other hosts through same network are able to report tasks and download more Albert work.

AgentB
AgentB
Joined: 17 Mar 12
Posts: 915
Credit: 513211304
RAC: 0

On the affected hosts what

On the affected hosts what happens if you follow this URL

http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_medium

you should see something like this.

<data_server_reply>
    <status>1</status>
    <message>no command</message>
</data_server_reply>

If blocked then maybe AV or firewall issue.

 

Richie
Richie
Joined: 7 Mar 14
Posts: 656
Credit: 1702989778
RAC: 0

I used Firefox and both

I used Firefox and both affected hosts were able to show exactly those four lines. I don't have any AV installed on any of these hosts. Mint's Firewall is ON (automatic settings). Same firewall settings are also on the other three hosts that work still fine. Same setup worked earlier without problems also on these two affected hosts.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.