Reporting error - file not found

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 2143
Credit: 2959706111
RAC: 703703
Topic 214223

Getting this error reporting completed tasks from BOINC v7.8.3:


27/03/2018 15:10:10 | Einstein@Home | Sending scheduler request: Requested by user.
27/03/2018 15:10:10 | Einstein@Home | Reporting 7 completed tasks
27/03/2018 15:10:10 | Einstein@Home | Requesting new tasks for Intel GPU
27/03/2018 15:10:10 | Einstein@Home | [sched_op] CPU work request: 0.00 seconds; 0.00 devices
27/03/2018 15:10:10 | Einstein@Home | [sched_op] Intel GPU work request: 4295.84 seconds; 0.00 devices
27/03/2018 15:10:10 | Einstein@Home | [http] HTTP_OP::init_post(): http://scheduler.einsteinathome.org/EinsteinAtHome_cgi/cgi
27/03/2018 15:10:10 | Einstein@Home | [http] HTTP_OP::libcurl_exec(): ca-bundle set
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: HTTP/1.1 200 OK
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: Date: Tue, 27 Mar 2018 14:10:12 GMT
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: Server: Apache
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: Last-Modified: Tue, 27 Mar 2018 14:04:52 GMT
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: ETag: "e2a92686bcbc0f9ffa5a9660514c7866-gunzip-gzip"
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: Expires: Sun, 19 Nov 1978 05:00:00 GMT
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: Cache-Control: must-revalidate
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: X-Content-Type-Options: nosniff
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: X-Frame-Options: sameorigin
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: Content-Type: application/rss+xml; charset=utf-8
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: Set-Cookie: SESS3e782c8988cf3721b1565654f7f6d21b=a36sl6tq3sp78nrt80fian3lb4; expires=Thu, 19-Apr-2018 17:43:32 GMT; Max-Age=2000000; path=/; domain=einsteinathome.org; secure
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: Vary: Accept-Encoding
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: Content-Encoding: gzip
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: Content-Length: 1357
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server:
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Info: Trying 130.75.116.40...
27/03/2018 15:10:10 | | [http] [ID#0] Info: Connection #7530 to host einsteinathome.org left intact
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Info: Connected to scheduler.einsteinathome.org (130.75.116.40) port 80 (#7531)
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Sent header to server: POST /EinsteinAtHome_cgi/cgi HTTP/1.1
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Sent header to server: Host: scheduler.einsteinathome.org
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Sent header to server: User-Agent: BOINC client (windows_x86_64 7.8.3)
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Sent header to server: Accept: */*
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Sent header to server: Accept-Encoding: deflate, gzip
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Sent header to server: Content-Type: application/x-www-form-urlencoded
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Sent header to server: Accept-Language: en_GB
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Sent header to server: Content-Length: 61416
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Sent header to server: Expect: 100-continue
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Sent header to server:
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server: HTTP/1.1 404 Not Found
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server: Date: Tue, 27 Mar 2018 14:10:12 GMT
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server: Server: Apache
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server: Content-Length: 220
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server: Connection: close
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server: Content-Type: text/html; charset=iso-8859-1
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server:
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server: <html><head>
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server: <title>404 Not Found</title>
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server: </head><body>
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server: <h1>Not Found</h1>
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server: <p>The requested URL /EinsteinAtHome_cgi/cgi was not found on this server.</p>
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Received header from server: </body></html>
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Info: Closing connection 7531
27/03/2018 15:10:11 | Einstein@Home | Scheduler request failed: HTTP file not found

Error appears to be with "The requested URL /EinsteinAtHome_cgi/cgi was not found on this server." - i.e. it's a server file which is not being found.

May also be related to the line "Expires: Sun, 19 Nov 1978 05:00:00 GMT" - I know Einstein were worried about certificate expiry in older clients, but it shouldn't affect this one.

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 2143
Credit: 2959706111
RAC: 703703

Well, I updated the

Well, I updated the certificate bundle to the one we're testing for v7.10, and they all went through. Coincidence?

I'll report it upstream just in case.

Mike.Gibson
Mike.Gibson
Joined: 17 Dec 07
Posts: 21
Credit: 3759410
RAC: 0

27/03/2018 16:41:40 |

27/03/2018 16:41:40 | Einstein@Home | Scheduler request failed: Server returned nothing (no headers, no data)
27/03/2018 16:41:45 |  | Project communication failed: attempting access to reference site
27/03/2018 16:41:48 |  | Internet access OK - project servers may be temporarily down.

This affects my latest 2 units.

Mike

Juha
Juha
Joined: 27 Nov 14
Posts: 49
Credit: 4964434
RAC: 0

Richard Haselgrove

Richard Haselgrove wrote:
Well, I updated the certificate bundle to the one we're testing for v7.10, and they all went through. Coincidence?

Yes. You have two connections mixed there. The first one, ID#0, is fetching notices (pressing Update does that). This is over HTTPS which you can verify from notices/feeds.xml and from the hints in the logs:

27/03/2018 15:10:10 | Einstein@Home | [http] HTTP_OP::libcurl_exec(): ca-bundle set
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
27/03/2018 15:10:10 | | [http] [ID#0] Received header from server: Set-Cookie: SESS...; ...; path=/; domain=einsteinathome.org; secure

The second, ID#1, is the connection to the scheduler. This over plain old HTTP:

27/03/2018 15:10:10 | Einstein@Home | [http] HTTP_OP::init_post(): http://scheduler.einsteinathome.org/EinsteinAtHome_cgi/cgi
27/03/2018 15:10:10 | Einstein@Home | [http] [ID#1] Info: Connected to scheduler.einsteinathome.org (130.75.116.40) port 80 (#7531)

Port 80 is HTTP, 443 would have been HTTPS. Notice that there are no messages about setting up a secure connection.

If there had been some problem with certificates the connection would have been broken before the client had a chance to tell the server what it wanted and before the server replying with 404.

You may want to do something about that cookie in the log messages. I'm not sure if it can be used to log in to your account but no need to give anyone a reason to try it.

Mike.Gibson
Mike.Gibson
Joined: 17 Dec 07
Posts: 21
Credit: 3759410
RAC: 0

Now 6 units are a

Now 6 units are a problem:

27/03/2018 23:13:27 | Einstein@Home | Sending scheduler request: Requested by user.
27/03/2018 23:13:27 | Einstein@Home | Reporting 6 completed tasks
27/03/2018 23:13:27 | Einstein@Home | Not requesting tasks: don't need (CPU: ; Intel GPU: job cache full)
27/03/2018 23:13:28 | Einstein@Home | Scheduler request failed: Server returned nothing (no headers, no data)

Mike

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 5872
Credit: 117719185671
RAC: 35003743

When I first checked my hosts

When I first checked my hosts this morning, there were three machines with the "HTTP file not found" message that Richard reported in his initial message.  They had large queues of tasks that were uploaded but couldn't be reported.  They had large backoffs from previous attempts to report and an 'update' only repeated the backoff.

I decided to keep forcing the updates until a master file download occurred (which it eventually did) and after that the problem resolved itself.  From memory, it's about a maximum of 10 or 11 updates before a master file download will be triggered.  In each of my cases, it only took a few since they were already part way through the cycle.

.

Cheers,
Gary.

Mike.Gibson
Mike.Gibson
Joined: 17 Dec 07
Posts: 21
Credit: 3759410
RAC: 0

In my case it worked on the

In my case it worked on the 12th attempt.

Cheers

Mike

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 2143
Credit: 2959706111
RAC: 703703

Yes, I triggered a 'master

Yes, I triggered a 'master file download' too along the way. I thought that was because of my fumbling with the ca-bundle.crt: I'd renamed the old one, and it took a couple of attempts to get a clean download of the new one. During the process, I lost contact with the scheduler - but that may have been simply the stage I'd reached with the original problem.

It only happened (so far) on one of the five machines I have attached to Einstein, and I still have no idea what started it: but at least we have an idea of how to mitigate the effects now.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.