No Work Sent
30 Aug 2014 2:03:07 UTC
Topic 197699
(moderation:
I am getting a No work sent message. The log file (below) refers me to scheduler log messages, but they dont mean anything to me.
Can someone tell me why I'm not getting new work?
Thanks.
8/29/2014 6:29:33 PM | Einstein@Home | Finished upload of h1_0993.80_S6Directed__S6CasAf40a_994.75Hz_439_0_1
8/29/2014 6:29:34 PM | Einstein@Home | Finished upload of h1_0993.80_S6Directed__S6CasAf40a_994.75Hz_439_0_0
8/29/2014 6:30:14 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 6:30:14 PM | Einstein@Home | Reporting 1 completed tasks
8/29/2014 6:30:14 PM | Einstein@Home | Requesting new tasks for CPU
8/29/2014 6:30:19 PM | Einstein@Home | Scheduler request completed: got 0 new tasks
8/29/2014 6:30:19 PM | Einstein@Home | No work sent
8/29/2014 6:30:19 PM | Einstein@Home | see scheduler log messages on http://einstein.phys.uwm.edu//host_sched_logs/3754/3754765
8/29/2014 6:31:12 PM | Einstein@Home | Computation for task h1_0998.70_S6Directed__S6CasAf40a_999.6Hz_425_1 finished
8/29/2014 6:31:14 PM | Einstein@Home | Started upload of h1_0998.70_S6Directed__S6CasAf40a_999.6Hz_425_1_0
8/29/2014 6:31:14 PM | Einstein@Home | Started upload of h1_0998.70_S6Directed__S6CasAf40a_999.6Hz_425_1_1
8/29/2014 6:31:18 PM | Einstein@Home | Finished upload of h1_0998.70_S6Directed__S6CasAf40a_999.6Hz_425_1_0
8/29/2014 6:31:18 PM | Einstein@Home | Finished upload of h1_0998.70_S6Directed__S6CasAf40a_999.6Hz_425_1_1
8/29/2014 6:31:20 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 6:31:20 PM | Einstein@Home | Reporting 1 completed tasks
8/29/2014 6:31:20 PM | Einstein@Home | Requesting new tasks for CPU
8/29/2014 6:31:23 PM | Einstein@Home | Scheduler request completed: got 0 new tasks
8/29/2014 6:31:23 PM | Einstein@Home | No work sent
8/29/2014 6:31:23 PM | Einstein@Home | see scheduler log messages on http://einstein.phys.uwm.edu//host_sched_logs/3754/3754765
8/29/2014 6:31:47 PM | Einstein@Home | Computation for task h1_0993.15_S6Directed__S6CasAf40a_994.35Hz_136_0 finished
8/29/2014 6:31:49 PM | Einstein@Home | Started upload of h1_0993.15_S6Directed__S6CasAf40a_994.35Hz_136_0_0
8/29/2014 6:31:49 PM | Einstein@Home | Started upload of h1_0993.15_S6Directed__S6CasAf40a_994.35Hz_136_0_1
8/29/2014 6:31:51 PM | Einstein@Home | Finished upload of h1_0993.15_S6Directed__S6CasAf40a_994.35Hz_136_0_1
8/29/2014 6:31:52 PM | Einstein@Home | Finished upload of h1_0993.15_S6Directed__S6CasAf40a_994.35Hz_136_0_0
8/29/2014 6:32:23 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 6:32:23 PM | Einstein@Home | Reporting 1 completed tasks
8/29/2014 6:32:23 PM | Einstein@Home | Requesting new tasks for CPU
8/29/2014 6:32:27 PM | Einstein@Home | Scheduler request completed: got 0 new tasks
8/29/2014 6:32:27 PM | Einstein@Home | No work sent
8/29/2014 6:32:27 PM | Einstein@Home | see scheduler log messages on http://einstein.phys.uwm.edu//host_sched_logs/3754/3754765
8/29/2014 6:35:22 PM | Einstein@Home | Computation for task h1_0992.05_S6Directed__S6CasAf40a_993.25Hz_118_0 finished
8/29/2014 6:35:24 PM | Einstein@Home | Started upload of h1_0992.05_S6Directed__S6CasAf40a_993.25Hz_118_0_0
8/29/2014 6:35:24 PM | Einstein@Home | Started upload of h1_0992.05_S6Directed__S6CasAf40a_993.25Hz_118_0_1
8/29/2014 6:35:28 PM | Einstein@Home | Finished upload of h1_0992.05_S6Directed__S6CasAf40a_993.25Hz_118_0_0
8/29/2014 6:35:28 PM | Einstein@Home | Finished upload of h1_0992.05_S6Directed__S6CasAf40a_993.25Hz_118_0_1
8/29/2014 6:35:28 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 6:35:28 PM | Einstein@Home | Reporting 1 completed tasks
8/29/2014 6:35:28 PM | Einstein@Home | Requesting new tasks for CPU
8/29/2014 6:35:32 PM | Einstein@Home | Scheduler request completed: got 0 new tasks
8/29/2014 6:35:32 PM | Einstein@Home | No work sent
8/29/2014 6:35:32 PM | Einstein@Home | see scheduler log messages on http://einstein.phys.uwm.edu//host_sched_logs/3754/3754765
8/29/2014 6:50:32 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 6:50:32 PM | Einstein@Home | Requesting new tasks for CPU
8/29/2014 6:50:45 PM | Einstein@Home | Scheduler request completed: got 0 new tasks
8/29/2014 6:50:45 PM | Einstein@Home | No work sent
8/29/2014 6:50:45 PM | Einstein@Home | see scheduler log messages on http://einstein.phys.uwm.edu//host_sched_logs/3754/3754765
8/29/2014 7:03:46 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 7:03:46 PM | Einstein@Home | Requesting new tasks for CPU
8/29/2014 7:03:47 PM | Einstein@Home | Scheduler request completed: got 0 new tasks
8/29/2014 7:03:47 PM | Einstein@Home | No work sent
8/29/2014 7:03:47 PM | Einstein@Home | see scheduler log messages on http://einstein.phys.uwm.edu//host_sched_logs/3754/3754765
8/29/2014 7:31:35 PM | Einstein@Home | Computation for task h1_0995.00_S6Directed__S6CasAf40a_995.85Hz_526_1 finished
8/29/2014 7:31:37 PM | Einstein@Home | Started upload of h1_0995.00_S6Directed__S6CasAf40a_995.85Hz_526_1_0
8/29/2014 7:31:37 PM | Einstein@Home | Started upload of h1_0995.00_S6Directed__S6CasAf40a_995.85Hz_526_1_1
8/29/2014 7:31:40 PM | Einstein@Home | Finished upload of h1_0995.00_S6Directed__S6CasAf40a_995.85Hz_526_1_1
8/29/2014 7:31:40 PM | Einstein@Home | Finished upload of h1_0995.00_S6Directed__S6CasAf40a_995.85Hz_526_1_0
8/29/2014 7:31:43 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 7:31:43 PM | Einstein@Home | Reporting 1 completed tasks
8/29/2014 7:31:43 PM | Einstein@Home | Requesting new tasks for CPU
8/29/2014 7:31:56 PM | Einstein@Home | Scheduler request completed: got 0 new tasks
8/29/2014 7:31:56 PM | Einstein@Home | No work sent
8/29/2014 7:31:56 PM | Einstein@Home | see scheduler log messages on http://einstein.phys.uwm.edu//host_sched_logs/3754/3754765
8/29/2014 7:38:57 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 7:38:57 PM | Einstein@Home | Requesting new tasks for CPU
8/29/2014 7:39:00 PM | Einstein@Home | Scheduler request completed: got 0 new tasks
8/29/2014 7:39:00 PM | Einstein@Home | No work sent
8/29/2014 7:39:00 PM | Einstein@Home | see scheduler log messages on http://einstein.phys.uwm.edu//host_sched_logs/3754/3754765
8/29/2014 7:44:48 PM | Einstein@Home | Computation for task h1_0995.70_S6Directed__S6CasAf40a_996.85Hz_209_1 finished
8/29/2014 7:44:51 PM | Einstein@Home | Started upload of h1_0995.70_S6Directed__S6CasAf40a_996.85Hz_209_1_0
8/29/2014 7:44:51 PM | Einstein@Home | Started upload of h1_0995.70_S6Directed__S6CasAf40a_996.85Hz_209_1_1
8/29/2014 7:44:54 PM | Einstein@Home | Finished upload of h1_0995.70_S6Directed__S6CasAf40a_996.85Hz_209_1_0
8/29/2014 7:44:54 PM | Einstein@Home | Finished upload of h1_0995.70_S6Directed__S6CasAf40a_996.85Hz_209_1_1
8/29/2014 7:44:56 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 7:44:56 PM | Einstein@Home | Reporting 1 completed tasks
8/29/2014 7:44:56 PM | Einstein@Home | Requesting new tasks for CPU
8/29/2014 7:44:58 PM | Einstein@Home | Scheduler request completed: got 0 new tasks
8/29/2014 7:44:58 PM | Einstein@Home | No work sent
8/29/2014 7:44:58 PM | Einstein@Home | see scheduler log messages on http://einstein.phys.uwm.edu//host_sched_logs/3754/3754765
8/29/2014 7:51:58 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 7:51:58 PM | Einstein@Home | Requesting new tasks for CPU
8/29/2014 7:52:08 PM | Einstein@Home | Scheduler request completed: got 1 new tasks
8/29/2014 7:52:10 PM | Einstein@Home | Starting task h1_0992.10_S6Directed__S6CasAf40a_993.25Hz_194_0
8/29/2014 7:53:12 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 7:53:12 PM | Einstein@Home | Requesting new tasks for CPU
8/29/2014 7:53:20 PM | Einstein@Home | Scheduler request completed: got 0 new tasks
8/29/2014 7:53:20 PM | Einstein@Home | No work sent
8/29/2014 7:53:20 PM | Einstein@Home | see scheduler log messages on http://einstein.phys.uwm.edu//host_sched_logs/3754/3754765
8/29/2014 8:00:21 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 8:00:21 PM | Einstein@Home | Requesting new tasks for CPU
8/29/2014 8:00:23 PM | Einstein@Home | Scheduler request completed: got 0 new tasks
8/29/2014 8:00:23 PM | Einstein@Home | No work sent
8/29/2014 8:00:23 PM | Einstein@Home | see scheduler log messages on http://einstein.phys.uwm.edu//host_sched_logs/3754/3754765
8/29/2014 8:22:24 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 8:22:24 PM | Einstein@Home | Requesting new tasks for CPU
8/29/2014 8:22:26 PM | Einstein@Home | Scheduler request completed: got 0 new tasks
8/29/2014 8:22:26 PM | Einstein@Home | No work sent
8/29/2014 8:22:26 PM | Einstein@Home | see scheduler log messages on http://einstein.phys.uwm.edu//host_sched_logs/3754/3754765
8/29/2014 9:21:33 PM | Einstein@Home | Sending scheduler request: To fetch work.
8/29/2014 9:21:33 PM | Einstein@Home | Requesting new tasks for CPU and NVIDIA
8/29/2014 9:21:41 PM | Einstein@Home | Scheduler request completed: got 1 new tasks
8/29/2014 9:21:43 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3472.bin4
8/29/2014 9:21:43 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3473.bin4
8/29/2014 9:22:05 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3472.bin4
8/29/2014 9:22:05 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3474.bin4
8/29/2014 9:22:14 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3473.bin4
8/29/2014 9:22:14 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3475.bin4
8/29/2014 9:22:35 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3474.bin4
8/29/2014 9:22:35 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3476.bin4
8/29/2014 9:22:46 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3475.bin4
8/29/2014 9:22:46 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3477.bin4
8/29/2014 9:23:10 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3476.bin4
8/29/2014 9:23:10 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3478.bin4
8/29/2014 9:23:17 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3477.bin4
8/29/2014 9:23:17 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3479.bin4
8/29/2014 9:23:45 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3479.bin4
8/29/2014 9:23:45 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3480.bin4
8/29/2014 9:23:51 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3478.bin4
8/29/2014 9:23:51 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3481.bin4
8/29/2014 9:24:13 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3480.bin4
8/29/2014 9:24:13 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3482.bin4
8/29/2014 9:24:25 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3481.bin4
8/29/2014 9:24:25 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3483.bin4
8/29/2014 9:24:36 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3482.bin4
8/29/2014 9:24:36 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3484.bin4
8/29/2014 9:24:55 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3484.bin4
8/29/2014 9:24:55 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3485.bin4
8/29/2014 9:25:04 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3483.bin4
8/29/2014 9:25:04 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3486.bin4
8/29/2014 9:25:20 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3485.bin4
8/29/2014 9:25:20 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3487.bin4
8/29/2014 9:25:30 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3486.bin4
8/29/2014 9:25:30 PM | Einstein@Home | Started download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000.zap
8/29/2014 9:25:32 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000.zap
8/29/2014 9:25:48 PM | Einstein@Home | Finished download of p2030.20140624.G43.70-02.08.C.b0s0g0.00000_3487.bin4
No Work Sent
)
probably because there is no work available for any of the CPU applications you have enabled for the location (aka venue) of that host.
For several hours now none of my four hosts have been getting any new CPU work. To summarize my (imperfect and incomplete) understanding by application:
1. Gamma ray pulsar search #4: has been in beta test of small batches this week. Last batch was issued well over 24 hours ago. Meant to go into full production "real soon now".
2. Gamma ray pulsar search #3: ending now--now work available.
3. S6 CasA--ending now, I don't have this one enabled, but believe no work is usually available now. I tried enabling it just now for the venue for one host, and just got an addition "no work available" line in my event log for that application.
4. Binary Radio Pulsar search (Arecibo)--until recently this one did not offer CPU work (it did offer Intel GPU work), but it was offering work for most of the last two days (all four of my hosts got some). However since a few hours ago none of my hosts get any, and receive a message saying none is available, despite the fact that the server status page shows quite a few tasks ready to send. My guess is that CPU job availability is again off--perhaps by accident, perhaps intentionally.
If you are not already doing so, you may find it more helpful to review the messages regarding your work request--if you have a modern boincmgr installed these may be viewed by selecting Advanced|Event log...
archae86 wrote:3. S6
)
Now that host has received eight S6 CasA work units. Interestingly enough, though they carry a Beta designation which one might expect to imply a short deadline, these came with a 14-day deadline. My personal choice, in this time of scarcity for Einstein CPU work, has been to enable both beta-test and S6 CasA for all four venues.
How do you see whether it´s
)
How do you see whether it´s beta or not?
So far, I managed to keep crunching FGRP3 work units thanks to a few resends, but when that source dried up and without FGRP4 released, I thought I might switch to a venue allowing everything but no beta testing (I have a slow cruncher without suitable GPU), to get whatever would be available.
I initially got that log message report, but it appeared later that I was actually downloading a few hundred files, which took some time (nothing critical, minutes, but slow enough to notice). Somehow to my surprise, a lot of GW having _S6CasAf40a_ in the file name... I was surprised because I thought it was done with, as the server pages state 100% done for S6CasAf40 (without that appended a). So is this a different run? I´m a bit worried that I might be crunching for nothing, seeing that none of the ones received last night have a wingman yet, but came with an estimated time to completion being way too short - DCF was set to over 1.6 when the first completed. I don´t remember from older GW runs what the file format was, when I did the same as FGRP2 ran out of work.
I also got a bunch of BRP4 work units, of which the first one seems to have a problem validating, something I do not remember having seen previously either, but that aside. However, that one too is missing a third cruncher at this moment.
Maybe there is a problem with the scheduler?
RE: ... Interestingly
)
That app is not new - it was released back in April and was announced in this thread. It's been running ever since. It always had the standard 14 day deadline.
There seems to be a stop/start nature to the availability of S6CasA tasks in general at the moment. I believe this may well be to try to manage server load.
A day or two ago, all the remaining S6CasA primary tasks (_0 and _1 suffix) seem to have been generated and inserted into the online database. On the status page, I noticed a big increase in 'Tasks to send' for S6CasA from just a few thousand to something close to 180K tasks if I recall correctly. That is now down to around 140K. This probably had an impact on server load. I noticed big delays in server response and on many occasions server error messages and also 1 hour backoffs with the 'down for maintenance' message. During that time, I'm guessing that S6CasA availability was disabled from time to time to try to calm things down at peak periods. Generating all the remaining tasks and inserting them all at once into the db is not unusual. It has been done in the past when a GW run has all but ended. Once those last 140K tasks have been finally dished out we can look forward to quite a few weeks of resends (_2 and above suffix) before the run is finally finished.
At the moment, server response seems a lot better so I'm hoping that the 'sticky tape and rubber bands' have managed to hold things together for long enough to get through the worst of the crisis :-). I guess there might be some more fun when the new FGRP4 (production) run joins the mix. I'm guessing that is being held back to allow things to settle a bit first. I'm glad I don't have Bernd's job :-).
Cheers,
Gary.
RE: How do you see whether
)
Take a look at the 'Application name' column in your tasks list. If it contains 'beta' in the name then ...
That source hasn't dried up completely. There are still ~50K FGRP3 tasks in progress of which a proportion will fail and further resends will be generated. If you happen to ask at the 'right' time, you will be given some. The resends will keep trickling out for a while yet.
That percentage refers to the generating of the WUs and not the issuing of the tasks. There are still ~140K primary tasks to go, and then weeks of resends. It is not a different run. You will help the project enormously if you can afford the download bandwidth for all the large data files in helping 'clean up the dregs' - both the remaining primary tasks as well as the subsequent resends.
The project servers are a bit overloaded at the moment. Scotty's trying to coax a bit more out of the warp drive :-).
Cheers,
Gary.
Thank you very much for the
)
Thank you very much for the extensive explanation Gary. You cleared everything that was bugging me.
Thanks for the replies. Last
)
Thanks for the replies. Last night, when I posted my original message, my other hosts were happily crunching away. This morning, one of them is also running a little short of work, although I did at least see some new work.
Thanks for the explanation.
RE: At the moment, server
)
The root cause of our current trouble is the implementation of "locality scheduling", which has been designed, implemented and tested almost ten years ago. While our DB is much more powerful these days, thanks to better hardware and Olivers optimizations in indexing and querying, it's improvement is still way behind the increase of computing power compared to back then.
We have been planning to re-design and re-implement this for years. However as with a few other things (like the new Drupal-based website) this has been continuously postponed in favor of other, more urgent things.
The trouble happens when there is only little "locality" work left compared to the number of requests for a particular set of data files the client has. Other applications that don't require the scheduler to match the client's data files don't cause such issues (they are, however, still affected by a high DB load). Thus what I did in the past few days was basically to transform the "locality scheduling" S6CasA app into a "normal" app where work is sent independently of the client's data files. As there are not many different sets of such data files around anymore, the additional download volume that this causes shouldn't be too high, and people have been warned about large download sizes anyway and can opt-out if they want or need to.
FGRP4 will be a "normal" application in BOINC terms that doesn't require "locality scheduling", so I'm not at all expecting any problems on the scheduler side from it.
What makes me worried, though, are the next GW runs, "S6Directed" and particularly the "S6Bucket follow-up", which is designed to "run" through the same (data file) frequency range as the original S6Bucket run in about 1/6 of the time.
BM
BM
RE: FGRP4 will be a
)
Any rough idea of when this run will start?
Thanks.
Cheers,
Gary.
RE: RE: FGRP4 will be a
)
This was scheduled for last week, but the DB trouble kept me from sorting out the last oddities. Current EST is early next week.
BM
BM