Discussion Thread for the Continuous GW Search known as O2MD1 (now O2MDF - GPUs only)

OK, thanks for that.  I just had a look at the latest log for that host and it's rather large and convoluted.  Here are some key entries together with my (possibly quite erroneous) interpretation.  The log starts with the successful return of 18 FGRPB1G tasks with a "SCHEDULER_REQUEST::parse(): unrecognized: <allow_multiple_clients>0</allow_multiple_clients>" thrown in for good measure.  Looks like the Einstein scheduler doesn't understand that particular option.

That is followed by

2019-10-06 06:43:27.5173 [PID=6927 ] [send] CPU: req 0.00 sec, 0.00 instances; est delay 0.00
2019-10-06 06:43:27.5173 [PID=6927 ] [send] CUDA: req 7932.99 sec, 0.00 instances; est delay 0.00

which shows no request for CPU work but a request for GPU work.  As an aside, there must be something at your end that is preventing your client from requesting CPU work.  Then there is a (I've trimmed the timestamps to just show the actual messages) request for locality scheduling (ie GW GPU) work.  There is always some sort of 'lottery' that goes on to check what gets sent first - locality or non-locality.  Locality won this time.  You can also see checks for 'old' work - I assume generated work in particular frequency bins that is getting 'stale' - not enough hosts requesting that particular bin so let's add this host to one of these bins if stale work exists.  Of course, with a brand new search, no stale work will exist.

[mixed] sending locality work first (0.3650)
[locality] [HOST#12789230] ignoring file JPLEPH.405
[locality] [HOST#12789230] ignoring file earth00-19-DE405.dat
[locality] [HOST#12789230] ignoring file sun00-19-DE405.dat
[locality] [HOST#12789230] ignoring file 20180220_O2_excludebadepochs_min5sft_9m60h.seg
[locality] send_work_locality(): locality_appid_filter 'appid IN (52,53) AND'
[send] send_old_work() no feasible result older than 336.0 hours
[locality] send_work_locality(): sending work for new file(s)
[locality] send_new_file_work(): try to send old work
[send] send_old_work() no feasible result younger than 324.9 hours and older than 168.0 hours
[locality] send_new_file_work(0): try to send from working set

Then there are a zillion lines where it tries to find available tasks belonging to particular large data files.  I'm not going to reproduce it all - I'll just list the filenames that were checked

[locality] send_new_file_working_set will try filename h1_0115.55_O2C02Cl1In0
[locality] send_new_file_working_set will try filename h1_0094.55_O2C02Cl1In0
[locality] send_new_file_working_set will try filename h1_0114.50_O2C02Cl1In0
[locality] send_new_file_working_set will try filename h1_0101.70_O2C02Cl1In0
[locality] send_new_file_working_set will try filename h1_0072.05_O2C02Cl1In0

Each of these searches fails.  Looks like 5 separate frequency bins will be tried before the scheduler gives up.  Here is the last fail invoking a potential order to the work generator (maybe it needs =1 rather then =0 to actually get work generated), together with what the scheduler then does.

[locality] make_more_work_for_file(h1_0072.05_O2C02Cl1In0, 1)=0
[locality] send_work_locality(): returning with work still needed
[mixed] sending non-locality work second

So the scheduler gives up the hunt for locality scheduling work and moves on to fill your request with non-locality work - ie. FGRPB1G.

At no point is there any complaint about the O2MD1 CPU app but it was obviously happy to try to fill a request for O2MD1 GPU work - which Bernd has disabled until sometime next week when he has worked out if any further mod to the GPU app is needed.

I've had several looks at the server status page over the last several hours.  My memory may be faulty but I seem to recall the tasks in progress (1828) and number available to send (6292) as being at those same values at each previous inspection.  Maybe none are being sent out anyway.  That still doesn't explain why your host is requesting 0.00 secs of CPU work - a puzzle for you to solve :-).

No, quite the contrary.  If an initial task for particular data has been sent out and a second task hasn't been sent (ie. the quorum is becoming stale and the original recipient might start complaining) then prioritise the sending of the second task to anyone asking for new work for the first time because they won't already have existing data files.  I'm guessing the 7 day mark will target just 'new' hosts whereas by 14 days things are getting desperate so the very next request will be targeted (existing data or not) just to get the very stale quorums moving.

With a new search like O2MD1, there will be no stale quorums so people will keep getting work aligned to data they already have.