Task time estimates and sent tasks

Hi George, welcome to Einstein@Home.

I agree with archae86 when he says that you may not have various preferences set correctly for this project. I'll also comment on a couple of other points that you make.

Is your "two days" the total of both settings values that control work fetch? Are you using website preferences or have you set them locally within BOINC Manager? What values are you using for each setting? I can assure you that whatever they really are, they aren't being ignored.

There are three separate pulsar searches currently going on. There is BRP5 (Binary Radio Pulsar search #5, aka Perseus Arm Survey) which is a GPU only search. It's using data from the Parkes Radio Telescope in Australia. There is BRP4 (Binary Radio Pulsar search #4) which uses data from the Arecibo Radio Telescope. The "G" version of this search is for the more powerful external GPUs like yours. You don't seem to have any of those. The third search is FGRP4 (Fermi Gamma Ray Pulsar search #4) and it is using data gathered by the Fermi space telescope. This search is currently a 'CPU only' search. Radio pulsar searches and the gamma ray pulsar search are looking for quite different signals.

You appear to be running BRP5 on your GPU and FGRP4 on your CPU cores. Because E@H uses highly customised server code based on 'old' BOINC server code, it still uses a single (per project) DCF (Duration Correction Factor) for correcting run time estimates. This means that when you are running different searches within the one project, the estimates will never really converge. They will tend to bounce around a bit if the different searches have different 'accuracies' built into their tasks, as they tend to do. There are a lot of different factors that make it impossible to remove all the oscillations.

Despite all this, it is possible to craft your preferences in such a way as to give you a 'set and forget' operation. The key is to keep your cache settings relatively small. Personally, I keep a 3 day supply. I use a 'low water mark' of 3.0 days and an 'extra days' setting of 0.01.

I really don't understand what you are saying here. Perseus Arm Survey (BRP5) is GPU only so on your machine with the GTX670 I can see 76 of these of which 49 are completed and only 27 are 'in progress'. For 48 out of the 49 completed tasks, the run time has been between 9500 and 10000 seconds. Are you perhaps referring to FGRP4 (Gamma Ray Pulsar) tasks?

This setting simply provides the interval after which BOINC will check to see if a task from a different project is more eligible to run than the current one. If you have managed to get into high priority (panic) mode, BOINC will not change to a different task until the panic is over.

It seems to me that you have inappropriate cache settings (and, perhaps, inappropriate resource shares for the current situation). You must have, if BOINC keeps requesting more work. What resource shares have you assigned to each different project? Do you realise that if your highest share project can't supply work, BOINC will tend to grab the whole lot from a low resource share project? This can create havoc (and panic mode) later on. If you are going to add a project to make up for a project that can't supply, you must reduce your cache size to a nice low value beforehand simply to avoid potential mayhem.

At first, I couldn't work out what "3rd" referred to - 3rd time?, 3rd host?, 3rd project? - but I couldn't find any aborted BRP5 tasks on any host and then I noticed a whole bunch of aborted FGRP4 tasks on your machine with the GTX670. So now I realise you must have been referring to the deadline date of 3rd December.

At the moment you have 129 FGRP4 tasks on that machine. No doubt there were more (going back to 19th Nov) which have since been deleted from the online database. Of the 129, 75 have been returned and a further 31 were aborted. There are 23 in progress, of which 4 have exceeded the deadline. They must have already been started at the time the others were aborted.

Once again, the crunch times for the 75 completed tasks seem to be reasonably consistent. The vast majority are in the range of 50000 to 55000 secs with the odd couple of tasks more towards 60000. These times seem perfectly normal for that machine (4 cores, 8 threads). I see no evidence of CPU tasks running for excessive lengths of time.

My strongest advice to you is to ignore estimates and simply concentrate on the fact that FGRP4 tasks are each going to take around 15 hours to crunch on average on that machine. The 23 'in progress' tasks you currently have could be crunched in about 340 hours total - about 14 days using just 1 of your 8 threads. If you could stand to use 2 threads, you should have no problem with deadlines any more.

You just need to trust that FGRP4 tasks are going to take around 15 hours on your rig, no matter what the estimate says. If the estimate says 5 hours at the start, of course it will increase as BOINC gradually works out that it's really going to take a lot longer than the estimate. Why does BOINC start off with such a low estimate? Probably because a number of BRP5 tasks were completed at less than their respective estimates so with a single DCF affecting all searches, the estimates of all FGRP4 tasks will also have been reduced.

I hope some of the above is understandable to you. If not, it wont matter much as long as you set something like 1.0 days / 0.1 days for your two cache settings and that you share your resources sensibly between all your projects. Don't give one project 90% and share the balance between the rest. BOINC is likely to get in a real mess with that scenario :-).

According to your last scheduler contact you have the minimum buffer set to 10 days (864000 seconds divided by 60 makes 14400 minutes, divided by 60 makes 240 hours, divided by 24 makes 10 days):

The other use of the minimum buffer is to determine how many days offline Boinc is going to have, meaning Boinc in this case will try and cache 10 days work (probably 10+2 days),
and try and get work done 10 days early, all on a project with max deadlines of only 14 days, basically meaning you have four days to do ten days work.

Another thing to Note, Boinc 7.4.27 doesn't show tasks running in 'High priority' any longer, DA was concerned that new users would confuse that with thread priority and get worried,
So now tasks running in High priority just have a status of 'running', no-one managed to convince him to use an alternate word:

http://boinc.berkeley.edu/gitweb/?p=boinc-v2.git;a=commit;h=bca7c006deae20cc31be20fc37396bb4c0cfafc2

Claggy

Would you care to point to a single example of a GPU task that has run longer than 22 hours? I can't find any! I see the vast bulk of GPU tasks taking around 2.7 hours each with an odd one or two taking a lot longer - perhaps 15 hours. This is possibly just an artifact due to the GPU not being properly serviced by the CPU for some odd reason. You can be pretty sure that your GPU tasks will take only 2.7 hours or so, on average, no matter what the estimate claims.

Are you really talking about GPU tasks? At 2.7 hours real crunch time, the whole 27 you have left will take less than 3 days - nothing like the 25 days you mention. Until everything settles, you need to ignore the estimates when they differ so widely from reality.

As I previously suggested, and as Claggy has subsequently proved, you have too big a work cache. You appear to have a 10 + 2 = 12 day cache setting. There are two settings you need to examine on your computing preferences page. Because of the highly customised old server code being used at this project, the first one says, "Computer is connected to the Internet about every x.xx days", when it probably should say something like, "What is the minimum amount of work to be maintained". The second one says, "Maintain enough work for an additional x.xx days". The first setting acts as a 'low water' mark and the second as the extra on top to create a 'high water' mark. What two values do you have?

Sounds perfect.

No project can 'hog the machine', against your wishes. Think about it for a moment. Did the project force you to set a 12 day cache? The BOINC client tries to comply with your choice of settings. The BOINC client requests work. The project just passively responds to the request as best it can. If you apparently get too much work, BOINC (through panic mode) forces the Einstein app to run. The project has no control over this.

Do I need to explain this all again?? That setting is NOT a command to change to another task. It's simply the period after which BOINC will make a decision as to whether or not some other task should be started (or re-started from suspension) in place of the current one. If BOINC doesn't see a need to change, why are you worried about it? Now that we can assume that your machine is in panic mode (whether or not BOINC is showing that fact - thanks Claggy, I didn't know about that change - aarrrggghhhh!!!!), BOINC will keep the supposedly 'at risk' tasks running without change until the panic is over.

The project simply sends what it is asked for. You need to convince your BOINC client to ask for an appropriate amount.

Except for beta test runs, all BRP5, BRP4G, FGRP4, and Grav Wave (GW) tasks have a standard 14 day deadline.

You appear to be confusing the FGRP4 CPU tasks (which take around 15 hours) with BRP5 (Binary radio pulsar - Perseus Arm Survey) GPU tasks which seem to average around 2.7 hours. If you go back and re-read my previous message you can see I was talking about the FGRP4 CPU tasks taking 15 hours.

For the BRP5 GPU tasks, there are just a couple of examples of much longer run times than the normal 2.7 hours. This may well be a sign that you need to free up a CPU core for GPU support duties. To achieve that you could try setting the use of 87.5% of the available processors rather than the full 100% in your computing preferences.

Set 2.0 for the "Connect to internet ..." setting. Set 0.1 for the "Additional days ..." setting. Every time your work on hand drops below 2.0 days, BOINC will top up to 2.1 days.

George,

With the greatest of respect, I must say that I entirely agree, but probably for rather different reasons than the ones you have in mind.

It actually takes quite a long time to compose answers like the preceding ones and obviously I'm rather stupid to do so. I suffer from the delusion that if I attempt to explain the full details about what is likely to be happening to cause the issue, the recipient of the advice might actually try to digest it. I'm genuinely sorry if my delusion has caused you angst. I'll stop bothering you very shortly.

It's good that Claggy was smart enough to get the server log excerpt that finally proved to you just how much work your BOINC client had been asking for. However, if you want to see where credit should go, you should go back to the start of this thread and look at the very first response you received from archae86, who told you specifically:-

Local preferences are the ones you set for just the current host using BOINC Manager. If you are actually using website preferences which would apply to all hosts, he said:-

Archae86 had nailed the cause of your woes right there. The thing I don't understand is why you didn't read and act on the very clear advice given? If you didn't understand, why didn't you ask for clarification?

It really doesn't matter any more since you've got your work cache settings under control now.

At the risk of causing you further angst, I'm going to point out that you still have a further problem that you really need to address. I've pointed it out previously but I'm going to make one last effort to help. Referring specifically to BRP5 GPU tasks, your GTX670 has proved that it can do these in 2.7 hours. Go check all the validated tasks on the website for that host. Notice that the vast bulk of these took around 2.7 hours. Notice also that there were two tasks only that deviated from this and took around 15.5 hours. This is the problem that only you can fix.

You've been telling me that some tasks take longer than 2.7 hours and I've tried several times to explain why this might be happening and what you could try to correct it. The fact remains that your GPU can do these tasks in 2.7 hours and you need to make sure you stop the 'long running' behaviour.

There are a number of things that could be causing this - it could even be a driver issue. However the first thing to try is to make sure there is a CPU core available to support the GPU. It seems likely that the excessive time is being caused by the fact that all 8 virtual cores in your quad core host appear to be tied up crunching CPU tasks full bore.

As the GPU is local to this machine, you could change local preferences using BOINC Manager as a quick way of testing. Make sure you are in BOINC Manager Advanced View and select Tools -> Preferences and then select the Processors tab. Down towards the bottom of the page you will see a setting for the % of processors to use. It will be on 100%. Change it to 87.5% and click OK. If you then go to the event log you should see an entry showing the number of processors changing from 8 to 7. If you look at your tasks tab, you should see that there is a CPU task that was running that is now 'waiting to run'.

If you have a long running GPU task you can easily tell if it is now going faster. Every thing updates on a per second basis. If a task takes 2.7 hours to complete, that represents 0.01% progress per second. Long running behaviour seems to be about 15.5 hours. That represents just under 0.002% progress per second. So you just need to ask yourself the question, "Is the % done value for the running GPU task on the tasks tab under the Progress column jumping by 0.010% per second or 0.002% per second?" If it is still running at the slow rate, I would stop BOINC, reboot the machine and then restart BOINC. If it still continues to run slow, the issue is not lack of CPU support and we have to try looking elsewhere.

The next time you see a slow running GPU task, why don't you give the above procedure a try and report back on what you are seeing? I hope you do realise that your GPU is capable of 2.7 hour completion times and you just need to work out why sometimes it's getting stuck in the slow lane.