BOINC and Einstein particularly don't play by the rules ...
12 Apr 2015 4:12:41 UTC
Topic 198053
(moderation:
Quote:
Actually BOINC should take into account your processing habits when fetching work and shouldn't get you work you can't finish within the deadline. If you're afraid you can't make 7d deadlines, you may de-select the BRP4G application. There's plenty of other (BRP6) work for your GPUs.
BM
But the problem is .... we all know BOINC and E@H particularly don't play by the rules of project resource priority and task loading. There is now way to allow E@H to continuously download tasks because for one:
1. It doesn't obey disk space allotment
2. It doesn't compute task completion within allotted timelines properly
3. It forces all higher priority projects into the background by running high priority to complete absurd task loads.
If you allow E@H to download at will, you will end up with tens of GB of data and 99% will be abandoned because they will never have a chance to finish in time. I only bring maybe 10 tasks on board at a time in a single download before I have to set NNT. It means I have to continuously monitor E@H while I can set and forgot my other projects to run without any intervention.
Why doesn't E@H play by the BOINC rules??
Keith
Cheers,
Gary.
BOINC and Einstein particularly don't play by the rules ...
)
Hello Keith! I'm crunching E@H almost all the time, working on Milky Way tasks on GPU (my GPU is broken and cannot work anymore for any project except MW) and sometimes I'm returning to SETI. But I don't see any problems you have described. Are you sure you have a small enough download cache in BOINC settings and set the disk limit for BOINC projects (also made in local BOINC project settings)?
Yes, I have a four day
)
Yes, I have a four day download cache which isn't too big for typical outages at SETI. I usually have the most onboard tasks allowed by all projects EXCEPT for Einstein. I have set disk limits globally and locally and every possible permutation. I have tried local venues globally and locally. I have tried everything. If I have set a maximum disk usage for Einstein at 50MB, it downloads 2 GB of tasks. I can hamstring SETI or MilkyWay globally or locally and they OBEY the settings. Einstein just ignores them. I have fought this issue ever since I started crunching for Einstein in 2011. It just doesn't play by the rules forcing me to constantly Allow New Tasks to download 5-10 tasks and while it is downloading quickly set No New Tasks before the download finishes or it will have already cycled to a new request for tasks. The only time I ever have had errors on Einstein is when I wasn't quick enough to set NNT or it didn't stick and I missed that fact and it constantly downloaded hundreds of tasks that I can never finish in time because of the 1 week deadline and I have to abandon them. MilkyWay is a set and forget project and never have more than the project restricted 80 tasks on board at any time. Of course since they finish in seconds, the computers are in constant communication with the project to maintain the quota. Same goes for SETI mostly as long as the project has work. Never a problem maintaining the 300 task quota per machine as long as work is available. I don't think Einstein has a quota set of any kind, it just keeps downloading until it maxes out my 2GB global BOINC disk usage. If I have SETI getting 98% of resource time and Milkyway getting 1% and Einstein getting 1%, you would think that BOINC and the project would figure out the maximum allowed work onboard and restrict the number of tasks downloaded that can be completed within the deadlines. It does not. I'm not the only one to have commented on this problem before. I have been dealing with it since 2011.
As I said, only project that gives me issues with the amount of work downloaded is Einstein.
Cheers, Keith
Keith, Can I remind you
)
Keith,
Can I remind you that deliberate thread hijacking is frowned on here. The thread you posted in, very clearly, was intended to elicit information about a sudden and unexpected release of new BRP4G tasks with a shortened deadline. You seemed to want to turn it into an Einstein (and perhaps BOINC) bashing exercise. This is quite off-topic and for that reason, I've moved the entire discussion about Einstein "not playing by the rules" to a new thread. You can regard yourself as the thread owner and make whatever further comments you wish in this new thread.
You are quite free to air whatever further reasonable grievances you might have about Einstein and BOINC but, for the purpose of getting some balance in your comments, please understand one undeniable fact. It is BOINC (controlled by the way you set your preferences and how you micromanage it) and NOT any individual project that decides what to download, when to download, how much to download, and how to prioritize the running of what has been downloaded. Projects don't have a say in this, so it is quite misleading to blame Einstein for not "playing by the rules" when the project doesn't "play" at all. It just supplies whatever work BOINC requests and that's the end of it. If BOINC is requesting far too much, you probably need to think about the real reasons for that and how you can make a better choice of preferences to achieve your aims - the project isn't forcing the work on you.
The OP of the thread you hijacked had a valid point to make. It was a light-hearted point about a whole bunch of work to be completed in 7 days. The point, put more bluntly, is that a sudden change in expected deadline can cause problems with orderly work completion at best or deadline misses at worst. It really would be nice to have advance warning of such changes, but that's not always possible. To his credit, the OP was just noting the fact and not complaining about it.
Cheers,
Gary.
RE: RE: Actually BOINC
)
Keith,
With all due respect, that statement implies that you don't (or perhaps more likely don't want to) understand what is happening and, as a consequence, you are blaming the one project that is actually not to blame for your stated experiences.
Despite your claims to the contrary, BOINC can be set to "continuously download tasks", which really means, "request work whenever it needs to", because that is what actually happens and what it is designed to do. If left alone to do its job, BOINC can request new work in very small increments of seconds or a few minutes, once the cache level falls slightly below the actual setting.
The real problem here is that the BOINC design was never able to cope with the very situations you want it to cope with, namely, (i) projects that often have work shortages and outages, (ii) projects that have severe restrictions on the number of tasks in play, and (iii) the proper honouring of resource shares when most of your projects are of type (i) or type (ii). The one thing that BOINC will always do (at least for the slightly older versions that I use) is fill your work cache with whatever it can at all times, even if all it can get is a low resource share project that you obviously don't want.
It's not much use blaming Einstein for this. I don't know anything about how the 'backup project' mechanism (resource share = 0) works as I've never used it, but it seems to me that you should have Einstein on zero resource share so that BOINC will ask for only one task per core and only ask for another when the first finishes and your other projects can't supply. Another possibility would be to set a quite low cache setting to keep Einstein tasks at a low level. Then, when your preferred project can supply, it should be the project that BOINC will prefer and you could increase your cache setting to what you want for the duration of task availability, setting it back to a low value when the drought begins again.
It's not helpful to make unreasonable claims. Einstein, for GW runs using Locality Scheduling, can have a number of very large data files that, over time can grow to several GBs (but not "tens of GB"). Other science runs have much more modest disk space requirements. The Locality Scheduling burden has been widely publicised and known about for many years. The advice has always been to deselect GW runs if you can't afford the disk space. You can't possibly need to abort 99% of tasks received. You say you have a 4 day cache setting. If you end up with all Einstein work, how many tasks do you actually get? Can't they all be finished in 4 days?
If you genuinely believe that these two quotes are factual (and not just a rant) and it is Einstein's fault and not to do with the way you have set your preferences, please give suggestions to the Admins on how they can change the project to prevent Einstein from screwing up your experience. I think you'll find you only have two real choices, (A) modify your preferences (or perhaps your aims if it can't be done with prefs) so that BOINC has a chance of working unattended, or, (B) convince the BOINC Devs to come up with a modified BOINC system that can cope with your requirements.
Perhaps you'd like to tell us exactly what these "rules" are and where they are documented, and perhaps how Einstein could play by them if it were to stop being so recalcitrant? I'd really like to see the list.
Maybe you'd like to consider these two "rules" (expectations really) that BOINC does have (tongue firmly in cheek) :-
1. Projects should be able to supply work when requested for a reasonably high proportion of the time and large unscheduled downtimes should be avoided whenever possible. Otherwise, it will make orderly life for participants and true unattended management, virtually impossible.
2. Projects should not (wherever possible) so restrict the supply of work to the extent that a participant can only get several minutes to perhaps an hour or two of work with which to fill their multi-day work caches. Otherwise, it will make orderly life for participants and true unattended management, virtually impossible.
Do you really think it's fair to blame Einstein for being reliable and non-restrictive with work supply?
Cheers,
Gary.
Thank you Gary for moving
)
Thank you Gary for moving this to a separate thread.
AFAIK there is one more way to get things back to normal - it is most likely that you have a problem with DCF (duration correction factor) and setting NNT doesn't allow BOINC to cope with this. Just allow BOINC to download as many tasks as it wants and let it crunch it all as it wants. I guess it will need maximum two weeks to deal with this, though some workunits may be lost due to deadline.
Well first I would like to
)
Well first I would like to state that this is the first time I was accused of "hijacking" a thread. I have never intentionally "hijacked" a thread. Where in the Einstein forum rules does it lay out what is acceptable or non-acceptable postings. The only "rule" that I know of is to avoid profanity or abuse of other posters. I do not think I attacked any of the posters in that original message. Second, it would have been courteous of the moderators to at least have left a message in the original thread that I was subscribed to that my posts had been moved to a separate thread. I was surprised to say the least to see my posts deleted without comment.
I already have dcf_debug set in the BoincManagers logging. Einstein is the only project I run that even uses DCF values any more. Seti stopped several years ago and I don't think MilkyWay ever did use DCF. It does change the estimated runtimes but it takes quite a few reportings to make significant changes. I really noticed it when I started the BRP6 tasks since I had never crunched them before.
I only crunch Einstein on the GPUs. Same for MilkyWay. Only SETI uses both CPU and GPUs.
Well I guess we could revisit this all over again. Especially if some expert could lay out EXACTLY what BOINC settings work for them which enables them to download Einstein work unattended. It would be helpful it that was from someone that also works other projects concurrently and doesn't have Einstein as their primary project, but only as a secondary backup project. I must have spent probably six months of experimentation back in 2011 in trying to get the systems to not download too much Einstein work. I had conversations with many members in trying different settings and scenarios to get BOINC to honor my settings and desires and enable me to configure my backup projects to work as I desired. As I stated, every setting I tried always ended up with too much Einstein work downloaded which always forced Einstein to commandeer my resources at 100% utilization and prevent my main project priority, SETI, to be shut down.
Granted, this was several years ago and the BOINC platform has changed since. From what I have read, BOINC is finally obeying a 0% resource share for backup projects and is apparently working for SETI users who dedicate 100% resource share to SETI and E&H work in never downloaded until the SETI project runs out of work. In my case, I WANT to work for multiple astronomy related projects, not just SETI. I actually just increased my resource share to be larger than for MW after the BRP6 work became available because it pays better for the amount of resource utilization it requires compared to MW. E@H is now set for 10% of resource utilization, double its historical settings.
So experts, just what BOINC setting will allow me to download E@H work that won't overcome my system and run me into inevitable deadlines?
Cheers, Keith
RE: RE: I only bring
)
OK, here is my request to the developers. Set maximum amount of tasks a host can have onboard at any time just as SETI and MW do. They work correctly with BOINC.
I guess maybe, at least in YOUR opinion that I was attacking Einstein. I did not intend that as the main message in my post. I was just "ranting" (in your words), on my frustration in the past of getting WAY too much work for Einstein that I never could complete in time due to WAY SHORT deadlines that the developers have set in place. The deadlines for SETI are way more reasonable. Maybe set deadlines according to resource share for example. That would make much more sense in my opinion.
Cheers, Keith
Well once again I have been
)
Well once again I have been foiled by BOINC. I once again tried to set venue preferences for Einstein and yet it seems to have affected SETI also in the default venue. I restricted E@H to 300MB of disk space for the Work venue. Assigned both computers to the Work venue for Einstein and left SETI and MW alone with the default venue and the 2GB disk space I have set for general BOINC settings. I saw a message in the log for E@H that seemed to indicate it was going to obey my settings and that it had met its GPU restriction
Pipsqueek
1817 Einstein@Home 4/12/2015 3:37:26 PM Not requesting tasks: don't need (CPU: ; NVIDIA GPU: job cache full)
I have set it to allow tasks and it hasn't downloaded anything new since I turned on the computer this morning and grabbed today's E@H workload.
Now, however I see this in the log for SETI:
Keith-Windows7
1186 SETI@home 4/12/2015 5:08:03 PM Reporting 28 completed tasks
1187 SETI@home 4/12/2015 5:08:03 PM Not requesting tasks: don't need (CPU: job cache full; NVIDIA GPU: job cache full)
1188 SETI@home 4/12/2015 5:08:06 PM Scheduler request completed
I am not at the project quota for SETI now, down about a 100 tasks. So it appears that setting the Work venue for Einstein has once again affected SETI. This is the BOINC settings page:
http://setiathome.berkeley.edu/prefs.php?subset=global&cols=1
So, could someone show me what I am doing wrong. How can I restrict just Einstein and not have it affect my other projects?
I am going to have to delete the venue again and set back to default and go back to NNT for Einstein so I can get work for my other projects.
Keith
RE: So experts, just what
)
The pair of settings which overall govern just how much work boinc on your host requests (often informally called queue size)
If you use a low enough value for the sum of these two (generic BOINC, not Einstein-specific) settings then your downloaded Einstein work will neither overcome your system nor run you into deadline problems unless your system is too tied up with work from other projects to attend to Einstein work.
RE: Well once again I have
)
On the account page for your Einstein (or your SETI) project account you have two links for preference settings. It is important for you to understand that one of the two is a generic BOINC preference which applies to ALL of your boinc projects, regardless of which account page you are looking at when you update it. The other, which is clearly labelled with the name of the project, contains project-specific settings.
Disk space is in the generic list, so is intended by the BOINC protocols to apply to all of your BOINC projects, not specifically the one from whose home page you revise it.