While I've done well over 3000 SetiClassic WUs, I've done only 3 or 4 under Boinc, over 6 months ago, on the 4.19 Boinc client, so I can't recall how To Completion behaved for queued work.
As far as workunits in progress, Einstein progresses as linearly as can be on a %complete/time ratio, while Seti has always been extremely non-linear in that regard, and To Completion estimates varying all over the map during processing, quick at first 15% or so, moderate from there up to ~25%, noticably slower to beyond the 80% stage, and accelerating again toward completion.
microcraft
"The arc of history is long, but it bends toward justice" - MLK
There are 2 major changes in the to completion calculations in the 5.x.x clients. In my opinion both are definate improvements over the 4.xx version. However it does require at least one, and often a few, completed workunits under 5.x.x before they work correctly.
Durration correction factor - after a completed workunit this adjusts to more accuratly reflect how long your host processes a workunit.
The 'to completion' value is the larger of a weighted average of the original estimated time and the observed time, and the observed time.
It is done this way because downloading too much work is a problem, but downloading to little is an annoyance. Too much work can cause work to go over deadlines and have to be reissued by the project. Too little work just means a connection is needed sooner than expected.
There are 2 major changes in the to completion calculations in the 5.x.x clients. In my opinion both are definate improvements over the 4.xx version. However it does require at least one, and often a few, completed workunits under 5.x.x before they work correctly.
Durration correction factor - after a completed workunit this adjusts to more accuratly reflect how long your host processes a workunit.
The 'to completion' value is the larger of a weighted average of the original estimated time and the observed time, and the observed time.
It is done this way because downloading too much work is a problem, but downloading to little is an annoyance. Too much work can cause work to go over deadlines and have to be reissued by the project. Too little work just means a connection is needed sooner than expected.
John,
Thank you for weighing in on this with your explanation.
I've noticed the duration correction factor on my server summary - mine currently stands at 1.14866, which is nowhere near reflecting reality, and that value is even further from explaining the in-process inflation of nearly 60%.
I will provide some hard figures here. My longest WU time under 5.2.1 is 19379.75 secs. The remaining 9 or 10 WU times are between 18505 and 18675. By my calculation, the variance is less than 5%. For example, I give my current in-process WU, 2344875. At the 4 hour mark, 76.78% complete, projecting to completion time of 18754.88, and leaving 4354.88 sec, or 1 hr 12 min to completion. The manager displays 2 hr 9 min to completion, and it knows it's incorrect, because it's decrementing to completion time by 9 sec for every 5 sec real time, the interval between display updates!
Edit - Update - The WU cited above, 2344875, was completed in 18620.55, 5hrs 10min,20.55sec.
Edit complete.
Geez, I hate to give you guys headaches, but that algorithm needs to be reworked. Or, how about this - allow that algorithm to remain, for the scheduler, but leave the 4.x algorithm for display purposes, since it was essentially right on-target with it's values, throughout the entire processing time, while the new one is incorrect, in degrees varying from slight to gross, over the entire duration. If that cannot be done, then I suggest doing away entirely with the To Completion value on the manager, at least for WU in-progress, because it simply isn't true, or even close to truth.
I'm trying to help here, really.
microcraft
"The arc of history is long, but it bends toward justice" - MLK
The desire to error on the side of caution is behind the overestimations and the fact that the duration correction factor will rise quickly but fall slower. Having the client go into EDF mode too soon will be annoying but better than starting too late and missing deadlines.
Some of the other projects, most notablly SETI, have non-linear processing times. This is why the estimate is averaged and held larger than might be expected.
The desire to error on the side of caution is behind the overestimations and the fact that the duration correction factor will rise quickly but fall slower. Having the client go into EDF mode too soon will be annoying but better than starting too late and missing deadlines.
Some of the other projects, most notablly SETI, have non-linear processing times. This is why the estimate is averaged and held larger than might be expected.
Hi John,
I understand the motive behind the DCF, and I have no problem with that as a solution to overloading and missing deadlines. In fact, I suspected the rationale behind the increase in the beginning of this thread. Do whatever needs to be done to alleviate that problem, just PLEASE do not *display* the falsified information. Einstein, unlike other apps, is a perfectly linear process, and unlike on the non-linear projects, the To Completion times have always had meaning and veracity.
My suggestion in the final paragraph of my reply to your previous post is still my bottom line. Whether the To Completion display has any meaning to users of the other apps or not, it does here. We are accustomed to seeing valid information; so if you can find no way to serve your purposes and also give us the truth, it would be preferable to receive no To Completion info than to be lied to.
Respects
microcraft
"The arc of history is long, but it bends toward justice" - MLK
I know of one lady living in a tech-poor part of the world who is absolutely thrilled to be successfully crunching Einstein, and eagerly monitors and reports to us her progress. Though an Einstein WU takes *98 hours* on her slow PC, and she has had to go through a great deal of updating and tweaking her machine, not to mention the learning curve to geekdom, it provides her a feeling of accomplishment to be contributing to science. 98 hours, and she counts them down. How do you think it will make her feel when, after several days she has managed to accumulate 60 hours of CPU time toward a WU despite random power outages, she looks at To Completion and it tells her that 70-80 MORE hours remain. Would you say "Discouraged" just about sums it up? Is she supposed to take it on faith that when her Boinc tells her 80 hours, it really MEANS only 35?
Isn't it possible to use your compensation factor for the scheduler while simultaneously using the accurate algorithm for the Manager UI? I suspect that it is, though I can't honestly say I know, my programming education being in SPS, COBOL, RPG (not RPG2, but the original), and FORTRAN for the IBM 360-40, and now lost for lack of use. How about giving it a shot?
microcraft
"The arc of history is long, but it bends toward justice" - MLK
I know of one lady living in a tech-poor part of the world who is absolutely thrilled to be successfully crunching Einstein, and eagerly monitors and reports to us her progress. . . . How do you think it will make her feel when, after several days she has managed to accumulate 60 hours of CPU time toward a WU despite random power outages, she looks at To Completion and it tells her that 70-80 MORE hours remain. Would you say "Discouraged" just about sums it up? Is she supposed to take it on faith that when her Boinc tells her 80 hours, it really MEANS only 35?
Sorry - my overactive trigger finger again. Let me finish what I started below:
Michael,
OK, I admit it. I said would "go away" and I didn't. But I just couldn't resist saying something here. One of the things I enjoy about reading your posts is your obvious passion for the subject matter. You want things to work right and love the challenge of helping to improve things. But as you admitted to Gary on another post recently, you also tend to rant on occasion. And, this may be one of those occasions. I think using Ariane to bolster your point of view is a bit much. Think about it, explaining the subtleties of this issue to her would provide the excuse to generate a multitude of new posts - which we all enjoy reading. Anyway, I suggest you "give it a rest" on this issue for a while. You've made some very goods points. I know I've learned a lot. And, I think John might appreciate it if "you let it go". As Gary said: "Now, now Michael.
OK, I admit it. I said would "go away" and I didn't. But I just couldn't resist saying something here. One of the things I enjoy about reading your posts is your obvious passion for the subject matter. You want things to work right and love the challenge ofdesire to do things right another thread recently, you admitted "ranting"
Good evening, Stick
Go away? I'm glad you haven't. I think you and I and hordes of others, including all who've posted on this thread, have something inside us that needs things to work right, for everyone, and we're quite reluctant to accept "good enough" for "most people" as satisfactory.
I am by nature an optimizer, it's that simple. It drives me, giving me energy and exhausting it, in turn. Years before I first posted to a message board, I was a devoted user, cruncher. These millions of crunchers contribute far more time and energy to these Boinc projectsthan you, I, the developers, and all the message board posters combined. By and large, they are adults, fully capable of managing their accounts intelligently, given simply knowledge. They don't want or need to be second-guessed or deceived into doing what is right. That is what I consider is happening with this aspect of Boinc, deception by design. I think it's wrong, I think the underlying assumption is insulting, and i feel strongly, yes passionately about it.
You have been very busy lately, devoting large amounts of time and energy in pursuit of solving problems, making people's experience with Einstein less stressful, less confusing, more productive and more satisfying.
The developers work as hard and as long as any of us, to much the same ends. Whew, but I'd love to own the coffee concessions by their workareas.
So here we all are, trying to make it work right, and we feel strongly about what IS right, and how we can get to that point. We just see it from different perspectives.
microcraft
"The arc of history is long, but it bends toward justice" - MLK
Stick, While I've done
)
Stick,
While I've done well over 3000 SetiClassic WUs, I've done only 3 or 4 under Boinc, over 6 months ago, on the 4.19 Boinc client, so I can't recall how To Completion behaved for queued work.
As far as workunits in progress, Einstein progresses as linearly as can be on a %complete/time ratio, while Seti has always been extremely non-linear in that regard, and To Completion estimates varying all over the map during processing, quick at first 15% or so, moderate from there up to ~25%, noticably slower to beyond the 80% stage, and accelerating again toward completion.
microcraft
"The arc of history is long, but it bends toward justice" - MLK
There are 2 major changes in
)
There are 2 major changes in the to completion calculations in the 5.x.x clients. In my opinion both are definate improvements over the 4.xx version. However it does require at least one, and often a few, completed workunits under 5.x.x before they work correctly.
Durration correction factor - after a completed workunit this adjusts to more accuratly reflect how long your host processes a workunit.
The 'to completion' value is the larger of a weighted average of the original estimated time and the observed time, and the observed time.
It is done this way because downloading too much work is a problem, but downloading to little is an annoyance. Too much work can cause work to go over deadlines and have to be reissued by the project. Too little work just means a connection is needed sooner than expected.
BOINC WIKI
BOINCing since 2002/12/8
RE: There are 2 major
)
John,
Thank you for weighing in on this with your explanation.
I've noticed the duration correction factor on my server summary - mine currently stands at 1.14866, which is nowhere near reflecting reality, and that value is even further from explaining the in-process inflation of nearly 60%.
I will provide some hard figures here. My longest WU time under 5.2.1 is 19379.75 secs. The remaining 9 or 10 WU times are between 18505 and 18675. By my calculation, the variance is less than 5%. For example, I give my current in-process WU, 2344875. At the 4 hour mark, 76.78% complete, projecting to completion time of 18754.88, and leaving 4354.88 sec, or 1 hr 12 min to completion. The manager displays 2 hr 9 min to completion, and it knows it's incorrect, because it's decrementing to completion time by 9 sec for every 5 sec real time, the interval between display updates!
Edit - Update - The WU cited above, 2344875, was completed in 18620.55, 5hrs 10min,20.55sec.
Edit complete.
Geez, I hate to give you guys headaches, but that algorithm needs to be reworked. Or, how about this - allow that algorithm to remain, for the scheduler, but leave the 4.x algorithm for display purposes, since it was essentially right on-target with it's values, throughout the entire processing time, while the new one is incorrect, in degrees varying from slight to gross, over the entire duration. If that cannot be done, then I suggest doing away entirely with the To Completion value on the manager, at least for WU in-progress, because it simply isn't true, or even close to truth.
I'm trying to help here, really.
microcraft
"The arc of history is long, but it bends toward justice" - MLK
Michael and John, Thank
)
Michael and John,
Thank you! I think I understand the issue better now (and, I'll go away - at least until I give the new 5.2.x a try.)
Stick
The desire to error on the
)
The desire to error on the side of caution is behind the overestimations and the fact that the duration correction factor will rise quickly but fall slower. Having the client go into EDF mode too soon will be annoying but better than starting too late and missing deadlines.
Some of the other projects, most notablly SETI, have non-linear processing times. This is why the estimate is averaged and held larger than might be expected.
BOINC WIKI
BOINCing since 2002/12/8
RE: The desire to error on
)
Hi John,
I understand the motive behind the DCF, and I have no problem with that as a solution to overloading and missing deadlines. In fact, I suspected the rationale behind the increase in the beginning of this thread. Do whatever needs to be done to alleviate that problem, just PLEASE do not *display* the falsified information. Einstein, unlike other apps, is a perfectly linear process, and unlike on the non-linear projects, the To Completion times have always had meaning and veracity.
My suggestion in the final paragraph of my reply to your previous post is still my bottom line. Whether the To Completion display has any meaning to users of the other apps or not, it does here. We are accustomed to seeing valid information; so if you can find no way to serve your purposes and also give us the truth, it would be preferable to receive no To Completion info than to be lied to.
Respects
microcraft
"The arc of history is long, but it bends toward justice" - MLK
I know of one lady living in
)
I know of one lady living in a tech-poor part of the world who is absolutely thrilled to be successfully crunching Einstein, and eagerly monitors and reports to us her progress. Though an Einstein WU takes *98 hours* on her slow PC, and she has had to go through a great deal of updating and tweaking her machine, not to mention the learning curve to geekdom, it provides her a feeling of accomplishment to be contributing to science. 98 hours, and she counts them down. How do you think it will make her feel when, after several days she has managed to accumulate 60 hours of CPU time toward a WU despite random power outages, she looks at To Completion and it tells her that 70-80 MORE hours remain. Would you say "Discouraged" just about sums it up? Is she supposed to take it on faith that when her Boinc tells her 80 hours, it really MEANS only 35?
Isn't it possible to use your compensation factor for the scheduler while simultaneously using the accurate algorithm for the Manager UI? I suspect that it is, though I can't honestly say I know, my programming education being in SPS, COBOL, RPG (not RPG2, but the original), and FORTRAN for the IBM 360-40, and now lost for lack of use. How about giving it a shot?
microcraft
"The arc of history is long, but it bends toward justice" - MLK
RE: I know of one lady
)
Sorry - my overactive trigger finger again. Let me finish what I started below:
Michael,
OK, I admit it. I said would "go away" and I didn't. But I just couldn't resist saying something here. One of the things I enjoy about reading your posts is your obvious passion for the subject matter. You want things to work right and love the challenge of helping to improve things. But as you admitted to Gary on another post recently, you also tend to rant on occasion. And, this may be one of those occasions. I think using Ariane to bolster your point of view is a bit much. Think about it, explaining the subtleties of this issue to her would provide the excuse to generate a multitude of new posts - which we all enjoy reading. Anyway, I suggest you "give it a rest" on this issue for a while. You've made some very goods points. I know I've learned a lot. And, I think John might appreciate it if "you let it go". As Gary said: "Now, now Michael.
Still your friend (I hope!),
Stick
RE: OK, I admit it. I
)
Good evening, Stick
Go away? I'm glad you haven't. I think you and I and hordes of others, including all who've posted on this thread, have something inside us that needs things to work right, for everyone, and we're quite reluctant to accept "good enough" for "most people" as satisfactory.
I am by nature an optimizer, it's that simple. It drives me, giving me energy and exhausting it, in turn. Years before I first posted to a message board, I was a devoted user, cruncher. These millions of crunchers contribute far more time and energy to these Boinc projectsthan you, I, the developers, and all the message board posters combined. By and large, they are adults, fully capable of managing their accounts intelligently, given simply knowledge. They don't want or need to be second-guessed or deceived into doing what is right. That is what I consider is happening with this aspect of Boinc, deception by design. I think it's wrong, I think the underlying assumption is insulting, and i feel strongly, yes passionately about it.
You have been very busy lately, devoting large amounts of time and energy in pursuit of solving problems, making people's experience with Einstein less stressful, less confusing, more productive and more satisfying.
The developers work as hard and as long as any of us, to much the same ends. Whew, but I'd love to own the coffee concessions by their workareas.
So here we all are, trying to make it work right, and we feel strongly about what IS right, and how we can get to that point. We just see it from different perspectives.
microcraft
"The arc of history is long, but it bends toward justice" - MLK
RE: ... living in a
)
Quebec City is tech-poor????