"Unsent" WUs

Winterknight
Winterknight
Joined: 4 Jun 05
Posts: 1221
Credit: 312299596
RAC: 650063

If Brian's figure of 61% of

If Brian's figure of 61% of hosts are using Windows then speeding up the Windows app to the same speed as the Linux/Mac apps and going by the published credits granted/day then approx 6,000 extra units would be processed/day.
That's the equivalent of ~500 extra core2 quads running Linux or Mac.

Or to put it another way, consistently twice the power of the recent input by Atlas that only used 1162 cores for a short period.

John Clark
John Clark
Joined: 4 May 07
Posts: 1087
Credit: 3143193
RAC: 0

It is interesting on this

It is interesting on this subject.

I see that some wingmen are returning units and the pending results numbers are starting to fall.

The reason I commented on the rise in pending was to see what the reasons were, and may be have a bit of a nagger.

But, after haveing a couple of WUs stuck in pending (another project) for 6 months before they cleared was nice. As these WUs are larger and more complex, I was hoping not to see a repeat of this situation. Especially as I was, until recently, only using the older rigs.

Shih-Tzu are clever, cuddly, playful and rule!! Jack Russell are feisty!

Brian Silvers
Brian Silvers
Joined: 26 Aug 05
Posts: 772
Credit: 282700
RAC: 0

RE: If Brian's figure of

Message 85558 in response to message 85556

Quote:

If Brian's figure of 61% of hosts are using Windows then speeding up the Windows app to the same speed as the Linux/Mac apps and going by the published credits granted/day then approx 6,000 extra units would be processed/day.
That's the equivalent of ~500 extra core2 quads running Linux or Mac.

Or to put it another way, consistently twice the power of the recent input by Atlas that only used 1162 cores for a short period.

It's not "my" figure. It came from BOINCstats.

I was guffawed at back during S5R2 when I dared mention the disparity between the platforms. Now, it is an agreed upon fact that there's a disparity.

It seems that people today don't realize that a small change in large numbers can make a big difference... You also have the intangible drawbacks of today, like exactly how many hosts have stopped processing because of the combination of reduced speed and reduced credit, how many decide to detach / abort tasks trying to find something that doesn't take as long, how many that get irritated that at their resource allocation "Einstein" (really BOINC) "hogs" their system and doesn't appear to honor resource allocations in the attempt to get the task processed by the deadline, etc, etc, etc...

Martin P.
Martin P.
Joined: 17 Feb 05
Posts: 162
Credit: 40156217
RAC: 0

RE: RE: .... It's gotten

Message 85559 in response to message 85523

Quote:
Quote:
.... It's gotten so bad that I dropped one of the cores and put it on another project until some of this is resolved....

I really don't understand why you think there is a "problem" when really it's just the way the locality scheduling system works. Eventually the scheduler will give this particular dataset to additional hosts and things will rapidly catch up. If you leave your cores the way they were, you'll get a nice credit boost when that happens.

Quote:
... Hope it gets straightened out soon.

Since there is nothing to "straighten out", your "hope" is bound to be dashed :-).

However, the scheduler will restore equilibrium in its own good time and with no intervention needed.

If it were that way, why does pending credit still grow and RAC for the project diminish from more than 16 Mio. credits/day to below 11 Mio. credits/day within the last month? And what is your timeframe for "Eventually..."? Is 1 month still too short for "Eventually..."?

Thunder
Thunder
Joined: 18 Jan 05
Posts: 138
Credit: 46754541
RAC: 0

The drop in RAC has

The drop in RAC has everything to do with;

a) no more 'power user' app paying a really large amount of credit for relatively little work
b) the overall adjustment that E@H did with credit.

Note that it's a 60 day graph. S5R4 started ~57 days ago. At that time there were still some using the S5R3 'power app' to wrap up a bit of work so the RAC won't adjust perfectly at 60 days, but it won't take much longer. Within the next week I would expect that graph to be about flat and you can see that it's already starting to do so.

So... yeah, 1 month is too soon to expect a BoincStats graph to adjust completely.

Personally, I don't see any of the ill effects that some are describing. My pending credit (adjusted for the changes in claimed credit) is about exactly the same as it was under S5R3 and S5R2. Actually, it's just slightly better than previous, but such a small difference as to be insignificant.

I agree that it would be great if Windows apps ran the same efficiency as Linux, but I have to agree with Gary here. It doesn't have any connection to the amount of 'unsent' workunits, which appears to not be terribly different from when S5R3 began.

Push for a new Windows app, but don't tie it in with locality scheduling, which appears to me to be working exactly as it's designed.

John Clark
John Clark
Joined: 4 May 07
Posts: 1087
Credit: 3143193
RAC: 0

Interesting question and

Interesting question and graph, Martin P.

Shih-Tzu are clever, cuddly, playful and rule!! Jack Russell are feisty!

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3522
Credit: 686138689
RAC: 562245

RE: .... If it were that

Message 85562 in response to message 85559

Quote:


....
If it were that way, why does pending credit still grow and RAC for the project diminish from more than 16 Mio. credits/day to below 11 Mio. credits/day within the last month? And what is your timeframe for "Eventually..."? Is 1 month still too short for "Eventually..."?

Hi!

Most of the drop in RAC is probably caused by the deliberate credit reduction (in combination with the re-packaging of the WUs) on the transition from S5R3 to S5R4. Because RAC is a moving-average, this is not visible as a clear cut drop at the time of the transition but rather as a stready decline in the RAC graph. Boincstats' daily awarded credit graph gives a clearer picture here,, not showing this steady decline:

Also there is no contradiction between rising pending credit and dropping RAC: Credit that is pending is credit that is NOT (yet) awarded after all. RAC is like "cash income" and pending credit is future "income" that the project owes...kind of.

I have a feeling tho that the current crisis on financial markets will not work in favor of people donating part of their electricity bill to BOINC projects, I'm wondering whether we will see a drop in participation across projects.

CU
Bikeman

Brian Silvers
Brian Silvers
Joined: 26 Aug 05
Posts: 772
Credit: 282700
RAC: 0

RE: I agree that it would

Message 85563 in response to message 85560

Quote:

I agree that it would be great if Windows apps ran the same efficiency as Linux, but I have to agree with Gary here. It doesn't have any connection to the amount of 'unsent' workunits, which appears to not be terribly different from when S5R3 began.

I'm going to assume that you do realize that there is a large difference between "unsent" here and "unsent" over at Cosmology. Cosmology's issue is HR-driven and/or project management ineptitude driven. Here, no matter how anyone wants to spin it, a component of a task being "unsent" is that there are no hosts available with the data set downloaded. The "sending" of the additional replication has to wait for a host with no files to come along or for the work to wrap up for the current data set and the host picks up that new data set along with a deletion request of the prior data set.

I will readily admit that I haven't paid full attention to all the ins and outs of what is different with S5R4, but locality scheduling and work distribution based on that schema will most definitely be delayed by a reduction in participation. A "reduction in participation" does not have to be the total inactivating of a host or a group of hosts. A slowdown will fit that description and will have a tendency to push out the delta between first replication and second replication, particularly if combined with a real physical reduction in the number of hosts (the hosts really do go inactive, like my AMD is currently inactive here).

I also agree with you that it doesn't appear that anything has truly changed all that much, just more of a perception of change and/or more sensitivity to it.

Charts and graphs are nice, but sometimes you can chart and graph yourself into "knowing for sure" what you're talking about and miss the forest for the trees. This is why, for example, in software development, you are supposed to have someone other than the developer test the application because the developer may develop convictions that things are working just fine and miss stuff that a fresh set of eyes may more easily catch.

As indicated, I encountered the same type of skepticism when I stated that the Linux S5R2 apps were significantly faster than the Windows S5R2 apps. Here we are a couple of years later and nobody doubts that anymore. I'm not saying that I'm 100% right, but I know I'm not 100% wrong...

John Clark
John Clark
Joined: 4 May 07
Posts: 1087
Credit: 3143193
RAC: 0

RE: I'm wondering whether

Message 85564 in response to message 85562

Quote:
I'm wondering whether we will see a drop in participation across projects.

Almost cretainly on the ball there, Bikeman

Shih-Tzu are clever, cuddly, playful and rule!! Jack Russell are feisty!

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 5842
Credit: 109405191288
RAC: 35450241

RE: RE: ... the scheduler

Message 85565 in response to message 85559

Quote:
Quote:
... the scheduler will restore equilibrium in its own good time and with no intervention needed.

If it were that way, why does pending credit still grow and RAC for the project diminish from more than 16 Mio. credits/day to below 11 Mio. credits/day within the last month?

Bikeman has answered your question about RAC but I'd like to add a couple of points.

The gap at the start of the daily credit graph corresponded to the period the project was offline for the server upgrade at the end of R3. The big spike on August 8 was the accumulation of several days of results whilst the server was offline. Once that spike was over, the daily credit for the following days gave an indication of the target area where the RAC would eventually have to drift to. Because of the big downward adjustment in the rate of credit for R4, as compared with R3, that drift would be substantial and would take quite a while to play out.

The first month or so of R4 results contained a mix of both new R4 tasks and outstanding and resend R3 tasks. The higher credit rate for R3 tasks is probably one of the reasons why the average daily credit for August is noticeably higher on the daily graph than the average for September. Factors like the startup of ATLAS would also have had an effect and it's possible that the "good" days around 6/7 September were associated with that. ATLAS was scaled back and so by the middle of the month you can see a fairly steady daily rate which now seems to have developed a bit of an upward trend. So despite the fact that RAC is still tending to decline, the true picture is that daily credit has been on a bit of an upward trend for the last two weeks or so.

If there were really a systemic pending credit blowout, you would expect to see corresponding drops in the daily awarded credit. No doubt some people are seeing increases in pendings but there probably are just as many others who are actually seeing decreases. Daily credit would also be sensitive to the number of active hosts. A week or two ago the figure was around 62K. It's now around 64K so this would be helping daily credit to rise.

Quote:
... And what is your timeframe for "Eventually..."? Is 1 month still too short for "Eventually..."?

Yes. "Eventually" is probably very much in the second half of the R4 run unless some scheduler tweaking is done.

At the moment there are simply too few hosts per data frequency band for the scheduler to be able to eliminate the disparity between the issue of the 1st and 2nd tasks for a quorum. As time goes on and frequency bands get completed, the scheduler will move the "newly released" hosts to the remaining bands and the greater supply of hosts per band will solve the behaviour. I'm sure it could be done now by limiting the available frequency bands like it was for R3. I suspect the project doesn't wish to waste manpower resources for what is essentially a cosmetic issue.

Cheers,
Gary.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.