I was in bed and it suddenly dawned on me... There's another flaw in the logic being used to lower credits, specifically in regards to SAH "being in agreement" with other projects based on that chart...
The BOINC 4.x client underclaims
No matter what Ned says about the odds of pairing up with one of those clients, it would still be skewing the SAH project-wide claim/grant data downward.
I'm too tired to go into explaining all of it at the moment, but if you feel like it, go ahead...or if you, Bruce, feel like checking into it, that would be great too.
Well, first let me say I understand where Dr. Allen is coming from, and the chart he referred to does indicate what he was saying is true. Also, my experience in almost 2 years for crunching for EAH shows me he and the team don't make decisions which impact the project frivolously, and thinks long and hard before they do make a change.
I guess it's just I'm getting tired of every time a scoring changed is deemed 'necessary', that means we take it on the nose here and/or over at SAH. The bottom line is I couldn't care less what the other projects pay after a certain point. I've been willing to compromise up until now since BOINC is experimental and this whole scoring issue goes to the root of a problem which has existed for as long comparing performance between dissimilar computing platforms has been attempted.
OTOH, I have observed all of my hosts credit rate drop time after time while they continue to crunch away 24/7 doing the exact same science they always have, and that's just plain discouraging from a motivational POV. It raises the questions of what is the standard, is the whole concept even valid or viable, and where does it stop? When we're back to counting WU's run and CPU time burned?
One other observation is that without changing the intrinsic value of any WU, you can put EAH (using the old value) at a 25-30% rate disadvantage by merely running the Coop's 2.2B app on SAH. Since one could argue the point that S5R1(I) were gone over with roughly the same level of care as the Coop's 2x series apps to get the best performance possible, then EAH should up their rate. IOW that's the standard of 'quality' the other projects should be shooting for.
As far as 4x clients go, I would almost bet their effect is insignificant compared to the effect running the Coop apps have as far as the projects overall credit rate goes.
Alinator
@ Richard and Andy:
Yeah I remember the converstions and the concerns Eric had for inflation back then. Funny though how it seems the opposite is what's coming to pass. ;-)
At the time I thought the ideas Eric put forth in the post you referenced made the most sense (and still do).
Also as far stating S5R1 was the target for MB/AP I don't recall any of the project team saying that per se. It was my observation when I started getting similar rates as EAH on the ones my hosts ran. It might be coincidence, but one I thought was a good thing if not deliberate.
OTOH, I have observed all of my hosts credit rate drop time after time while they continue to crunch away 24/7 doing the exact same science they always have, and that's just plain discouraging from a motivational POV. It raises the questions of what is the standard, is the whole concept even valid or viable, and where does it stop? When we're back to counting WU's run and CPU time burned?
Good point... The credit scoring is a motivational factor, not just something for "credit whores"... I didn't really get enough work from LHC to be able to be 100% sure, but it seemed to me that LHC's work gets a pitiful amount of credit compared to other projects. They're not on that chart, so who knows.
Quote:
One other observation is that without changing the intrinsic value of any WU, you can put EAH (using the old value) at a 25-30% rate disadvantage by merely running the Coop's 2.2B app on SAH. Since one could argue the point that S5R1(I) were gone over with roughly the same level of care as the Coop's 2x series apps to get the best performance possible, then EAH should up their rate. IOW that's the standard of 'quality' the other projects should be shooting for.
Exactly. That's what I tried to say, but Dan Neely didn't seem to understand it. It's ok, because according to his sig he doesn't run that project, so he may not have any idea of what we're talking about.
Quote:
As far as 4x clients go, I would almost bet their effect is insignificant compared to the effect running the Coop apps have as far as the projects overall credit rate goes.
Bear in mind though that there are people running 2.2B with BOINC 4.45. I believe that means that they would significantly underclaim. Of course you then have all of those "calibrating" clients still out there too. :shrug:
Wait... it underclaims? And here I was, thinking that SETI has been giving fixed credit for at least a year...
To add to what Andy (Winterknight) said, there is some sort of proxy issue with the newer fpops counting clients that is making SETI make the decision to allow the older BOINC clients to submit results.
Wait... it underclaims? And here I was, thinking that SETI has been giving fixed credit for at least a year...
To add to what Andy (Winterknight) said, there is some sort of proxy issue with the newer fpops counting clients that is making SETI make the decision to allow the older BOINC clients to submit results.
SETI determines by FlOPS, but does not force the credits on server side. So, clients still can play with numbers, and send back false numbers. The quorum of 3/4 fixed most of this, but the 2/3 can now make it worse. Someone could really under-evaluate the work (somewhat like 4.x clients do), and cause many issues. Over-evaluation can be done, also if enough people install the fake client.
I wish SETI would go with server side crediting, like many others have decided to do, but because it is a testbed software for BOINC they need to allow many things to happen.
Well just don't mix the metaphor here. The IR has nothing to do with 'fixing' the bad claims, extra's beyond the quorum just waste resources since they are unnecessary from a science POV. It's the quorum of three which does the trick.
So where it's not strictly a science need to issue 3, it serves a useful purpose by helping to workaround a project limitation.
Also, server side works well here at EAH (and perhaps others) because the project has a much tighter protocol than SAH. Also the way the analysis is designed lends itself better to server side. I don't see any fundamental problem with the way SAH does it, as long as the known limitations are worked around.
Still using three 4.19 boxes for proxy reasons. They do not know how to send the flops count to the server but the 4.19 (windows) almost claims what Einstein S5R1 granted. One Linux machine I had to adjust - of course this kind of adjustment isn't much liked but underclaiming clients aren't much liked either and the 4.x Linux core client was known for its bad benchmark.
p.s.: There's still one major disadvantage of 4.19 - it does not know about locality scheduling so it can have ghost WUs, which have been mostly eliminated with locality scheduling.
SAH grant 25-50% more credits than EAH for my Xeon Cloverdale. I have only received one type of EAH WUs for this machine all with equal score. For SAH the score and WUs varies. The problem with all credit comparisons with SAH is that SAH is open source and there is competision of getting it most optimized. The SETI site recomends this and have link to sites where optimized code can be downloaded. For modern CPUs this represents a huge gain in speed.
For my AMD64 X2 this gets even worse. I get higher credits/core on SAH, but lower on EAH, than I get using the Xeon Cloverdale. The AMD had about 100% speed increase with optimized code.
I don't know the percentage of people using the optimized SETI code, I guess it is not high because I most people, like I did initionally, probably only expects a low gain using optimized code and don't want the extra hazzle of download and install. That is probably why the table is not useful for comparison with SAH stats.
Other projects have to take into consideration the optimized SAH code when trying to give equal credits.
I hoped EAH could compete with SAH on credits, but you redused it just before I started. I am not a SAH fan, but if I want the most boinc credits out of my machines I have to do SAH.
On my first S5R2 result I get about twice the credit per unit CPU time vs my last S5R1 result: 204 credits/ 292k s vs 54 credits/ 147k s. SuSE Linux 10.1 on a 400 MHz Pentium II, BOINC 5.8.17.
Tullio
PS On QMC@home i got 172 credits/ 294k s
This points out another issue with credits. When trying to match credits granted between two projects, it is only possible to make the credits match between one CPU achitecture family at any time, because each project use the different computational aspects of a CPU achitecure in different amounts. E.g. if you match the score for AMD64 X2s, it is only luck if the scores match for Intel C2Ds. I don't see any solution to this with the current score scheme.
The score should ideally have been designed so that the project doing the most useful science gave the highest score, but who can judge what is most useful science? I think the projects should give one cent to charity for each credit they grant, then they could have granted as many credits they wanted.
This would stop projects of having a replication of 4 or higher :)
RE: @Alinator: I was in
)
Well, first let me say I understand where Dr. Allen is coming from, and the chart he referred to does indicate what he was saying is true. Also, my experience in almost 2 years for crunching for EAH shows me he and the team don't make decisions which impact the project frivolously, and thinks long and hard before they do make a change.
I guess it's just I'm getting tired of every time a scoring changed is deemed 'necessary', that means we take it on the nose here and/or over at SAH. The bottom line is I couldn't care less what the other projects pay after a certain point. I've been willing to compromise up until now since BOINC is experimental and this whole scoring issue goes to the root of a problem which has existed for as long comparing performance between dissimilar computing platforms has been attempted.
OTOH, I have observed all of my hosts credit rate drop time after time while they continue to crunch away 24/7 doing the exact same science they always have, and that's just plain discouraging from a motivational POV. It raises the questions of what is the standard, is the whole concept even valid or viable, and where does it stop? When we're back to counting WU's run and CPU time burned?
One other observation is that without changing the intrinsic value of any WU, you can put EAH (using the old value) at a 25-30% rate disadvantage by merely running the Coop's 2.2B app on SAH. Since one could argue the point that S5R1(I) were gone over with roughly the same level of care as the Coop's 2x series apps to get the best performance possible, then EAH should up their rate. IOW that's the standard of 'quality' the other projects should be shooting for.
As far as 4x clients go, I would almost bet their effect is insignificant compared to the effect running the Coop apps have as far as the projects overall credit rate goes.
Alinator
@ Richard and Andy:
Yeah I remember the converstions and the concerns Eric had for inflation back then. Funny though how it seems the opposite is what's coming to pass. ;-)
At the time I thought the ideas Eric put forth in the post you referenced made the most sense (and still do).
Also as far stating S5R1 was the target for MB/AP I don't recall any of the project team saying that per se. It was my observation when I started getting similar rates as EAH on the ones my hosts ran. It might be coincidence, but one I thought was a good thing if not deliberate.
Alinator
RE: OTOH, I have observed
)
Good point... The credit scoring is a motivational factor, not just something for "credit whores"... I didn't really get enough work from LHC to be able to be 100% sure, but it seemed to me that LHC's work gets a pitiful amount of credit compared to other projects. They're not on that chart, so who knows.
Exactly. That's what I tried to say, but Dan Neely didn't seem to understand it. It's ok, because according to his sig he doesn't run that project, so he may not have any idea of what we're talking about.
Bear in mind though that there are people running 2.2B with BOINC 4.45. I believe that means that they would significantly underclaim. Of course you then have all of those "calibrating" clients still out there too. :shrug:
RE: Wait... it underclaims?
)
To add to what Andy (Winterknight) said, there is some sort of proxy issue with the newer fpops counting clients that is making SETI make the decision to allow the older BOINC clients to submit results.
RE: RE: Wait... it
)
SETI determines by FlOPS, but does not force the credits on server side. So, clients still can play with numbers, and send back false numbers. The quorum of 3/4 fixed most of this, but the 2/3 can now make it worse. Someone could really under-evaluate the work (somewhat like 4.x clients do), and cause many issues. Over-evaluation can be done, also if enough people install the fake client.
I wish SETI would go with server side crediting, like many others have decided to do, but because it is a testbed software for BOINC they need to allow many things to happen.
Well just don't mix the
)
Well just don't mix the metaphor here. The IR has nothing to do with 'fixing' the bad claims, extra's beyond the quorum just waste resources since they are unnecessary from a science POV. It's the quorum of three which does the trick.
So where it's not strictly a science need to issue 3, it serves a useful purpose by helping to workaround a project limitation.
Also, server side works well here at EAH (and perhaps others) because the project has a much tighter protocol than SAH. Also the way the analysis is designed lends itself better to server side. I don't see any fundamental problem with the way SAH does it, as long as the known limitations are worked around.
Alinator
Still using three 4.19 boxes
)
Still using three 4.19 boxes for proxy reasons. They do not know how to send the flops count to the server but the 4.19 (windows) almost claims what Einstein S5R1 granted. One Linux machine I had to adjust - of course this kind of adjustment isn't much liked but underclaiming clients aren't much liked either and the 4.x Linux core client was known for its bad benchmark.
p.s.: There's still one major disadvantage of 4.19 - it does not know about locality scheduling so it can have ghost WUs, which have been mostly eliminated with locality scheduling.
SAH grant 25-50% more credits
)
SAH grant 25-50% more credits than EAH for my Xeon Cloverdale. I have only received one type of EAH WUs for this machine all with equal score. For SAH the score and WUs varies. The problem with all credit comparisons with SAH is that SAH is open source and there is competision of getting it most optimized. The SETI site recomends this and have link to sites where optimized code can be downloaded. For modern CPUs this represents a huge gain in speed.
For my AMD64 X2 this gets even worse. I get higher credits/core on SAH, but lower on EAH, than I get using the Xeon Cloverdale. The AMD had about 100% speed increase with optimized code.
I don't know the percentage of people using the optimized SETI code, I guess it is not high because I most people, like I did initionally, probably only expects a low gain using optimized code and don't want the extra hazzle of download and install. That is probably why the table is not useful for comparison with SAH stats.
Other projects have to take into consideration the optimized SAH code when trying to give equal credits.
I hoped EAH could compete with SAH on credits, but you redused it just before I started. I am not a SAH fan, but if I want the most boinc credits out of my machines I have to do SAH.
On my first S5R2 result I get
)
On my first S5R2 result I get about twice the credit per unit CPU time vs my last S5R1 result: 204 credits/ 292k s vs 54 credits/ 147k s. SuSE Linux 10.1 on a 400 MHz Pentium II, BOINC 5.8.17.
Tullio
PS On QMC@home i got 172 credits/ 294k s
This points out another issue
)
This points out another issue with credits. When trying to match credits granted between two projects, it is only possible to make the credits match between one CPU achitecture family at any time, because each project use the different computational aspects of a CPU achitecure in different amounts. E.g. if you match the score for AMD64 X2s, it is only luck if the scores match for Intel C2Ds. I don't see any solution to this with the current score scheme.
The score should ideally have been designed so that the project doing the most useful science gave the highest score, but who can judge what is most useful science? I think the projects should give one cent to charity for each credit they grant, then they could have granted as many credits they wanted.
This would stop projects of having a replication of 4 or higher :)
Tullio is right, I found that
)
Tullio is right, I found that a P3 Coppermine running Linux
has nearly exactly the same time to credits ratio for S5R2
that it had for S5R1/S5Ri
So it might be just a question of a stronger akosification
for the Windows app.