My rant: Why you should care about the credit-system

ledi
ledi
Joined: 7 Mar 06
Posts: 15
Credit: 260,202
RAC: 0

RE: Personally, I'd settle

Message 45330 in response to message 45328

Quote:
Personally, I'd settle for one point per valid WU completed. Long, short or whatever, you finish one WU, it validates, and you get one point. What more do you need? I never, and still don't, understand why anyone felt the need to "level the playing field". You use the 'puters you have and do as many as you can in any given time period.


One of my computers is crunching a BBC-CPDN WU which will keep a 3 Ghz machine busy for almost 3-4 months. Do you think I will ever get a second WU if I only get 1 point for all this work ?

I am Homer of Borg. Prepare to be ...ooooh donuts!


Scott Brown
Scott Brown
Joined: 9 Feb 05
Posts: 38
Credit: 215,235
RAC: 0

RE: Sorry for my sloppines.

Message 45331 in response to message 45327

Quote:

Sorry for my sloppines. I actually didn't distinguish between psychology and sociology, and didn't mean to attribute any of their work as being non-scientific. I was writing my reponse to Alexanders "rant", picking up his understanding of "scientific" (in terms of "science the project is about"). I should have been more careful, and surely your post would have pointed me in that direction - It was just that I was busy writing my response while you were posting yours.

So thanks for the clarification. I think that apart from my sloppy and incorrect use of terminology we aren't that much apart, and I fully agree with you in that the "social science of DC" hasn't got enough attention yet. SETI (and BOINC) has been started by computer scientists, which are rather focused on technical issues, and which are continously surprised by the social aspects of the projects, the participation, the engagement of the participants in every direction. The currently most burning question for me is what way of granting credit (or reward of any kind) would be the most helpful for DC in general and BOINC projects in particular. However, this again kind of reduces a social question (of fairness) to a technical aspect - I'm a computer scientist, after all.

BM

Thanks for your reply, and I'd also add an "I'm sorry" for the bit of venom that might be gleaned from my post. Having stated such points multiple times across several years to the UCB folks, it is not easy to repeat myself without being someitmes abrupt. While I am glad that you and I are pretty much on the same page, I think that you shouldn't have to be the one to figure out this mess regarding fair rewards--that should be the responsibility of the creators of BOINC.

Nevertheless, since you are tackling this issue anyway, I would argue that you cannot 'solve' this question in a technical manner. Without understanding what participants view as 'fair', it is impossible to address the technical aspect of fairness implementation. Ideally, one would conduct a survey of participants to understand this, but given the obvious funding issues involved, this may not be feasible. Still, some information can be taken from the various forum debates across different projects that may be informative for deriving the technical solution to the issue. For example, I have seen at least four statements of what is or is not 'fair credit':

1. The one credit for one workunit proposition
2. The counting computer time or effort position (with various forms)
3. The counting contribution to the project effort position
4. Cross-project comparability is impossible position

Each of these positions is in disagreement with some (or even all) of the other positions. However, most, if not all, share some fundamental characteristics. For example, the first three at least seem to have in agreement with each other the need for consistency (i.e., the same credit for the same amount of work, regardless of how 'amount of work' is defined). What also seems clear is that participants agree that there are multiple ways in which to contribute (and thus be 'rewarded') for their contribution(s). Position 4 seems to be the ultimate statement of this view in that it holds that different projects reward these different dimensions of contribution differentially.

Other points of similarity among these can almost certainly be seen, but the mere fact that there are multiple positions among the particpants about what fairness is (and that each position seems to be held by a significant proportion of the participant-base) may point to how to implement a technical solution. That is, if participants disagree about which of the systems that they have discussed is fair, it is unlikely that a single credit system solution can be found. Thus, one must then question why a single system is being sought as the 'answer' to the problem. Why not implement a credit system that measures different dimensions?

For example, on CPDN, one might implement both the fixed cobblestone credit system and a completed workunit count. While the cobblestones would be comparable across projects (assuming an appropriate standardization exists) and would measure compter time (or effort), the completed workunit count would be a non-cross-project measure that shows the 'contribution to project comopletion'. Here at E@H, this might not be the best solution since cobblestones are not rewarded for workunits that do not complete. Still, a variant of this system might count the number of completed long and completed short workunits separately (I think the database already includes an indicator of these for sending the shorter ones to slower machines) as a measure of project contribution in addtion to the cross-project comparable cobblestones. This might also address the nagging issues surrounding application optimization and credit reductions since, though cobblestones would be adjusted down to maintain the cross-project comparability, the workunit count would necessarily increase to represent the inceased contribution to the project.

Other dual (or multi-) systems may be possible or better in each separate project or across all projects. The point here is that, by listening and analyizing the discussion and disagreements regarding the fairness of credits, the orientation to a unidimensional reward system is shown to likely be flawed and opens up the possibility of exploring multidimensional systems.

gopher65
gopher65
Joined: 18 May 05
Posts: 2
Credit: 2,281,535
RAC: 0

RE: The currently most

Message 45332 in response to message 45327

Quote:
The currently most burning question for me is what way of granting credit (or reward of any kind) would be the most helpful for DC in general and BOINC projects in particular. However, this again kind of reduces a social question (of fairness) to a technical aspect - I'm a computer scientist, after all

If you are asking why people start using BOINC (or other DC projects) and continue to use it, there are a few small things I'd like to note about my own tendencies:

1) I don't like it when my computer is just sitting here being useless. BOINC projects keep it active, and give me a reason not to continually start and shut down my computer every time I start or stop actively using it. This way I can leave it running for 2 or 3 hours while I go do something else without feeling guilty.

2) I started SETI@HOME because it sounded cool. When they switched to BOINC I switched too. Then several more projects came out in rapid succession, and I chose the one that I liked the most. Gravity waves sounded neat, so I chose this project.

3) Competing credit wise might entice me to put BOINC on more computers, but it won't help keep me with the project if I tire of it (which is why I am no longer attached to SETI@HOME).

4) BOINC should automatically run upon windows start-up. If it doesn't, people will install BOINC, attach to a project, and then forget about it the next time they turn on their computer (before they can become a credit-whore;-]).

5) I was going to say something else here but I forgot what it was.

6) Credit might initially keep me interested, but remember, I start projects because I think they are doing cool work. If they fail to regularily produce meaningful public results and updates (once every 2 weeks or so it is imperative to post a note on the front page (and if possible, on a "message from your project creator" tab within BOINC itself. There is no such tab right now, but it should exist), I will likely drop the project altogether as a waste of time.

EDIT: My first post;)

Metod, S56RKO
Metod, S56RKO
Joined: 11 Feb 05
Posts: 135
Credit: 812,347,933
RAC: 68,430

RE: So instead of Einstein

Message 45333 in response to message 45329

Quote:
So instead of Einstein dropping credit, all other projects should raise their granted credit to align with a better project.

This would cause another insult to some people. Consider this example:

I've got a moderately fast computer working on a project for a year. During this time I accumulated, say, 1000 cobblestones. Another user has got a really fast machine (twice as fast as mine) and starts participating in the same project today. This way we'd have equal amount of cobblestones (2000) in another year.
If the project increased number of cobblestones per work done, say by 25% , my already earned cobblestones wouldn't be worth a year of work anymore but less. Basically, the other user would catch me up (at 2000 cobblestones) in 292 days (instead of 365). Or the other way around: my work so far (one year) would loose in value of 2,5 months.

In short: change in value of a WU only makes even project internal crediting not fair.

The only problem with recent credit changes in Einstein I see is that it penalises some users: those owning machines that can not benefit from recent optimisations in scientific software.

Metod ...

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,276
Credit: 245,631,742
RAC: 11,173

RE: The only problem with

Message 45334 in response to message 45333

Quote:
The only problem with recent credit changes in Einstein I see is that it penalises some users: those owning machines that can not benefit from recent optimisations in scientific software.


Well, the gap between the two types of machines (actually there are more than two) is opened by the optimization, which, I think, isn't seen to be bad by anyone.

The reduction of credit within the project makes this look like a penalty to some participants, which it actually isn't from a different viewpoint - from this it simply doesn't matter if some hosts get the same and others get more credit, or some get less and others get the same.

In (our) reality, this actually even comes out as some getting less and others are getting more, because it's the average that stays (more or less) constant.

BM

BM

[AF>HFR>RR] Black Hole Sun
[AF>HFR>R...
Joined: 14 Mar 05
Posts: 31
Credit: 890,358
RAC: 0

RE: RE: The only problem

Message 45335 in response to message 45334

Quote:
Quote:
The only problem with recent credit changes in Einstein I see is that it penalises some users: those owning machines that can not benefit from recent optimisations in scientific software.

Well, the gap between the two types of machines (actually there are more than two) is opened by the optimization, which, I think, isn't seen to be bad by anyone.

The reduction of credit within the project makes this look like a penalty to some participants, which it actually isn't from a different viewpoint - from this it simply doesn't matter if some hosts get the same and others get more credit, or some get less and others get the same.

In (our) reality, this actually even comes out as some getting less and others are getting more, because it's the average that stays (more or less) constant.

BM


Hi Bernd

Could we have a dedicated sticky topic to discuss on credit / hour matter please ?
It is obvious that there are many people concerned with this matter, and as for now, we have several running discussion over several topics.
A dedicated topic could also be useful to Bruce for annoucing new credit "average calibration" when it applies (please, Bruce, DON'T reduce it again)

Thanks

Alexander W. Janssen
Alexander W. Janssen
Joined: 20 Feb 05
Posts: 56
Credit: 4,543,686
RAC: 0

RE: In DC, there are

Message 45336 in response to message 45326

Quote:
In DC, there are essentially three components to get it working and get it working effeciently:
[...]
3. the social science of DC (e.g., why people participate (or don't)? Why they stop participating? etc.).


Well, UCB conducted a survey which I also quote in parts; I also read elsewhere that you basically found that survey kind of useless because the questionary itself was not meaningful enough - from a scientific point of view.

I can only imagine how a scientifically good survey would look like. I just know the "employee satisfaction surveys" my company make every thre months: The question are all kind of strange and awkward, all the time - i suspect that there's a system behind it which tells those people more about the "collective satisfaction" than i possibly could imagine.

So, would you say that conducting a scientifically constructed (but maybe awkward) survey would push the understanding of what the participants (let's stick to this term) want/adore/dislike ahead? If yes, is it possible to create such a survey on our own according to "scheme F" or would you suggest to hire people for this?

Cheers, Alex.

P.S.: Sorry that it took so long, but work kept me from away from my terminal.

"I am tired of all this sort of thing called science here... We have spent
millions in that sort of thing for the last few years, and it is time it
should be stopped."
-- Simon Cameron, U.S. Senator, on the Smithsonian Institute, 1901.

Alexander W. Janssen
Alexander W. Janssen
Joined: 20 Feb 05
Posts: 56
Credit: 4,543,686
RAC: 0

Bert Swart

Message 45337 in response to message 45329

Bert Swart wrote:

Quote:
Jim Bailey wrote:
Quote:
Personally, I'd settle for one point per valid WU completed.

Probably a lot of people would abort the long ones.


I totally agree with Bert. That's what people would do.

Quote:
I am curious, what you mean with to "be as fair as possible with the credits", Alexander...


First of all i want to point out that i din't point on any project, caliming "OMG! Their credit-system is totally fscked up!". The fact that the projects Einstein and Rosetta appeared in my article was that I'm reading the fori of those projects and that I'm participating. My rant was more about communication, not about the fairness of a certain credit-system. However, i opened the can of worms so i have to show my colours.

My personal opinion is about "scientific work done" - not about FLOPS or MIPS. MIPS and FLOPS are a fixed value of your CPU and OS but more effective algorithms can produce more results per instructions. If you got an algorithm running in O(n) and one running O(log(n)) there's no discussion about which algorithm will do more work per FLOPS or MIPS.
So using MIPS and FLOPS is *no* benchmark at all IMHO.

Participants should be rewarded of "contributed scientific work".

But i also got one point: Lot's of people are dedicating their oldish machines to projects - the only very existance of those "vintage machines" is to crunch work-units for the project. Slow, but steady. That's something which really should be rewarded, it's a sign of DEDICATION.

That's just my personal opinion.

Quote:
[...] I'd say, work on a continuous calibration, to keep average credit level for every day for all projects, instead only after an improvement of the apps for a single project.


Sounds like a request for a standard-workunit? Even if i got you wrong, it still sounds like a good idea though...

Cheers, Alex.

"I am tired of all this sort of thing called science here... We have spent
millions in that sort of thing for the last few years, and it is time it
should be stopped."
-- Simon Cameron, U.S. Senator, on the Smithsonian Institute, 1901.

Alexander W. Janssen
Alexander W. Janssen
Joined: 20 Feb 05
Posts: 56
Credit: 4,543,686
RAC: 0

Black Hole Sun

Message 45338 in response to message 45335

Black Hole Sun wrote:

Quote:
A dedicated topic could also be useful to Bruce for annoucing new credit "average calibration" when it applies


I'd rather to have that on the frontpage in the news. Really. It's not much used anyway, example: Last mention of S4 was on 6/16, but no mention when a S4-report will be available.
I know people who let their bunch of workstations crunch for a project and they're just visiting the homepage every now and then - especially in situation when the RAC drops by a significant amount.
I wouldn't expect him to dig himself through all the threads until he found a little note that there was some recalibration.

So I kindly request: Up to the frontpage with it, including a pointer to the discussion-thread.

Alex.

"I am tired of all this sort of thing called science here... We have spent
millions in that sort of thing for the last few years, and it is time it
should be stopped."
-- Simon Cameron, U.S. Senator, on the Smithsonian Institute, 1901.

Alexander W. Janssen
Alexander W. Janssen
Joined: 20 Feb 05
Posts: 56
Credit: 4,543,686
RAC: 0

gopher65 wrote: RE: (and

Message 45339 in response to message 45332

gopher65 wrote:

Quote:
(and if possible, on a "message from your project creator" tab within BOINC itself. There is no such tab right now, but it should exist)


Now that would be a cool feature. Rom Walton pretty much summed it up in his latest rambling, where he said that customization of the BOINC-software ain't bad but can improve usability (the article is about something entirely else, but he also talks about the BBC-branded BOINC-software).

Quote:
EDIT: My first post;)


Oh, welcome to the club of the crazy and weird people ;) (just kidding)

Cheers, Alex.

Edit: Rom's posting is at http://www.romwnet.org/dasblogce/PermaLink,guid,5ad7b46a-6616-41ca-8b0b-b1fdae12e279.aspx

"I am tired of all this sort of thing called science here... We have spent
millions in that sort of thing for the last few years, and it is time it
should be stopped."
-- Simon Cameron, U.S. Senator, on the Smithsonian Institute, 1901.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.