[…] Would you consider making the Einstein@Home application open source so other developers can take a stab at it?
From previous discussions here I gather that open-source development isn’t possible for this project because of the standards that will have to met in publishing any results. The methods used to obtain the data will have to withstand exacting scrutiny in the peer-review process; if bits of code were being contributed from all over the place it would be too difficult to document, and justify theoretically, with sufficient rigour.
[…]From previous discussions here I gather that open-source development isn’t possible for this project because of the standards that will have to met in publishing any results. The methods used to obtain the data will have to withstand exacting scrutiny in the peer-review process; if bits of code were being contributed from all over the place it would be too difficult to document, and justify theoretically, with sufficient rigour.
I would not even imply that a single code change by the open-source community would go into the code without being peer-reviewed, then reviewed by the current developers/maintainers of the project, then validated against the previous versions of the application, and only then committed to the code.
Most stable enterprise-grade open source projects, for example, Linux kernel, are being closely monitored by the small group of supervisors who has the power to "sign-off" on a given code change.
Actually we're considering to publish the code every few months or so.
With the philosophy and implementation of the BOINC framework there is no way to ensure that a BOINC Core Client runs a specific program (or refuses to run self-built programs), so we currently have no other measurement to ensure that a certain version of the program is run than to keep the source mol. closed.
The ultimate goal would be to improve the validator so that results that are scientifically valid (and only those) would pass validation, regardless of the program that computed them. But at the moment with the new algorithm and the problems we're currently dealing with we are nowhere near this.
Actually we're considering to publish the code every few months or so.
With the philosophy and implementation of the BOINC framework there is no way to ensure that a BOINC Core Client runs a specific program (or refuses to run self-built programs), so we currently have no other measurement to ensure that a certain version of the program is run than to keep the source mol. closed.
The ultimate goal would be to improve the validator so that results that are scientifically valid (and only those) would pass validation, regardless of the program that computed them. But at the moment with the new algorithm and the problems we're currently dealing with we are nowhere near this.
BM
You're right, there is literally no way to guarantee a given BOINC user will run one or another application, even when keeping source closed, as akos has demonstrated. The problem, inherent to volunteer computing, has been documented by BOINC community. So, the best guarantee that the results are valid is a high quality low level validator.
I wonder what is the fastest way to most science within BOINC framework. I guess the first step is to develop a stable scientific algorithm, usually as a closed source effort. Then, the same group familiar with core algorithm and core results would develop a fool-proof closed source validator. At the same time, the application is made open source so that public can contribute ideas, like algorithm optimizations, that has not been thought of by the original developers. For the latter, I assume that there are more open source contributors than core developers, which is a reasonably safe assumption.
It sounds like you are still in phase 1. Again, thank you for your prompt feedback.
P.S. Bernd -- I am sure you are well aware of the URLs in my post. This is for benefit of the greater community.
I wonder what is the fastest way to most science within BOINC framework. I guess the first step is to develop a stable scientific algorithm, usually as a closed source effort. Then, the same group familiar with core algorithm and core results would develop a fool-proof closed source validator. At the same time, the application is made open source so that public can contribute ideas, like algorithm optimizations, that has not been thought of by the original developers. For the latter, I assume that there are more open source contributors than core developers, which is a reasonably safe assumption.
It's right that with the new program we are still in the first "phase".
Some further remarks:
- With the concept of BOINC I think a suitable validator would need to be developed and running before the source code (i.e. all of it) is made public.
- Validation is always a tradeoff between time and security. The only way one could be really sure of a valid result (a "fool-proof validatior") would be to perform the same calculations on a controlled machine hidden to the public, and compare the results. Everything else is based on mol. reasonable assumptions that might fail in a particular case.
- A "stable scientific algorithm" is hard to achieve, because it's not only the algoritm that counts. Even compilers have bugs and flaws that could render a particular algorithm numerically unstable. I just had one such case where the Microsoft C++ compiler (7.1) ignored an explicite cast (probably for performance reasons).
Actually we're considering to publish the code every few months or so.
With the philosophy and implementation of the BOINC framework there is no way to ensure that a BOINC Core Client runs a specific program (or refuses to run self-built programs), so we currently have no other measurement to ensure that a certain version of the program is run than to keep the source mol. closed.
The ultimate goal would be to improve the validator so that results that are scientifically valid (and only those) would pass validation, regardless of the program that computed them. But at the moment with the new algorithm and the problems we're currently dealing with we are nowhere near this.
BM
I'm intrigued by the approach used by the optimized SETI "chicken" app (www.lunatics.at). It seems like they have some sort of validator, and some test WUs. There's even instructions on compiling your own app. I have mad visions of someday being able to find the time develop my own version, using AMD-optimized libraries, as the C2Ds get all the cool stuff (their optimized app is really fast on C2Ds).
I'm intrigued by the approach used by the optimized SETI "chicken" app (www.lunatics.at). It seems like they have some sort of validator, and some test WUs. There's even instructions on compiling your own app. I have mad visions of someday being able to find the time develop my own version, using AMD-optimized libraries, as the C2Ds get all the cool stuff (their optimized app is really fast on C2Ds).
I'm tempted by the ideo of having an installer that choses an App version suitable for a specific computer. It's much easier to compile and optimize a whole App for a specific CPU than to have various places that detect CPU features and chose different code paths within the App.
As long as we're observing serious bugs in the App code, however, I don't dare to rely on Apps that have to be manually updated, as anonymous Apps are.
RE: […] Would you
)
From previous discussions here I gather that open-source development isn’t possible for this project because of the standards that will have to met in publishing any results. The methods used to obtain the data will have to withstand exacting scrutiny in the peer-review process; if bits of code were being contributed from all over the place it would be too difficult to document, and justify theoretically, with sufficient rigour.
RE: […]From previous
)
I would not even imply that a single code change by the open-source community would go into the code without being peer-reviewed, then reviewed by the current developers/maintainers of the project, then validated against the previous versions of the application, and only then committed to the code.
Most stable enterprise-grade open source projects, for example, Linux kernel, are being closely monitored by the small group of supervisors who has the power to "sign-off" on a given code change.
Actually we're considering to
)
Actually we're considering to publish the code every few months or so.
With the philosophy and implementation of the BOINC framework there is no way to ensure that a BOINC Core Client runs a specific program (or refuses to run self-built programs), so we currently have no other measurement to ensure that a certain version of the program is run than to keep the source mol. closed.
The ultimate goal would be to improve the validator so that results that are scientifically valid (and only those) would pass validation, regardless of the program that computed them. But at the moment with the new algorithm and the problems we're currently dealing with we are nowhere near this.
BM
BM
RE: Actually we're
)
You're right, there is literally no way to guarantee a given BOINC user will run one or another application, even when keeping source closed, as akos has demonstrated. The problem, inherent to volunteer computing, has been documented by BOINC community. So, the best guarantee that the results are valid is a high quality low level validator.
I wonder what is the fastest way to most science within BOINC framework. I guess the first step is to develop a stable scientific algorithm, usually as a closed source effort. Then, the same group familiar with core algorithm and core results would develop a fool-proof closed source validator. At the same time, the application is made open source so that public can contribute ideas, like algorithm optimizations, that has not been thought of by the original developers. For the latter, I assume that there are more open source contributors than core developers, which is a reasonably safe assumption.
It sounds like you are still in phase 1. Again, thank you for your prompt feedback.
P.S. Bernd -- I am sure you are well aware of the URLs in my post. This is for benefit of the greater community.
RE: I wonder what is the
)
It's right that with the new program we are still in the first "phase".
Some further remarks:
- With the concept of BOINC I think a suitable validator would need to be developed and running before the source code (i.e. all of it) is made public.
- Validation is always a tradeoff between time and security. The only way one could be really sure of a valid result (a "fool-proof validatior") would be to perform the same calculations on a controlled machine hidden to the public, and compare the results. Everything else is based on mol. reasonable assumptions that might fail in a particular case.
- A "stable scientific algorithm" is hard to achieve, because it's not only the algoritm that counts. Even compilers have bugs and flaws that could render a particular algorithm numerically unstable. I just had one such case where the Microsoft C++ compiler (7.1) ignored an explicite cast (probably for performance reasons).
BM
BM
RE: Actually we're
)
I'm intrigued by the approach used by the optimized SETI "chicken" app (www.lunatics.at). It seems like they have some sort of validator, and some test WUs. There's even instructions on compiling your own app. I have mad visions of someday being able to find the time develop my own version, using AMD-optimized libraries, as the C2Ds get all the cool stuff (their optimized app is really fast on C2Ds).
RE: I'm intrigued by the
)
I'm tempted by the ideo of having an installer that choses an App version suitable for a specific computer. It's much easier to compile and optimize a whole App for a specific CPU than to have various places that detect CPU features and chose different code paths within the App.
As long as we're observing serious bugs in the App code, however, I don't dare to rely on Apps that have to be manually updated, as anonymous Apps are.
BM
BM