If the scientists at different universities say that the data comes from a source they didn't approve of, then they will not use it. There is no way to fight that.
So why not just wait it out until the new patch is approved? What is wrong with that?
Hmm. I don't think Akos withdrawing his optimised apps helps this situation very much.
1. Crunch3r is right. If two versions of the app get exactly the same result *consistently* then for all intents and purposes they are the same app. If this were not the case, most of the measurements in the world of science would be invalid. Imagin having to measure everything with the *same* measuring tape. If you can prove consistency between measuring methods, you are OK. With measuring tapes we have standards. With science apps we have testing, and we have Quorum.
2. What broke down in the recent case was that both testing and Quorum failed. Lots of versions of the app got into the wild without sufficient testing, and the 2-result Quorum was shown to be insufficient to handle the situation.
3. In the Einstein case, even without Akos' apps, we have many apps out there - Windows, Linux, PowerPC, etc. What is more, those apps are being run on many different steppings of CPU. That's why a lot of testing was done to check that they got the same results, and we have Quorum to catch the cases where, for some reason or other, a bad result is returned. (e.g. cosmic ray, data corruption, unforseen FPU error in some stepping of some CPU, or a corner case that breaks one compiler but not another.)
4. The science people chose the 2-result quorum because they did the numbers and decided that this gave an acceptably low chance of a bad result getting into the science database. Note the term "acceptably low". There is still minor chance that a spurious result will get through. Perhaps one in a million. Perhaps one in a Billion. But there are a lot of results. The chance that there are some bad results in there will approach unity over time.
5. This is not necessarily a problem. Distributed computing is being correctly applied here - it is essentially preprocessing all the data to weed out all the chaff and find the grains of wheat in there. Those grains of wheat are still going to have to be checked. After all, science is not just about being able to get results. It is about getting results that can be replicated using a different set up. ("This is what my measuring tape/s say/s. Now you try measuring the thing with *your* tape/s and see if you get the same result.") But of course we have made lots of progress with the preprocessing because we now know the places of interest to apply the measuring tapes.
6. With the many versions of Akos' app out there, some of which demonstrably produced bad results, and many of which were being widely used, unfortunately the probability of error in a particular result getting past a 2-result Quorum was more like one in a thousand or less. This significantly reduces the efficiency of the preprocessor and makes the checking process infeasible because there are so many potential false positives over a whole run. (OR EVEN WORSE, FALSE NEGATIVES. Think about it.)
7. BUT. Akos withdrawing his apps doesn't help the situation much. It does reduce the known error rate, but it does nothing for the unknown error rate. In a distrubuted computing dituation like this, you have to assume a percentage of the computation is being done by bogus apps, unless you are using cryptographic methods to prove validity of the apps. Anybody who is *relying* on all the science apps in their validity proof out there being untampered is kidding themselves and the methodology is automatically invalid.
8. Point 7 is not a problem because in the end, I am confident that the science team is *not* relying on 100% valid apps out there. Postprocessing will pick up the strays.
9. So what have we learned? IMHO:
a) 3rd party optimised apps need to be allowed. Because if they are not allowed, the activity is gonna go underground, which is really bad. Right now the only way you can tell if a result came from an Akos app is because Akos deliberately put an identifier in the stderr output. Someone who didn't care about scientific validity wouldn't do that. Akos is obviously very clever, but he isn't the only one out there. (Hell, even I have been known to hand optimise machine code with a hex editor and disassembler in my younger, cleverer days, and it isn't quite a lost art even now.)
b) There needs to be a disciplined protocol for testing these apps. To my mind, what broke down in the previous situation was the testing regime. Too many bad apps escaped into the wild because of insufficient testing before release. I think many people raised this issue.
c) Quorum of 2 probably isn't sufficient. The science team need to urgently rethink that one.
Live Long and crunch. It's very coool, except for my PC farm room, which is flaming hot right now. :-)
It is causing problems in the database. Now that tells me they CAN tell what is optimized and what is not, because there are issues with how it's interpreted in the database after the validation.
If this is true, then the numbers are NOT EXACTLY the same, and are giving results that are bad. Even though certain numbers match for the validator, they do not match enough for the database, and causing issues.
I believe Akos, in what he says. He made the apps, he has been at Einstein, he knows what is going on behind the scenes. Please heed his warning.
You know what Crunch3r? I've always had the utmost respect for you and your optimized apps. And when you asked us to stop using yours over at SETI I complied. I know a lot of folks didn't stop using them, and I'm really sorry about that, but I did. And when Akos asked us to stop using his apps, apparently you didn't. What is that saying?
And add to that the motivation behind the respective requests.
Has anyone here established credentials to judge the validity of results? I haven't because I'm not smart enough. I trust the E@H crew to do that for me.
Particularly Mr Crunch3r - exactly what is your expertise that enables you to dictate to the project scientists as to what to do with their project? Have you tested and compared the unofficial vs official apps on each and every possible WU? Describe for us what testing was done in house by the developers. Please elaborate for us the algorithms, their strenghs & weaknesses, the limit cases and exceptions to guard for. Give us your greater skill here, that exceeds the guy ( Akos, all hail!! ) who first tried to optimise S5 apps, found it was invalid and then recommended that they be discarded. Why did you answer the first but not the second ( which I've highlighted for you ) part of my question?
Quote:
Your faulty WU's have no science content. If you disagree - please tell us why, and why you think that you are capable of judging that.
Put simply, compared the massive row of talent that has constructed, developed and maintained this project over a 20+ year period, why should we give you the time of day? You have '100% the facts' on this for us?
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
Mike, isn't your role that of a MODERATOR? And if so, is lashing out at Crunch3r your idea of moderating? It isn't mine.
You have a couple of people on this forum who understand 386 machine instructions. Cruncher is one of them. Their skill is entirely different from a gravitational physics expert, but it does apply to the problem at hand.
People who understand program code can indeed tell you whether two different object code program will generate the same result [or not]. They don't need a physicist or an economist or a molecular biologist or whatever. Their skill is code and/or code generation.
Please remember that one of these people, Akos, volunteered to optimize the S4 application. When he first did, everybody told him that the apps were already perfect. In the end Akos contributed in two ways: by optimizing the algorithms used - this can be written into the source code of the S4/S5 science application - and by optimizing some of the machine code sequences in the actual executable program - you usually can't really do that via the source code because the compiler generates the code for you.
A lot of what can be done for the current S5 application is object code modification. Physicists are not experts at that: Intel/AMD machine code is not part of what you learn as a physicist. You can win a physics Nobel prize and have no clue what moving a variable to a register from memory will do or not do to the result.
Get a clue or ask questions. That is more appropriate for your role. Remember, moderator, not instigator.
If its not enough for the project to get absolutely valid results in 99,9999999999999999% or all cases then it should NOT let the results be crunched by any people they dont know!!
Thats as unsafe as optimized clients are.
How do you know if the output of an athlon 64, athlon 64 x2, athlon xp, athlon, amd k6, pentium I, pentium II, pentium III, Pentium IV, Core Duo, Pentium M, Celeron M, Ahlon XP M, PPC, Mac, or all these running windows 98, windows xp, Windows ME, Linux (what linux ? thounsands of different versions) or whatever.
All these CPU types give a little different output, even variable types like integer, short, long, whatever are different on different OS.
And is work only distributed to PCs that proved they run Prime-stable at least 24 hours?
If you need 100% fine results that hit all topics you just mentioned you can ONLY let them by crunched by 1 single, 100% safe, years testet Computer in a laboratory but for sure not by thoundsands of people all over the world using the strangest PC, OS and whatever.
edit: and thats still not so bad, the worst thing is that no official is saying anything here if its planned to test those optimizations for a while and then include them to the official program.
That would be the best way in my eyes.
Mike, isn't your role that of a MODERATOR? And if so, is lashing out at Crunch3r your idea of moderating? It isn't mine.
Well, bad luck there! :-)
Quote:
You have a couple of people on this forum who understand 386 machine instructions. Cruncher is one of them. Their skill is entirely different from a gravitational physics expert, but it does apply to the problem at hand.
People who understand program code can indeed tell you whether two different object code program will generate the same result [or not]. They don't need a physicist or an economist or a molecular biologist or whatever. Their skill is code and/or code generation.
Please remember that one of these people, Akos, volunteered to optimize the S4 application. When he first did, everybody told him that the apps were already perfect. In the end Akos contributed in two ways: by optimizing the algorithms used - this can be written into the source code of the S4/S5 science application - and by optimizing some of the machine code sequences in the actual executable program - you usually can't really do that via the source code because the compiler generates the code for you.
A lot of what can be done for the current S5 application is object code modification. Physicists are not experts at that: Intel/AMD machine code is not part of what you learn as a physicist. You can win a physics Nobel prize and have no clue what moving a variable to a register from memory will do or not do to the result.
Get a clue or ask questions. That is more appropriate for your role. Remember, moderator, not instigator.
Try pausing before you answer next time, you've shot off in anger haven't you? Chill out.... :-)
Moderators say nice things all the time? :-)
What about suppressing erratic discussion? Please remember there is a vast audience unseen ( the read/post ratio is high ) many of whom have been easily confused by posters who have exceeded their brief ( they email me ). So let's think beyond ourselves, eh?
'Get a clue' - I have an opinion! :-)
'or ask questions' - I just did! :-)
Instigator? I'm responding to an important point which has been well and truly mislead! If the physics is trashed it's all a ( possibly well optimised ) waste of time.
If you like Cruncher that's cool - he's probably a nice guy - but let's stay with the project's aim's.
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
Lets face it there arent many that are gonna show gravity waves. Im sure it will be checked, double checked and triple checked for good measure anyway. Whats the problem?
Do you think this project costs no money to run? Do you think Bruce, Bernd, et al. pay out-of-pocket for this?
Funding proposals typically require progress reports and updates to the agency or agencies granting the money. There may be higher powers that dictate the QA.
The developers are not doing this on a whim; they have reasons. If you can't accept the science or respect what the official leaders are doing, that's fine, but don't throw stones and act like you have some insider knowledge about their intent.
Okay, E@H participants carry out one of the steps in the search. Say unit A is validated and goes on to show the signs of a gravitational wave. You expect me to believe that the scientists that discover this at the later stage will not carry out their own personal computation of the work unit in question? (Heck they can use the offcial client if they want). My original post was in reply to an argument that stated we cannot use the optimised clients because they are not accountable.
If, the results being provided are actually useless because the computation is wrong, something which hasnt been stated yet. Then i can understand for us not to use it. We havent been told, but i have been assuming that there is a good reason and therefore started using the official client as soon as it was stated. However, the lack of information is absurd especially when i dont know why i am being told to use something that is far more innefficient.
Also,
-can people stop coming out with really vague comments like "the science is not accepting them", "they are bad" etc. etc.
-i never claimed to have insider knowledge but was giving a logical rebuttal. Dont put words in my mouth, otherwise argue away.
RE: RE: So what's the
)
Hmm. I don't think Akos withdrawing his optimised apps helps this situation very much.
1. Crunch3r is right. If two versions of the app get exactly the same result *consistently* then for all intents and purposes they are the same app. If this were not the case, most of the measurements in the world of science would be invalid. Imagin having to measure everything with the *same* measuring tape. If you can prove consistency between measuring methods, you are OK. With measuring tapes we have standards. With science apps we have testing, and we have Quorum.
2. What broke down in the recent case was that both testing and Quorum failed. Lots of versions of the app got into the wild without sufficient testing, and the 2-result Quorum was shown to be insufficient to handle the situation.
3. In the Einstein case, even without Akos' apps, we have many apps out there - Windows, Linux, PowerPC, etc. What is more, those apps are being run on many different steppings of CPU. That's why a lot of testing was done to check that they got the same results, and we have Quorum to catch the cases where, for some reason or other, a bad result is returned. (e.g. cosmic ray, data corruption, unforseen FPU error in some stepping of some CPU, or a corner case that breaks one compiler but not another.)
4. The science people chose the 2-result quorum because they did the numbers and decided that this gave an acceptably low chance of a bad result getting into the science database. Note the term "acceptably low". There is still minor chance that a spurious result will get through. Perhaps one in a million. Perhaps one in a Billion. But there are a lot of results. The chance that there are some bad results in there will approach unity over time.
5. This is not necessarily a problem. Distributed computing is being correctly applied here - it is essentially preprocessing all the data to weed out all the chaff and find the grains of wheat in there. Those grains of wheat are still going to have to be checked. After all, science is not just about being able to get results. It is about getting results that can be replicated using a different set up. ("This is what my measuring tape/s say/s. Now you try measuring the thing with *your* tape/s and see if you get the same result.") But of course we have made lots of progress with the preprocessing because we now know the places of interest to apply the measuring tapes.
6. With the many versions of Akos' app out there, some of which demonstrably produced bad results, and many of which were being widely used, unfortunately the probability of error in a particular result getting past a 2-result Quorum was more like one in a thousand or less. This significantly reduces the efficiency of the preprocessor and makes the checking process infeasible because there are so many potential false positives over a whole run. (OR EVEN WORSE, FALSE NEGATIVES. Think about it.)
7. BUT. Akos withdrawing his apps doesn't help the situation much. It does reduce the known error rate, but it does nothing for the unknown error rate. In a distrubuted computing dituation like this, you have to assume a percentage of the computation is being done by bogus apps, unless you are using cryptographic methods to prove validity of the apps. Anybody who is *relying* on all the science apps in their validity proof out there being untampered is kidding themselves and the methodology is automatically invalid.
8. Point 7 is not a problem because in the end, I am confident that the science team is *not* relying on 100% valid apps out there. Postprocessing will pick up the strays.
9. So what have we learned? IMHO:
a) 3rd party optimised apps need to be allowed. Because if they are not allowed, the activity is gonna go underground, which is really bad. Right now the only way you can tell if a result came from an Akos app is because Akos deliberately put an identifier in the stderr output. Someone who didn't care about scientific validity wouldn't do that. Akos is obviously very clever, but he isn't the only one out there. (Hell, even I have been known to hand optimise machine code with a hex editor and disassembler in my younger, cleverer days, and it isn't quite a lost art even now.)
b) There needs to be a disciplined protocol for testing these apps. To my mind, what broke down in the previous situation was the testing regime. Too many bad apps escaped into the wild because of insufficient testing before release. I think many people raised this issue.
c) Quorum of 2 probably isn't sufficient. The science team need to urgently rethink that one.
Live Long and crunch. It's very coool, except for my PC farm room, which is flaming hot right now. :-)
--Mark
--miw
To those thinking the apps
)
To those thinking the apps are giving the same work and numbers, here is the exact quote from Akos:
Hi people!
Please, don't use any patched versions of Einstein@Home app.
The bad results caused some problems in the database.
Thanks!
Here is the message.
It is causing problems in the database. Now that tells me they CAN tell what is optimized and what is not, because there are issues with how it's interpreted in the database after the validation.
If this is true, then the numbers are NOT EXACTLY the same, and are giving results that are bad. Even though certain numbers match for the validator, they do not match enough for the database, and causing issues.
I believe Akos, in what he says. He made the apps, he has been at Einstein, he knows what is going on behind the scenes. Please heed his warning.
RE: You know what Crunch3r?
)
And add to that the motivation behind the respective requests.
me-[at]-rescam.org
RE: It is causing problems
)
The *bad results* are causing problems in the database. That doesn't say anything about the good results that are marked as valid, does it?
RE: RE: It is causing
)
And those are inflicting problems in the database, not with the results themselves.
Has anyone here established
)
Has anyone here established credentials to judge the validity of results? I haven't because I'm not smart enough. I trust the E@H crew to do that for me.
Particularly Mr Crunch3r - exactly what is your expertise that enables you to dictate to the project scientists as to what to do with their project? Have you tested and compared the unofficial vs official apps on each and every possible WU? Describe for us what testing was done in house by the developers. Please elaborate for us the algorithms, their strenghs & weaknesses, the limit cases and exceptions to guard for. Give us your greater skill here, that exceeds the guy ( Akos, all hail!! ) who first tried to optimise S5 apps, found it was invalid and then recommended that they be discarded. Why did you answer the first but not the second ( which I've highlighted for you ) part of my question?
Put simply, compared the massive row of talent that has constructed, developed and maintained this project over a 20+ year period, why should we give you the time of day? You have '100% the facts' on this for us?
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
Mike, isn't your role that of
)
Mike, isn't your role that of a MODERATOR? And if so, is lashing out at Crunch3r your idea of moderating? It isn't mine.
You have a couple of people on this forum who understand 386 machine instructions. Cruncher is one of them. Their skill is entirely different from a gravitational physics expert, but it does apply to the problem at hand.
People who understand program code can indeed tell you whether two different object code program will generate the same result [or not]. They don't need a physicist or an economist or a molecular biologist or whatever. Their skill is code and/or code generation.
Please remember that one of these people, Akos, volunteered to optimize the S4 application. When he first did, everybody told him that the apps were already perfect. In the end Akos contributed in two ways: by optimizing the algorithms used - this can be written into the source code of the S4/S5 science application - and by optimizing some of the machine code sequences in the actual executable program - you usually can't really do that via the source code because the compiler generates the code for you.
A lot of what can be done for the current S5 application is object code modification. Physicists are not experts at that: Intel/AMD machine code is not part of what you learn as a physicist. You can win a physics Nobel prize and have no clue what moving a variable to a register from memory will do or not do to the result.
Get a clue or ask questions. That is more appropriate for your role. Remember, moderator, not instigator.
If its not enough for the
)
If its not enough for the project to get absolutely valid results in 99,9999999999999999% or all cases then it should NOT let the results be crunched by any people they dont know!!
Thats as unsafe as optimized clients are.
How do you know if the output of an athlon 64, athlon 64 x2, athlon xp, athlon, amd k6, pentium I, pentium II, pentium III, Pentium IV, Core Duo, Pentium M, Celeron M, Ahlon XP M, PPC, Mac, or all these running windows 98, windows xp, Windows ME, Linux (what linux ? thounsands of different versions) or whatever.
All these CPU types give a little different output, even variable types like integer, short, long, whatever are different on different OS.
And is work only distributed to PCs that proved they run Prime-stable at least 24 hours?
If you need 100% fine results that hit all topics you just mentioned you can ONLY let them by crunched by 1 single, 100% safe, years testet Computer in a laboratory but for sure not by thoundsands of people all over the world using the strangest PC, OS and whatever.
edit: and thats still not so bad, the worst thing is that no official is saying anything here if its planned to test those optimizations for a while and then include them to the official program.
That would be the best way in my eyes.
RE: Mike, isn't your role
)
Well, bad luck there! :-)
Try pausing before you answer next time, you've shot off in anger haven't you? Chill out.... :-)
Moderators say nice things all the time? :-)
What about suppressing erratic discussion? Please remember there is a vast audience unseen ( the read/post ratio is high ) many of whom have been easily confused by posters who have exceeded their brief ( they email me ). So let's think beyond ourselves, eh?
'Get a clue' - I have an opinion! :-)
'or ask questions' - I just did! :-)
Instigator? I'm responding to an important point which has been well and truly mislead! If the physics is trashed it's all a ( possibly well optimised ) waste of time.
If you like Cruncher that's cool - he's probably a nice guy - but let's stay with the project's aim's.
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
RE: RE: Lets face it
)
Okay, E@H participants carry out one of the steps in the search. Say unit A is validated and goes on to show the signs of a gravitational wave. You expect me to believe that the scientists that discover this at the later stage will not carry out their own personal computation of the work unit in question? (Heck they can use the offcial client if they want). My original post was in reply to an argument that stated we cannot use the optimised clients because they are not accountable.
If, the results being provided are actually useless because the computation is wrong, something which hasnt been stated yet. Then i can understand for us not to use it. We havent been told, but i have been assuming that there is a good reason and therefore started using the official client as soon as it was stated. However, the lack of information is absurd especially when i dont know why i am being told to use something that is far more innefficient.
Also,
-can people stop coming out with really vague comments like "the science is not accepting them", "they are bad" etc. etc.
-i never claimed to have insider knowledge but was giving a logical rebuttal. Dont put words in my mouth, otherwise argue away.