Accelerating Signal Detection via Entropy Transforms
20 May 2017 2:24:39 UTC
Topic 207748
(moderation:
This thread was created at Gary Roberts' suggestion. As requested, I will repost original my message here. In order to save time, I will then respond to Gary's input below it.
I would just like to suggest that we test out entropy transforms as a way of doing exactly this. They hold the potential to hugely accelerate needle-in-a-haystack searches for faint signals in a sea of noise. Basically, they do this by allowing the search to proceed many times faster (O(N) asymptotic compute complexity), in exchange for a small, if not negligible, cost to sensitivity.
I explained this in person to a group of exoplanet hunters late last year. They thought it would be useful, but concluded that their data set was so small that they weren't really concerned about compute time. However, Einstein@Home clearly wants as much crunch power as it can get, so I thought I should post this.
I realize that it's always more convenient to defer the review of new ideas forever, or to assume that your opinion doesn't matter because you have no management authority, but this technique could effectively add as much compute power as millions of dollars worth of hardware, if I'm reading your stats right. I promise you that the math isn't that complicated, but it does take some effort to test it out. How much will it help Einstein@Home, if at all? I don't know. But I do know that it works quite well in signal injection simulation using real SETI data.
Some people like videos, some like whitepapers, and still others like free code that they can compile themselves. It's all here on this one page, under the appropriate section headers: https://agnentropy.blogspot.com. (As you can confirm elsewhere, Blogspot is owned by Google.) Even if you're not on the coding team but have something to add one way or another, I'd like to hear from you.
I will now respond to Gary
)
I will now respond to Gary Robert's message, imported from the Parallela thread:
-----------------------------------------------------------------------
18 May 2017 21:39:35 UTC
Message 158023 in response to message 158020
fhh wrote:
"I realize that it's always more convenient to defer the review of new ideas forever ..."
That sounds a bit harsh and quite unwarranted. There's quite a history of new ideas and methods being put into practice at this project. All new volunteers are welcomed and their contributions appreciated. Please take some time to look around. There's a lot of information about new methods and discoveries that have been made here.
The thing I don't understand is why you chose to air your 'thoughts' to this particular audience (the volunteers in general and those wanting to discuss RPi style hardware in particular) rather than taking it directly to the developers. It's not as if they hide away in some dark corner somewhere. In any case this really isn't the thread you should be hijacking for your own purposes.
If you want to continue talking to volunteers, please do start your own thread and I'll move this stuff there when you have done so. In the meantime, thank you for joining and we hope you enjoy yourself here.
Cheers,
Gary.
-----------------------------------------------------------------------
The quoted comment was general; it wasn't directed at any particular group or person, here or elsewhere. That said, it's not at all unwarranted. It's fantastically difficult, as an outsider, to convince anyone to utilize ideas which even they themselves may find useful. I've lost count of the number of people I've talked to, face-to-face and otherwise, who find real value in this and understand the math, but never follow up. It's not because they're sociopathic or lazy. It's because (1) it requires unplanned work and focus and (2) they're fighting enough of their own battles for resources and funding. For them, even just exploring enhancements is often too hard to justify. My comment was precisely intended to demonstrate that I already understand these pervasive challenges.
That said, I don't doubt that new ideas have been adopted around here. That was already clear to me. I mean, that's like the entire history of astrophysics. Ideas deserve to be evaluated on their merits, but access to communication is highly asymmetric. Just imagine if you were a baker who discovered a novel way to do proteomics research. You'd be on the internet for months just trying to get anyone to return an email, let alone seriously analyze your ideas. Look, I'm not advocating for symmetric communication access; we'd be buried in spam all day. I'm just trying to explain why it's nowhere near as simple as just contacting the developers or the right manager. The next best thing, if not _the_ best thing, is to get people who are well-informed on the topic at hand to have a look at and criticize or improve upon the idea. That way, the idea gets a vetting based more on its merits, and less on unknown variables pertaining to bureaucracy. Don't get me wrong, I'm not making any comments one way or another about your developers or managers; I don't know them. It's just that I've been in this endeavor long enough to develop some street smarts, and a real admiration for crowdsourced critical analysis of ideas.
As to hijacking your thread, well, damned if you do, damned if you don't. I did my best to keep it relevant by choosing that particular one. In some forums, they scream at you for posting a new thread. In other forums, they scream at you for posting something they perceive as insufficiently relevant, as though there's some math formula that tells you the relevancy radius. Anyway, here we are. I have made this thread as you requested.
By the way, these aren't "my purposes" in any materially gainful sense. I'm honestly just trying to help, so I guess if you consider that "my purpose", then I'm OK with that. I don't get a paycheck out of this, and I can hardly imagine that I ever will. I spent like a year and a half trying to put this all together, for zero money, with zero help, just to be of help to others. I don't have a corporate sponsor, or a university tenure, or a rich uncle. I'm a mathematician with a passion for technological progress founded on good ethics. Sure, maybe in the end, it can't actually work for some reason. That's life. But this is not an attempt to build something for myself, other than the satisfaction of helping others.
Anyways, I'm not here for trolling or flame wars or whatever time-wasting nonsense. So with that said, let's see if we can make anything of this, even if that's just to discover that there's some other better way. And, sincerely, thank for welcoming me here.
Thank you for starting your
)
Thank you for starting your own thread. Just realise that you are talking to volunteers who, in the main (myself definitely included) don't necessarily have any sort of decent theoretical understanding of the science/mathematics underpinning the project they are supporting. You would be far better off talking to the scientists who do.
Seeing as you have copied everything from the other thread to here, I'll remove it there rather than shift it here. For future reference, it's desirable not to take existing threads off into an entirely new direction. This applies particularly to ones that are long running and have settled into a particular pattern and are useful to those actively participating. There is no problem with starting your own thread when it's a brand new topic. The policy is, "If in doubt it's better to start a new thread rather than disturb an existing one." If you want to present theoretical information aimed at scientists rather than volunteers, you could consider doing so in the Science forum. I'm not sure how successful you might be in overcoming the "asymmetries of communication" though :-).
Good luck with your endeavours!
Cheers,
Gary.
I see your point. I'll have
)
I see your point. I'll have to try to get the word out to the scientists somehow. Yeah maybe I need to create yet another thread. I'm not immediately sure how to balance annoyingness with directedness, however. I would suspect that they lurk around over here, but what do I know.
I'd still like to keep this thread open just in case someone comes along who wants to offer feedback or ask a question. At least, the math is mostly just high-school-level and there is no scientific knowledge required. I will admit, though, that intuition will defeat any effort to assess whether or not it's useful; only a computer can tell you that, given sufficient amounts of data. Nevertheless, maybe someone will find a mistake in my formulas or code or something. That would be great because it could probably be fixed, and quickly.
I also understand your policy as stated; it's not a problem. Thank you for being clear about that.