Discussion Thread for the Continuous GW Search known as O2MD1 (now O2MDF - GPUs only)
Language
Copyright © 2025 Einstein@Home. All rights reserved.
Copyright © 2025 Einstein@Home. All rights reserved.
I'm out of GW tasks and I'm
)
I'm out of GW tasks and I'm back to pulsars, I guess 2nd prize is better than nothing and it does improve the RAC.
Betreger wrote:I'm out of GW
)
Same here, though I ran out a couple days ago. Since then, while running GRP tasks, a GW task will occasionally download and run. I figure that the GW tasks will return in full force eventually, so until then, like you say, I'll enjoy the RAC bump.
Ideas are not fixed, nor should they be; we live in model-dependent reality.
Well posing about it worked,
)
Well posting about it worked, I picked up 3.
Edit: a bunch more have followed
I know the shortage of work
)
I know the shortage of work was fixed some time ago now but I thought I would add some observations about the matter and particularly about what might have been a significant problem for many users once there was new work available. Obviously, the longer the lack of work existed, the bigger would be the work fetch request that depleted hosts would be making. I expected that the resumption of available work might cause a whole batch of "Timed out - no response" style errors. Fortunately, this doesn't seem to have occurred and there have been no complaints or examples that I've noticed.
If anybody is interested in understanding what I'm talking about, you might like to review my rather lengthy answers to the problem mentioned in this thread.
In short, under the conditions that existed at that time, if a host made a large work request, the scheduler would take an inordinately long amount of time to assemble the full list of tasks to send and in doing so would not be able to deliver them when it had finally constructed the list. It looked like the time was so long that the client had timed out and stopped waiting for the response. Normally, these would become 'lost tasks' and the 'resend lost tasks' mechanism would be expected to deliver them in batches of 12 at a time. There was no official announcement but it seems that Bernd had disabled that mechanism so that the whole batch was just being dropped and branded immediately as "Timed out - no response" errors, rather than having them as lost tasks available for resend.
The final paragraph of my longest response (in the linked thread above) contained a plea to staff to stop the scheduler from taking so much time by setting a 'cut-off' number of tasks - something like 12 max that the 'resend lost tasks' mechanism used to have. There was no response from staff so I didn't know if anything had been done about this problem.
So when the recent shortage of new work started causing work caches to be depleted, I wondered what might happen for people with large caches that would be making large requests. I expected there could be some degree of bedlam. I had 'skin in the game'. I have a ryzen 5 2600 with an RX 570 GPU with a cache size of 2.5 days just doing the GW GPU tasks 3x. It had less than a full day's work left, even though it had managed to secure lots of resends.
The availability of new work happened around 3:45am local time, 30th Dec. I know this because I've analysed the event log for this particular machine to see what exactly happened and whether there were any 'Timed out - no response' errors. It's quite interesting to look at what happened between client and server so for anyone interested, here is a commented set of excerpts from the event log to show how the scheduler handled a pretty large work request scenario. For the larger requests, I've listed the actual elapsed time between the client request and the scheduler response. The biggest delay was just shy of 2mins and the client was still listening at that time :-).
My guess from all the above is that there is now a limit of around 48 tasks (possibly expressed as a time value rather than a specific number of tasks) for any individual GW work fetch. In the case of my host, the client had not timed out any scheduler response so there were no "Timed out - no response" errors.
I was quite relieved to find everything in order a few hours after the event when I arrived on the scene :-).
The only downside was the fact that it took a total of 45 work requests to download the 401 tasks needed to fill the work cache of 2.5 days. The scheduler was engaged with my single host for just over an hour in total. It's rather mind boggling to think of how many other hosts there might have been trying to do something similar. Maybe not too many of them were running tasks x3 or larger and so wouldn't have taken nearly as long as my host did to completely fill the work cache.
There are very good reasons why the scheduler/client exchanges took the pattern they did. If anyone would like to read an explanation for the behaviour, just ask and I'll explain the factors involved. It's an artifact of the way multiple concurrent GPU tasks works and also the fact that the variable crunch times for these tasks can keep creating extra work requests when the current tasks are consistently taking a little less than what the current estimate suggests.
Maybe that's enough for people to figure it out for themselves :-).
Cheers,
Gary.
Is the scheduler running on
)
Is the scheduler running on the same host as the webserver? Because latter is apparently very slow, so I wouldn't be surprised if the scheduler couldn't handle bigger requests.
But the differences duration factor correction between FGRP and GW was definitely a problem on my end.
Stef wrote:Is the scheduler
)
The short answer is, "I don't know", but I'd be rather surprised if it was.
The server status page lists some of the servers and what functions are running on each. Notice that the scheduler has 'einstein' all to itself, whilst other servers (eg. 4, 5, 10) are listed for other functions. I imagine all the other 'number variants' are in use as well. Whilst anything to do with the efficient processing of work probably gets lots of attention, I imagine there's not too much worry if the webserver is a bit pedestrian in its performance :-).
It's understandable that the scheduling of GW work is time consuming. A lot of effort is needed to make sure locality scheduling works as well as possible. There are no such constraints on the scheduling of GRP work so large numbers of tasks can be dispensed very rapidly. The constraint with that work is not the dispensing but rather the generation of new work at times when the dispensing has fully depleted the available stock on hand. I notice that effect quite often when my hosts are all trying to top up work caches.
Yes, absolutely. I understand that there will always be variations based on different hardware, architecture, OS, task multiplicities, etc., but to have one search so violently wrong in one direction whilst the other is also violently wrong in the completely opposite direction is hard to fathom. I just don't understand why something hasn't been done about that. It's making life miserable for lots of people.
The Devs must be fully aware of it so there must be a reason why it can't be addressed. They just need to tell us why so that we can all be less 'aggravated' by it. I was tempted to use rather more descriptive language, but then had second thoughts :-).
Cheers,
Gary.
I am getting a lot of
)
I am getting a lot of invalids on my AMD cards , RX480 and R9 290x . My Nvidia cards have 0 invalids. I updated drivers and the cards test fine stability and the vram tests pass with no errors. I heard some of the older amd/ati cards having problems. Is it true for newer cards too?
https://einsteinathome.org/task/914075098
VinodK wrote:I am getting a
)
As you are getting a much higher proportion of valid work on Gamma-Ray Pulsar tasks than on Gravity-wave, I suggest you considering disabling Gravitational Wave GPU task download to your machine.
At the Einstein web site you can do this at your acount|Preferences|Project, then selecting the location (aka Venue) in which your machine operates.
The Applications section lets you de-select GW GPU work, and unticking the Allow non-preferred apps option in the Other Settings section will help prevent leakage.
VinodK wrote:I am getting a
)
You appear to have just one computer. It's currently listed as having two nvidia GPUs (at least one of which is apparently a GTX 1080) and a single RX 480. There's no sign of an R9 290X but perhaps at some earlier stage you replaced it, maybe with the RX 480 or maybe with the 2nd nvidia??
When you have problems and you need assistance, you need to fill in all sorts of details so that a person trying to help can understand exactly your setup, how you have it configured and exactly what changes were made, at what point, and with what outcome. Is your hardware being operated under stock (standard) conditions? Have you applied performance tweaks of any sort? Have you tried just one brand of GPU only at a time to see if there's any sort of conflict going on when two different brands are together? If you don't provide these sorts of details, it's just about impossible to give useful answers.
A link to the source of your 'information' would be helpful. If you are referring to some forum posts concerning GCN 1st gen AMD cards, the problem seems to be specific to SI GPUs. Neither of the GPUs you mention are 1st gen. I use RX 570s, same series as your RX 480. I'm having no issues with compute errors or invalid results when running GW tasks.
Cheers,
Gary.
Yeah, I have/had a mix of
)
Yeah, I have/had a mix of cards in my system. That is the one thing I didn't try. I will have only the AMD cards in the system, remove nvidia+amd drivers and reinstall the drivers and try. Thanks