Discussion Thread for the Continuous GW Search known as O2MD1 (now O2MDF - GPUs only)
Language
Copyright © 2024 Einstein@Home. All rights reserved.
Copyright © 2024 Einstein@Home. All rights reserved.
it would be great if they are
)
it would be great if they are able to fine tune the scheduler to no longer send WUs requiring large amounts of GPU memory to those systems that do not have a sufficient GPU.
_________________________________________________________________________
Thanks for the heads up!
)
Thanks for the heads up!
Team AnandTech - SETI@H, Muon1 DPAD, F@H, MW@H, A@H, LHC@H, POGS, R@H.
Main rig - Ryzen 5 3600, 32GB DDR4 3200, RTX 3060Ti 8GB, Win10 64bit
2nd rig - i7 4930k @4.1 GHz, 16GB DDR3 1866, HD 7870 XT 3GB(DS), Win 7 64bit
I received GW GPU tasks a
)
I received GW GPU tasks a little under an hour ago for the first time in a couple of days. One hopes that revised memory requirement estimates succeed in taming the situation quite a bit.
GW gpu tasks are back. Just
)
GW gpu tasks are back. Just got a big refill. Assume the problem of the low memory cards is fixed now.
Yes, GW WUs are back. But
)
Yes, GW WUs are back.
But I am curious - how exactly these "memory issues" have been fixed.
It just scheduler on the server side fixed to not to sent high memory demand WUs to low mem GPU while some WUs still actually need a lot of RAM to do a work and still be distributed to GPUs with >= 4GB.
Or high memory consumption (>=3 GB) is a due to some errors in GW app leading to excess memory allocation.
Because i saw a strange patterns in GPU usage: in <2 GB range it start to slowly increasing after WU startup in small chucks (few dozen MB in each step): this is CPU part prepare data and pump it to the GPU(i know it by GPU usage at this stage near 0% but CPU is working hard). And after it reach some amount (like 1500-1800 MB) GPU starts its part of computation while GPU memory usage stays stable until WU computation is finished.
While with high memory demand WU its quite different:
Beginning of WU computation is same, slowly reaching memory usage of about 1500-1800 MB druring "CPU pre-processing" stage. But after some time it suddenly almost doubled in just one big jump. Like if same RAM/buffer allocation just done twice without preparing a new data to fill this additional amount.
Based on Bernd’s previous
)
Based on Bernd’s previous comments, it’s probably the former.
so I’d guess that 2-3GB GPUs will no longer be sent the WUs requiring >3GB.
if you have a 2-3GB GPU, best to also set your settings to only run 1 WU at a time.
_________________________________________________________________________
I have AMD 4GB cards, but was
)
I have AMD 4GB cards, but was running 2x GW and it caused some trouble sometimes, while most of the time it works OK. Tasks did not crash, but being slow down significantly and some general system instability.
Probable due to overloaded PCI-E bus by GPU RAM <==> system RAM transfer at the moment when GPU catch "big" WU and 2 tasks can not fit in 4GB of GPU RAM and GPU drives try to use system RAM as extension with access via PCI-E.
Right now i have switched 4GB cards to run only 1 GW WU (or 2 GammaRay WUs). But would like know if it possible to switch back to running 2x GW as it gives significant performance gain.
The Gtx 1060 with 3Gb memory
)
The Gtx 1060 with 3Gb memory was suddenly getting loads of errored wu,s on GW. Guess I can run binary pulsar wu,s instead.
bigendian81 wrote: The Gtx
)
Nice test cases there. You have dozens of tasks sent out after the two-day shut down, so presumably done under the revised "how much memory is enough" regime.
Your 24 valid tasks fall in the DF range .60 to .65, so on my observations should need slightly under 2 Gb, and thus perform comfortably on your 3Gb card at 1X (though not at 2X).
Your 23 errored tasks fall in the DF range of .75 to .95, though almost all of them are in the .90 to .95 range. On my observations all 23 would need almost 3.4 Gb, so could not be expected to run well (or in your case, at all) on a 3 Gb card.
Either the revised memory requirement for task dispatch rule is not working, or the parameters were not set so strictly as they would have been had my observations coincided with the presumptions used.
I have been scolded because I
)
I have been scolded because I was using a PC wih a GTX 1060 and 3 GB Video RAM, which works perfectly in GPUGRID. Then I bought a new PC with a GTX 1650 and 4 GB. But now most of the GW tasks use no more than 2 GB of Video RAM. They would have worked perfectly on the old PC.
Tullio