Automating project (gpu) suspend - for summer heat issues
15 Jul 2022 19:14:28 UTC
Topic 227830
(moderation:
I believe that suspending the GPU processing during the hottest part of the day will allow Boinc systems to continue CPU processing 24/7 during the local summer. This can be done manually (which is what I am doing right now). However.....
I am wondering what kind of setup we could use to automate suspending a Gpu project and resuming a project?
I am interested in a Linux solution. I don't have a problem with a Windows solution but I don't have a Windows box doing crunching right now. I am assuming that this solution can be used on both the repository Boinc Manager and the Petri/Tbar All-in-One Boinc Manager solution.
The goal is suspend GPU processing during the hottest part of the day (10am-Noon or so) to about 8pm or so. (Local time/Daylight time).
My review of the boinc cmdline documentation makes me think that it should be possible.
But what kind of script/code/start/stop timing would make sense? Would CRON be a better choice?
I am no longer trying to be a "real" programmer so someone else would have to step up.
Thank you.
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
A bonus problem would be to
)
A bonus problem would be to get a full "Hacker Day" setup that would suspend the GPU processing when the inside temperature exceeds a threshold.
I suspect it would need some kind of time delay when switching on/off to keep it from rapidly switching modes at the threshold temperature.
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
You would still have to use
)
You would still have to use cron to get the job to fire off at your predetermined time that you want gpu operation to stop.
But the boinccmd tool would be how you control that. Read the boinccmd tool wiki.
https://boinc.berkeley.edu/wiki/Boinccmd_tool
You would invoke this probably for however many seconds of "off" time you want. For example for 6 hours, that be for 21600 seconds.
./boinccmd --set_gpu_mode never 21600
instead of using the duration
)
instead of using the duration argument, it might be better to make the script in a way that it checks for the current time and makes decisions based on that. that way the commands aren't relative to when you initially start them. say your system crashes in the middle of the duration, when you start it back up you'd have to edit that duration to reflect when you want it to start back up again. could be annoying.
some logic like this:
loop, every 5mins
-check current time, t
-is time 10:00<t<20:00 ?
---if yes -> ./boinccmd --set_gpu_mode never
---if no -> ./boinccmd --set_gpu_mode auto
that way, no matter what time of day you start the script, it will always put you in your desired mode.
_________________________________________________________________________
Yes, a better idea
)
Yes, a better idea considering how unstable Tom's hosts generally are.
Tom M wrote:The goal is
)
In BOINC Manager, in Computing preferences / Daily schedules, set "Compute only between" to 20:00 - 10:00, than in the Activity menu, set CPU computing to always and GPU to based on preferences.
I think this should be the most simple solution without any scripts or external software and should work on basically any OS.
If there's no BOINC Manager available for your OS, just command line, you'd have to edit your global_prefs_override.xml and tell BOINC to use that instead of global preferences (and set CPU computing to always and GPU to based on preferences). Or you edit your preferences online and set than CPU computing to always and GPU to based on preferences.
.
On hot day, I cap the gpu
)
On hot day, I cap the gpu power limit lower. Gpu is still far more efficient than cpu when crunching FGRPB1G, assuming if you still want to crunch that sub-project. Only when it is extremely hot and unbearable, I shut down the whole rigs ;)
I think someone with more experience in linux (not me) should be able to write a simple crontab job with nvidia-smi command to reduce the power limit based on gpu, cpu temp or time.
Depending on your gpu card, here is how low one can set the power limit using nvidia-smi command. Using 3070 Ti for example:
nvidia-smi -q|grep Limit
Power Limit : 200.00 W
Default Power Limit : 290.00 W
Enforced Power Limit : 200.00 W
Min Power Limit : 100.00 W
Max Power Limit : 300.00 W
I think 100W on 3070 Ti should run quite cool.
P.S. I prefer just to shutdown the rigs.
Quote: I think this should
)
I agree it would be the simplest solution. It looks like I should be able to run the CPU crunching without picking up much room heat. So I wanted to automate shutting down the largest source (GPU).
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
pututu wrote: On hot day, I
)
Thank you. It had not occurred to me that lowering the GPU power limit could give me the same results. (Reduced crunching but bearable inside temperatures).
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!