Automating project (gpu) suspend - for summer heat issues

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5585
Credit: 7672532904
RAC: 1728769
Topic 227830

I believe that suspending the GPU processing during the hottest part of the day will allow Boinc systems to continue CPU processing 24/7 during the local summer.  This can be done manually (which is what I am doing right now).  However.....

I am wondering what kind of setup we could use to automate suspending a Gpu project and resuming a project?

I am interested in a Linux solution.  I don't have a problem with a Windows solution but I don't have a Windows box doing crunching right now.  I am assuming that this solution can be used on both the repository Boinc Manager and the Petri/Tbar All-in-One Boinc Manager solution.

The goal is suspend GPU processing during the hottest part of the day (10am-Noon or so) to about 8pm or so.  (Local time/Daylight time).

My review of the boinc cmdline documentation makes me think that it should be possible.

But what kind of script/code/start/stop timing would make sense?  Would CRON be a better choice?

I am no longer trying to be a "real" programmer so someone else would have to step up.

Thank you.

Tom M

 

 

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5585
Credit: 7672532904
RAC: 1728769

A bonus problem would be to

A bonus problem would be to get a full "Hacker Day" setup that would suspend the GPU processing when the inside temperature exceeds a threshold.

I suspect it would need some kind of time delay when switching on/off to keep it from rapidly switching modes at the threshold temperature.

Tom M

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Keith Myers
Keith Myers
Joined: 11 Feb 11
Posts: 4699
Credit: 17541170881
RAC: 6356293

You would still have to use

You would still have to use cron to get the job to fire off at your predetermined time that you want gpu operation to stop.

But the boinccmd tool would be how you control that.  Read the boinccmd tool wiki.

https://boinc.berkeley.edu/wiki/Boinccmd_tool

You would invoke this probably for however many seconds of "off" time you want.  For example for 6 hours, that be for 21600 seconds.

./boinccmd --set_gpu_mode never 21600

 

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 3681
Credit: 33811444691
RAC: 37813622

instead of using the duration

instead of using the duration argument, it might be better to make the script in a way that it checks for the current time and makes decisions based on that. that way the commands aren't relative to when you initially start them. say your system crashes in the middle of the duration, when you start it back up you'd have to edit that duration to reflect when you want it to start back up again. could be annoying.

some logic like this:
 

loop, every 5mins
-check current time, t
-is time 10:00<t<20:00 ?
---if yes -> ./boinccmd --set_gpu_mode never
---if no -> ./boinccmd --set_gpu_mode auto

 

that way, no matter what time of day you start the script, it will always put you  in your desired mode.

_________________________________________________________________________

Keith Myers
Keith Myers
Joined: 11 Feb 11
Posts: 4699
Credit: 17541170881
RAC: 6356293

Yes, a better idea

Yes, a better idea considering how unstable Tom's hosts generally are.

 

Link
Link
Joined: 15 Mar 20
Posts: 97
Credit: 590324
RAC: 1499

Tom M wrote:The goal is

Tom M wrote:
The goal is suspend GPU processing during the hottest part of the day (10am-Noon or so) to about 8pm or so.  (Local time/Daylight time).

In BOINC Manager, in Computing preferences / Daily schedules, set "Compute only between" to 20:00 - 10:00, than in the Activity menu, set CPU computing to always and GPU to based on preferences.

I think this should be the most simple solution without any scripts or external software and should work on basically any OS.

If there's no BOINC Manager available for your OS, just command line, you'd have to edit your global_prefs_override.xml and tell BOINC to use that instead of global preferences (and set CPU computing to always and GPU to based on preferences). Or you edit your preferences online and set than CPU computing to always and GPU to based on preferences.

.

pututu
pututu
Joined: 6 Apr 17
Posts: 23
Credit: 467024069
RAC: 2415

On hot day, I cap the gpu

On hot day, I cap the gpu power limit lower. Gpu is still far more efficient than cpu when crunching FGRPB1G, assuming if you still want to crunch that sub-project. Only when it is extremely hot and unbearable, I shut down the whole rigs ;)

I think someone with more experience in linux (not me) should be able to write a simple crontab job with nvidia-smi command to reduce the power limit based on gpu, cpu temp or time.

Depending on your gpu card, here is how low one can set the power limit using nvidia-smi command. Using 3070 Ti for example: 

nvidia-smi -q|grep Limit
        Power Limit                       : 200.00 W
        Default Power Limit               : 290.00 W
        Enforced Power Limit              : 200.00 W
        Min Power Limit                   : 100.00 W
        Max Power Limit                   : 300.00 W
 

I think 100W on 3070 Ti should run quite cool.

P.S. I prefer just to shutdown the rigs.

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5585
Credit: 7672532904
RAC: 1728769

Quote: I think this should

Quote:

I think this should be the most simple solution without any scripts or external software and should work on basically any OS.

I agree it would be the simplest solution.  It looks like I should be able to run the CPU crunching without picking up much room heat.  So I wanted to automate shutting down the largest source (GPU).

Tom M

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5585
Credit: 7672532904
RAC: 1728769

pututu wrote: On hot day, I

pututu wrote:

On hot day, I cap the gpu power limit lower. Gpu is still far more efficient than cpu when crunching FGRPB1G, assuming if you still want to crunch that sub-project. Only when it is extremely hot and unbearable, I shut down the whole rigs ;)

Thank you.  It had not occurred to me that lowering the GPU power limit could give me the same results.  (Reduced crunching but bearable inside temperatures).

Tom M

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.