GPU Utilization question

Keith Myers
Keith Myers
Joined: 11 Feb 11
Posts: 4,782
Credit: 17,820,839,478
RAC: 3,979,340

Just a FYI comment here.  The

Just a FYI comment here.  The cpu percentage value assigned to the gpu task has nothing to do with the actual cpu utilization.

The science app itself is the only determining factor about how much cpu is used to process a gpu task.

Depending on the coding by the science app developers, gpu tasks can use as little as possible of a cpu core to support the gpu task to as many as 2 cpu cores depending on each application.  The cpu_usage can also change according to card type.  Different values for AMD cards and Nvidia cards or Intel igpus sometimes.

The GW gpu tasks in fact often use 120-140 percent of a cpu core.  So actually use more than a single cpu core to support each task. The default cpu_usage values assigned to the gpu task are set up by the app developers in their task generation templates and some templates get it right and some template values are way off.

The value is only useful for BOINC scheduling to profile how many total cpu and gpu tasks can be started concurrently on the host.

 

Jimmothy
Jimmothy
Joined: 28 Apr 23
Posts: 7
Credit: 17,232,865
RAC: 0

Keith Myers wrote: Just a

Keith Myers wrote:

Just a FYI comment here.  The cpu percentage value assigned to the gpu task has nothing to do with the actual cpu utilization.

 

Thank you! That actually explains another question I wondered about as to why it consistently used more of my CPU than I set it to. I understand the cautions about not overloading the CPU much more now. It's a bit counter-intuitive to me that the cores requested by the scheduler don't always line up with the cores required to do the job.

Now I know that it becomes a question of finding a number of cores that puts me just below, but not at 100% cpu utilization (or at least that seems to be how I should go about optimizing it). This would really explain the strangely long times I've gotten.

mikey
mikey
Joined: 22 Jan 05
Posts: 12,076
Credit: 1,834,325,298
RAC: 15,548

Jimmothy wrote: Keith Myers

Jimmothy wrote:

Keith Myers wrote:

Just a FYI comment here.  The cpu percentage value assigned to the gpu task has nothing to do with the actual cpu utilization.

 

Thank you! That actually explains another question I wondered about as to why it consistently used more of my CPU than I set it to. I understand the cautions about not overloading the CPU much more now. It's a bit counter-intuitive to me that the cores requested by the scheduler don't always line up with the cores required to do the job.

Now I know that it becomes a question of finding a number of cores that puts me just below, but not at 100% cpu utilization (or at least that seems to be how I should go about optimizing it). This would really explain the strangely long times I've gotten. 

As you do that tweaking be careful to keep the temps within the limits you are comfortable with, ie my pc's have their own a/c system so I can run them harder than people whose pc is in their family room or even bedroom. With that in mine I too would suggest you go about the tweaking as you suggested finding that sweet spot of as near 100% as you can get while still getting thru the tasks as fast as possible. Essentially that's the same way you tweak a gpu as to whether 2, 3 or more tasks at a time is better than running a single task at a time.

Boca Raton Community HS
Boca Raton Comm...
Joined: 4 Nov 15
Posts: 216
Credit: 8,506,781,570
RAC: 4,495,375

Jimmothy wrote: Hello again!

Jimmothy wrote:

Hello again! I've noticed something very strange to me, maybe someone can help me out. I've tried both 0.5 and 0.33 for my GPU utilization, and time to crunch seems to go up linearly with number of jobs going at once. It takes ~600 seconds with one, ~1200 with two, and now ~1800 with three. That is not the behavior I would expect because my GPU is not maxed out on memory bandwidth or processing capacity according to Afterburner.

 

I have Boinc set to use 20 of my 24 cores, so I should never get more tasks than I have cores. Should I be changing something to set a full 1.0 core for each task rather than the 0.9 that is scheduled by default? Is the gravitational wave search not designed to effectively utilize a GPU when ran in parallel, or is there something else strange going on at my end?

 

Also, since you are using Windows, in the Task Manager --> Performance, check your GPUs "CUDA" usage. Memory might not be maxed out (it won't be for these tasks) but your CUDA cores might be. I think you can see this in the Afterburner graphs, but don't hold me to that. On my Windows systems, I will usually leave the Task Manager pulled up with CUDA processing showing because I will know if something is up with the system almost immediately if there is a drop. 

Boca Raton Community HS
Boca Raton Comm...
Joined: 4 Nov 15
Posts: 216
Credit: 8,506,781,570
RAC: 4,495,375

Keith Myers wrote: The GW

Keith Myers wrote:

The GW gpu tasks in fact often use 120-140 percent of a cpu core. 

 

For my own knowledge, is there a specific way you figured this out? Of course, I know I can watch core usage across the system, but is there a more accurate way to track the CPU usage across the cores in Linux (or Windows)? I feel that most core usage graphs in OSs are not very detailed. 

Keith Myers
Keith Myers
Joined: 11 Feb 11
Posts: 4,782
Credit: 17,820,839,478
RAC: 3,979,340

Jimmothy wrote: Now I know

Jimmothy wrote:

Now I know that it becomes a question of finding a number of cores that puts me just below, but not at 100% cpu utilization (or at least that seems to be how I should go about optimizing it). This would really explain the strangely long times I've gotten.

Don't know what Windows utilities can show the historical cpu utilization, but on Linux the simple uptime command in the terminal provides the average cpu utilization in 1 minute, 5 minute and 15 minute intervals.

keith@Serenity:~$ uptime
 17:36:15 up  3:18,  1 user,  load average: 31.13, 30.65, 30.48
 

This my output on this 5950X daily driver.  If you are overcommitted, the averages will show more utilization than your actual available cores.  32 in my instance here. I'm set for 94% in the Manager.

The BoincTasks application can show how much of a cpu that a task uses in the Tasks and History cpu% columns.

That is where I first noticed the old runs of the OMDF tasks were using 120-140% of a cpu.

 

Jimmothy
Jimmothy
Joined: 28 Apr 23
Posts: 7
Credit: 17,232,865
RAC: 0

Thank you all for your help.

Thank you all for your help. I've got settings locked in that I'm happy with, using ~80% of my CPU which leaves some head room for web browser stuff since this is my actual personal computer that I use.  Everything has stabilized and science is getting done. Right now my goal is to get into the 99% club on Einstein. Project for this weekend is to dig my first gaming PC I built back in 2014 out of storage and get it set up to crunch. Hopefully that pushes me over the hump.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.