System stability and efficiency

Keith Myers
Keith Myers
Joined: 11 Feb 11
Posts: 4704
Credit: 17548992195
RAC: 6433422

MontanaDoug wrote: Some more

MontanaDoug wrote:

Some more questions:

1.  My recent change to try 3 jobs/GPU seems to have done something.  I had been running consistently 2/GPU, now I'm running 3 on device 0, and 1 on device 1.  Am I limited in GPU memory??  CPU memory???  device 0 is my 8GB unit, device 1 is 4GB.

2.  "Running (0.9 CPUs + 0.5 AMD/ATI GPUs(device #))".  Can you refer me to an explanation of this, like - why .9 CPUs and is that toggled in the preferences somewhere?  I also thought that the 0.5 GPUs would change when I went to 3/GPU. but no????

Fun on Xmas eve!

Cheers, Doug

No, that is set by the scheduler in the server settings.  The application developer set the 0.9 cpu usage for the application as natively delivered.

If you want to change that to your own preferences you need to write and run an app_info.xml for the application.

You can read about it here. Client/Application configuration document

 

MontanaDoug
MontanaDoug
Joined: 16 Dec 20
Posts: 17
Credit: 15418321
RAC: 0

MontanaDoug wrote: 1.  My

MontanaDoug wrote:

1.  My recent change to try 3 jobs/GPU seems to have done something.  I had been running consistently 2/GPU, now I'm running 3 on device 0, and 1 on device 1.  Am I limited in GPU memory??  CPU memory???  device 0 is my 8GB unit, device 1 is 4GB.

 

Answered my question:  I was CPU limited.  I've increased my CPU utilization to 50% of the processors and now have both GPUs processing 3 tasks.  My temps (GPU and CPU) are a bit higher than I'd like - may need the liquid methane cooling!

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5589
Credit: 7675779308
RAC: 1854565

MontanaDoug

MontanaDoug wrote:

Annoyance!!

In BOINC Manger>Options>Computing preferences, It states that it is using web based preferences from WGC.  I've tried many different ways of changing that to Einstein - NO LUCK!

Suggestions?

This is part of my system stability project.  Progress is being accomplished (at a snail's pace)

 

Normally the "last" website you make changes on in "preferences" will (eventually) be the one listed as your "preference" website.

Since I normally use the local preferences on each machine it is a non-issue and gives me more system-specific control over each machine (I do have several).

Tom M

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5589
Credit: 7675779308
RAC: 1854565

MontanaDoug

MontanaDoug wrote:

MontanaDoug wrote:

1.  My recent change to try 3 jobs/GPU seems to have done something.  I had been running consistently 2/GPU, now I'm running 3 on device 0, and 1 on device 1.  Am I limited in GPU memory??  CPU memory???  device 0 is my 8GB unit, device 1 is 4GB.

 

Answered my question:  I was CPU limited.  I've increased my CPU utilization to 50% of the processors and now have both GPUs processing 3 tasks.  My temps (GPU and CPU) are a bit higher than I'd like - may need the liquid methane cooling!

There is the gross number of CPU threads you are using for the system (I run 2-4 threads idle depending on the core count of the CPU).  And the CPU "amount" per GPU thread.

For a while, I was running 0.125 CPU threads per GPU thread.  And not seeing any apparent loss in production/processing speed with up to 3 GPU threads on Radeon 5700's.

Then the processing times started extending.  So I regressed to 1 cpu thread per gpu thread (on Gamma-Ray tasks).

When I want to squeeze more cpu production out of my cpus I will start playing with sub 1 cpu threads again.

Tom M

 

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5589
Credit: 7675779308
RAC: 1854565

Is there a way to run e@h gpu

Is there a way to run e@h gpu tasks and cpu tasks on the same machine under windows?

My last attempt caused my gpu to stop receiving tasks and a large over supply of cpu tasks.

I basically have to run a non-gpu project on my e@h gpu machine to utilize the CPUs at all for cpu tasks.

 

Tom M

 

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Keith Myers
Keith Myers
Joined: 11 Feb 11
Posts: 4704
Credit: 17548992195
RAC: 6433422

Use a small cache and a

Use a small cache and a max_concurrent on the cpu applications.

 

archae86
archae86
Joined: 6 Dec 05
Posts: 3145
Credit: 7024634931
RAC: 1810885

Tom M wrote:Is there a way to

Tom M wrote:

Is there a way to run e@h gpu tasks and cpu tasks on the same machine under windows?

My last attempt caused my gpu to stop receiving tasks and a large over supply of cpu tasks.

I basically have to run a non-gpu project on my e@h gpu machine to utilize the CPUs at all for cpu tasks.

Yes, although I don't do it.

I think the secret to avoid the "large over supply" category of trouble is to use a really small queue requested size.  Try 0.1 day + 0.0 day.

At Einstein, mixing task types (even mixing GW and GRP on the GPU) gives big fluctuations in task duration correction factor, thus big swings in prefetching.  On the high end of the swing too much comes in, and later the machine goes to panic mode.

That may not be exactly what happened to you, but I think that keys to getting peaceful coexistence include very small cache size, and appropriate choice of "Use at most" nn% "of the processors".  The last point is especially important if the actual number of processors is overstated (as some people do to raise their maximum daily task quota to feed highly capable GPUs).

 

[edit: ninja overlap with Keith]

MontanaDoug
MontanaDoug
Joined: 16 Dec 20
Posts: 17
Credit: 15418321
RAC: 0

Hi Tom Thanks for your

Hi Tom

Thanks for your suggestions.  I've been running smoothly for about a week now.  The settings I'm using don't seem to impair any of my spreadsheet work ( or other tasks), so I'm running 24/7.  I've given an answer to your question below, though I kinda feel like the amateur instructing the expert!  Beware!!

 

Tom M wrote:

Is there a way to run e@h gpu tasks and cpu tasks on the same machine under windows?

My last attempt caused my gpu to stop receiving tasks and a large over supply of cpu tasks.

I basically have to run a non-gpu project on my e@h gpu machine to utilize the CPUs at all for cpu tasks.

Tom M

 

Win10 Up-to-date.

My Preferences: (In Einstein) 

Running 4 GPU tasks (with 2 GPU's), 3 CPU tasks (all WGC)  all at the same time

Use at most:    50.0 % of the processors
Use at most:   100.0 % of the CPU time
Page/swap file: use at most:   75.0 % of total
Memory: when computer is in use, use at most:  50.0% of total

Resource share:   50  (WCG  90)
Use CPU:   NO
Use NVIDIA GPU:   NO
Use AMD GPU:   YES
Use INTEL GPU:  NO
 
Run CPU versions of applications for which GPU versions are available:  NO
Allow non-preferred apps:  YES
GPU utilization factor of BRP apps:     1.00
GPU utilization factor of FGRP apps:   0.50
GPU utilization factor of GW apps:      0.50

I think I listed the operative factors.  I'm running a bit warmer than I'd like (mid 60's max) on the GPU's and CPU's.

Doug

MontanaDoug
MontanaDoug
Joined: 16 Dec 20
Posts: 17
Credit: 15418321
RAC: 0

Hi Tom BOINC stored about

Hi Tom

BOINC stored about 44 hrs of Einstein work (all GPU) and about 52 hrs of WGC CPU work using the suggested .1 and .25 "store at least" and "store up to" factors.

 

BOINC must think my system is 10x faster than it is!!!

 

Don't know if either of my answers is helpful!

 

Cheers, Doug

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5589
Credit: 7675779308
RAC: 1854565

MontanaDoug wrote:I've

MontanaDoug wrote:

I've given an answer to your question below, though I kinda feel like the amateur instructing the expert!  Beware!!

Tom M wrote:

Is there a way to run e@h gpu tasks and cpu tasks on the same machine under windows?

My last attempt caused my gpu to stop receiving tasks and a large over supply of cpu tasks.

I basically have to run a non-gpu project on my e@h gpu machine to utilize the CPUs at all for cpu tasks.

Tom M

Doug,

Your not "instructing" your offering your experience.  Your experience is valuable to the community and me.

I have had reliable results combining non-gpu projects like World Community Grid or other projects like Prime Grid or MLC@Home with E@H gpu tasks.

I have had trouble, multiple times, with trying to combine gpu and cpu tasks under E@H.

The system under discussion is this one.

E@H has regularly given me too many CPU tasks and then refused to download any more GPU tasks because there are "too many runnable tasks".

Under Linux, under a limited release, privately available distribution of the Boinc Manager called "Pandora", you can limit the number of specific application cpu tasks downloaded.

Since I was previously having trouble with getting the AMD drivers under Linux 18/20 to install I switched to Windows10 where I was finally able to run 2 Rx 580's and 3 Rx 5700's on the same machine. Something I couldn't do under Linux.  The problem with installing the Linux driver has been fixed but I am upto 8th place on the top 50 computer listings so I am reluctant to revert.

I am driving my GPUs with an AMD 2700x (8c/16t) that under standard usage I can run up to 14 threads without overcommitting (and slowing down) the system.

So far I have discovered the Rx 580 8GB GPU's will not run Gamma-Ray tasks for 3 tasks at a time.  The GPU tasks on those cards start stalling.

Yesterday I took the Rx 580's offline (on a mining rack you simply unplug the data connections) and resumed running 3 threads per Rx 5700 to see if I could "sustain" the current RAC I have.

This morning I discovered I was about to (again) run out of GPU tasks.  So my backup "0" resource project PrimeGrid started running GPU tasks in place of E@H because "it" couldn't download any more GPU tasks from E@H.

I have done two things.  1) PrimeGrid is now set to NNT (No New Tasks) and 2) I aborted about 500 Gravity Wave CPU tasks.

This caused Boinc Manager to ask for and get more Gamma Ray GPU tasks.

I am currently running without faking out the CPU count because so far the GPU task supply has been sufficient to keep my box fed.  I have played with faking a higher count of CPU cores but didn't reduce the % of available CPU cores in proportion so experienced a flood of downloaded GW CPU tasks.

One of the questions I have is would I experience the same issues if I was processing Gamma-Ray tasks on the CPU instead of Gravity Wave tasks?

Since I can get it to "run" GPU tasks by manually aborting the oversupply of CPU tasks why do I "have a problem"?  I am a big fan of "set and forget".  Sometimes it can be days or a week before I look at my Boinc machines.  If I have to manually intervene on a regular basis it means it is no longer a "set and forget" rig.

If I get aggravated enough I will simply switch back to a GR GPU-only profile like I have on this machine and run probably run World Community Grid tasks on the CPU. Or even go to a GPU only server.

Tom M

 

 

 

 

 

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.