Generic Multiple GPU discusssion

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5585
Credit: 7672972902
RAC: 1745116

Ian&Steve C. wrote: Tom M

Ian&Steve C. wrote:

Tom M wrote:

This system is running 6 Radeon VII's.  I wonder what Mother Board he is running with his 

GenuineIntel Intel(R) Core(TM) i9-9900X CPU @ 3.50GHz [Family 6 Model 85 Stepping 4]

Tom M

the CPU is LGA2066, so you know it's an X299 motherboard

there are 6 GPUs so it's a board that likely has at least 6 PCIe slots (assuming no m.2 adapters)

so probably one of these: https://www.newegg.com/p/pl?N=100007627%20600307108%20601299335

 

if he's using M.2 adapters or slot bifurcation, then it could be just about any X299 board

In a PM he said it was an "Asus WS X299 SAGE".

He is one busy guy so I will admit I wasn't expecting a reply.  But got one.

Tom M

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5585
Credit: 7672972902
RAC: 1745116

I now have 5 Rx 5700 GPUs.  2

I now have 5 Rx 5700 GPUs.  2 Rx 580 GPUs, 4 Rx 570 GPUs, and a Gtx 1060 3GB as well as a Gtx 1660 Super (6GB).

I also have an 18 slot Mother Board (MSI B360-F Pro) with an 8c/16t CPU.

I am missing two things.

1) a mining chassis to mount all 13 GPUs on.

2) a version of BOINC Manager that would allow me to change the number of application tasks depending on each GPU or class of GPUs.

For example 3 threads on the 5700's, 2 threads on the 570/580s, and probably 1 thread on the gtx 1060 3GB.

Short of taking the source code and trying to modify it, is there a way to run this kind of heterogeneous GPU collection on a single Project/Application?  eg. Gamma Ray.  (Yes, the idea would be to blow past the first place system (if possible)).

Would something out of the custom app_info.xml settings or PANDORA do this?

Tom M

 

 

 

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 3681
Credit: 33817584318
RAC: 37826806

Tom M wrote: Ian&Steve C.

Tom M wrote:

Ian&Steve C. wrote:

Tom M wrote:

This system is running 6 Radeon VII's.  I wonder what Mother Board he is running with his 

GenuineIntel Intel(R) Core(TM) i9-9900X CPU @ 3.50GHz [Family 6 Model 85 Stepping 4]

Tom M

the CPU is LGA2066, so you know it's an X299 motherboard

there are 6 GPUs so it's a board that likely has at least 6 PCIe slots (assuming no m.2 adapters)

so probably one of these: https://www.newegg.com/p/pl?N=100007627%20600307108%20601299335

 

if he's using M.2 adapters or slot bifurcation, then it could be just about any X299 board

In a PM he said it was an "Asus WS X299 SAGE".

He is one busy guy so I will admit I wasn't expecting a reply.  But got one.

Tom M

thats one of the boards in my link.

_________________________________________________________________________

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5585
Credit: 7672972902
RAC: 1745116

Tom M wrote:2) a version of

Tom M wrote:

2) a version of BOINC Manager that would allow me to change the number of application tasks depending on each GPU or class of GPUs.

For example 3 threads on the 5700's, 2 threads on the 570/580s, and probably 1 thread on the gtx 1060 3GB.

Or maybe not.

I have now demonstrated that I can get a Gtx 1060 3Gb (and therefore presumably a Gtx 1660 Super) to run 3 GR gpu threads.

I have just kicked off a test of the Rx 570/Rx 580 to see if they will run (reliably) 3 GR tasks per card.

So once I have validated that.  Maybe all I need is "the frame".

===edit===

Nope.  I had forgotten that so far under Linux I have not gotten Rx 5700's and Rx 570/580's to be recognized by Ubuntu/Amd drivers at the same time.  With everything plugged in, depending on which card is "first" I get one group or the other group recognized.

And Windows may still have its 8 same brand gpu limit.

===edit===

Tom M

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5585
Credit: 7672972902
RAC: 1745116

This system is set for 0.1

This system is set for 0.1 additional tasks but it has slowed down its polling so that it is 2-4 hours between automated updates.

I noticed it because my RAC would go up and then retrograde slightly.  But the Graph on Boinc Mgr is going up constantly (for now).

Any ideas on how to step up the polling speed?

Higher <ncpu> ?  A really small adidtional tasks #?  

==edit===

Bingo!

3/10/2021 3:46:01 PM | Einstein@Home | (reached daily quota of 1952 tasks)

Which basically answers the question.  More ncpu. :)

====edit===

Tom M

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 5842
Credit: 109381492833
RAC: 35970721

Tom M wrote:This system is

Tom M wrote:

This system is set for 0.1 additional tasks but it has slowed down its polling so that it is 2-4 hours between automated updates.

Any ideas on how to step up the polling speed?

What you describe is exactly what the system is designed to do.  A work fetch is supposed to top up to the total of both settings (days + additional days) and then NOT ask again until the work on hand drops below just the days setting.   The 0.1 additional days effectively provides a 2.4+ hour delay between work requests.

Just set the days setting to what you want and put zero for additional days.

Cheers,
Gary.

Keith Myers
Keith Myers
Joined: 11 Feb 11
Posts: 4699
Credit: 17542335049
RAC: 6372174

Tom M

Tom M wrote:

 

===edit===

Nope.  I had forgotten that so far under Linux I have not gotten Rx 5700's and Rx 570/580's to be recognized by Ubuntu/Amd drivers at the same time.  With everything plugged in, depending on which card is "first" I get one group or the other group recognized.

And Windows may still have its 8 same brand gpu limit.

===edit===

Tom M

Tom, could you give a read of this comment section on the AMD ROCm/ROCr Linux drivers at Phoronix.  I still don't understand why you can't have both RX 5700's and RX 570/580's both be recognized by Ubuntu and have working OpenCL drivers that work for BOINC.

Is it just a matter of getting the right driver stack?  From my reading, the OpenCL component of the ROCr driver stack should work for all generations of cards.

Could you pose your question to Bridgeman on the forum and ask why it doesn't work and/or have him explain why it doesn't work.

https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-amd-linux/1241772-radeon-rocm-updates-documentation-reinforcing-focus-on-headless-non-gui-workloads

 

[Edit] Should have posted the main article that the comment section was about.

https://www.phoronix.com/scan.php?page=news_item&px=ROCm-Will-Support-GUI-Apps

 

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 3681
Credit: 33817584318
RAC: 37826806

Ian&Steve C. wrote: 7x RTX

Ian&Steve C. wrote:

7x RTX 2080 - https://einsteinathome.org/host/12803483

  • each 2080 power limited to 185W
  • not currently running any Einstein tasks (GPU cooling not hooked up)
  • will run 1x task at a time

GPU water cooling loop has been reconnected. this host is now active again. running 1x on GR

_________________________________________________________________________

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5585
Credit: 7672972902
RAC: 1745116

Tom M wrote:Nope.  I had

Tom M wrote:
Nope.  I had forgotten that so far under Linux I have not gotten Rx 5700's and Rx 570/580's to be recognized by Ubuntu/Amd drivers at the same time.

Once I am done (again) diagnosing riser card instability (May end up ordering a new set of riser hardware) I can try re-creating this issue by swapping an Rx 5700 and an Rx 570 between the two systems.

This will reduce the impact on my top performing system while I "play" with the gpu-server box.

If the suggestion I try a re-install with a different command line comes through I will have to run the gpu-server box dry.  And re-load the OS with the 5.4 kernel.  I have been allowing updates so it doesn't nag me so much.

Tom M

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5585
Credit: 7672972902
RAC: 1745116

Keith Myers wrote: Tom,

Keith Myers wrote:

Tom, could you give a read of this comment section on the AMD ROCm/ROCr Linux drivers at Phoronix.  I still don't understand why you can't have both RX 5700's and RX 570/580's both be recognized by Ubuntu and have working OpenCL drivers that work for BOINC.

Is it just a matter of getting the right driver stack?  From my reading, the OpenCL component of the ROCr driver stack should work for all generations of cards.

Could you pose your question to Bridgeman on the forum and ask why it doesn't work and/or have him explain why it doesn't work.

https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-amd-linux/1241772-radeon-rocm-updates-documentation-reinforcing-focus-on-headless-non-gui-workloads

 

[Edit] Should have posted the main article that the comment section was about.

https://www.phoronix.com/scan.php?page=news_item&px=ROCm-Will-Support-GUI-Apps

Keith,I have now skimmed the thread.  Once I have re-created the inability to run an Rx 570/580 card and an Rx 5700 card under Ubuntu 2.0 I will see if I can correctly explain what I did and how it is not working.

I am going to start the NNT stuff so that I can reduce the offline delay(s).  I may have to re-install the OS/Drivers if the Rx 5700 is not recognized at all.

Tom M

 

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.