Generic Multiple GPU discusssion

Tom M
Tom M
Joined: 2 Feb 06
Posts: 533
Credit: 1,150,919,281
RAC: 5,782,999
Topic 224547

I am starting this thread so that anyone owning any system with more than 2 gpus can discuss their system(s).

By my count 12 out of the top 50 are running 3 or more gpus.  It is possible that some of these video cards have more than one GPU on the same card.  If that pushes your system into 3 or more gpus and I missed it please let us know.

I now have two.  A 4 card box here.  And a 9 card box here.

The 9 card box is not (yet) a member of the top 50.  But then I just re-started it.  It was a 7 NVIDIA GPU box.  It is now a NVIDIA/Radeon GPU server.

Tom M

Over the hill?  What hill?  I don't REMEMBER any hill....
Your are entitled to your own opinion but not your own Data. (Senator and Prof. Pat Moynihan).
In detail, I am a big picture person.

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 382
Credit: 1,388,842,888
RAC: 15,603,265

5x RTX 2080ti -

6x RTX 2080ti - https://einsteinathome.org/host/12803486

  • each 2080ti power limited to 215W
  • currently running Gravitational Wave tasks only
  • running 1x task at a time

 

8x RTX 2070 - https://einsteinathome.org/host/12803503

  • each 2070 power limited to 150W
  • currently running Gravitational Wave tasks only
  • running 1x tasks at a time

 

7x RTX 2080 - https://einsteinathome.org/host/12803483

  • each 2080 power limited to 185W
  • not currently running any Einstein tasks (GPU cooling not hooked up)
  • will run 1x task at a time

_____________________________________________


Keith Myers
Keith Myers
Joined: 11 Feb 11
Posts: 1,066
Credit: 1,181,232,011
RAC: 2,765,718

1 X 1080Ti @215W 1 X 2080

1 X 1080Ti @215W

1 X 2080 @ 215W

2 X 2070 @ stock 175W

https://einsteinathome.org/host/12291110

All running GR tasks one at a time.

 

3 X 2080 @200W

https://einsteinathome.org/host/12444941

All running GR tasks one at a time.

 

1 X 1080Ti @ 215W

2 X 2070 Supers @ 200W

https://einsteinathome.org/host/4284634

All running GR tasks one at a time.

 

Tom M
Tom M
Joined: 2 Feb 06
Posts: 533
Credit: 1,150,919,281
RAC: 5,782,999

So far setting my "store at

So far setting my "store at least 0.01 days of work" and "0.1" additional days has resulted in very modest GW "waiting to run" cpu queues on my Amd 2700x/4 Rx 5700 system.

I have previously had a lot of trouble trying to run E@H CPU tasks on the same machine as E@H gpu tasks.

Tom M

 

Over the hill?  What hill?  I don't REMEMBER any hill....
Your are entitled to your own opinion but not your own Data. (Senator and Prof. Pat Moynihan).
In detail, I am a big picture person.

Tom M
Tom M
Joined: 2 Feb 06
Posts: 533
Credit: 1,150,919,281
RAC: 5,782,999

https://einsteinathome.org/ho

https://einsteinathome.org/host/12784895

It looks like Gaurav Khanna has switched from an i9-9900k CPU on the above machine to an i9-9900X (different socket, 10c/20t).

Apparently, his co-location had some kind of power outage which knocked his systems off their game.

I have switched my "gpu-server" to GW gpu tasks.  I have a CPU upgrade due in next weekend.  I will switch back to 9 GPUs running then. The gpu-server now sports an MSI B360-F Pro MB.  I don't expect to be limited (much) by the number of GPUs I can run.

If I drop the P106-90's offline I will be able to go to (probably) 2 threads per gpu. Maybe.

Tom M

 

Over the hill?  What hill?  I don't REMEMBER any hill....
Your are entitled to your own opinion but not your own Data. (Senator and Prof. Pat Moynihan).
In detail, I am a big picture person.

Tom M
Tom M
Joined: 2 Feb 06
Posts: 533
Credit: 1,150,919,281
RAC: 5,782,999

So what are the main benefits

So what are the main benefits of limiting the power usage of a gpu?

You probably are limiting absolute production unless you are hitting a thermal limit that is throttling your gpu processing speed.

Tom M

Over the hill?  What hill?  I don't REMEMBER any hill....
Your are entitled to your own opinion but not your own Data. (Senator and Prof. Pat Moynihan).
In detail, I am a big picture person.

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 382
Credit: 1,388,842,888
RAC: 15,603,265

Tom M wrote: So what are the

Tom M wrote:

So what are the main benefits of limiting the power usage of a gpu?

You probably are limiting absolute production unless you are hitting a thermal limit that is throttling your gpu processing speed.

Tom M

Increased power efficiency. You lose some production as the clock speeds don't boost as high when it hits the power limit. I like to overclock on top of a power limit which brings some performance back that would otherwise be lost. but overall "performance per watt" goes up.

In the case of my 8x RTX 2070 system, I've power limited the GPUs mainly to try to keep the total power draw of the system acceptable for a standard 15A 120V circuit. 8x150W = 1200W, plus the Epyc CPU at another 200W, then factoring in PSU efficiency, it puts me close to the 80% circuit load. I don't really have a good way to add a dedicated 20A or 240V circuit to this location so I'm stuck dealing with the 15A 120V.

Back when I had 10x RTX 2070 on a single machine without this power delivery constraint (was on a 30A 240V circuit), I power limited to 165W more for thermal reasons as I packed the GPUs pretty close together.

_____________________________________________


Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 382
Credit: 1,388,842,888
RAC: 15,603,265

Tom M

Tom M wrote:

https://einsteinathome.org/host/12784895

It looks like Gaurav Khanna has switched from an i9-9900k CPU on the above machine to an i9-9900X (different socket, 10c/20t).

that system has always been on a 9900X, as long as I can remember. I don't think it was ever on a 9900K

_____________________________________________


Tom M
Tom M
Joined: 2 Feb 06
Posts: 533
Credit: 1,150,919,281
RAC: 5,782,999

Ian&Steve C. wrote: Tom M

Ian&Steve C. wrote:

Tom M wrote:

https://einsteinathome.org/host/12784895

It looks like Gaurav Khanna has switched from an i9-9900k CPU on the above machine to an i9-9900X (different socket, 10c/20t).

that system has always been on a 9900X, as long as I can remember. I don't think it was ever on a 9900K

Yet another detail I missed :(

Over the hill?  What hill?  I don't REMEMBER any hill....
Your are entitled to your own opinion but not your own Data. (Senator and Prof. Pat Moynihan).
In detail, I am a big picture person.

Tom M
Tom M
Joined: 2 Feb 06
Posts: 533
Credit: 1,150,919,281
RAC: 5,782,999

The next multiple gpu

The next multiple gpu experiment will start once I have an i9-9900 installed on the "gpu server".

I have 2 XFX Rx 580's, 4 XFX Rx 570's, and 3 P106-90's I want to crank up full-time with 1 thread per gpu on Gravity Wave processing.

I expect to move the P106-90's offline as I get them sold. But until then I will use them. 

I think the "final" goal is likely to be an all XFX Rx 570 8GB machine running 2 GW's per gpu (9 GPUs) but even those GPUs are pretty expensive right now. 

This is probably the only way I can give GW tasks under gpu a fair shake (GW only machine).

So far I have been able to (lately) run GW CPU tasks on a GR gpu box without getting overrun by GW tasks (yet).  The AMD 2700x is taking on the order of 18 hours per GW CPU task.

While I am pretty sure the AMD 3950x would crunch them faster, right now that box seems to be pretty stable and I am loath to play with the CPU mix.

Tom M

 

Over the hill?  What hill?  I don't REMEMBER any hill....
Your are entitled to your own opinion but not your own Data. (Senator and Prof. Pat Moynihan).
In detail, I am a big picture person.

Tom M
Tom M
Joined: 2 Feb 06
Posts: 533
Credit: 1,150,919,281
RAC: 5,782,999

Ian&Steve C. wrote:6x RTX

Ian&Steve C. wrote:

6x RTX 2080ti - https://einsteinathome.org/host/12803486

  • each 2080ti power limited to 215W
  • currently running Gravitational Wave tasks only
  • running 1x task at a time

 

8x RTX 2070 - https://einsteinathome.org/host/12803503

  • each 2070 power limited to 150W
  • currently running Gravitational Wave tasks only
  • running 1x tasks at a time

Ian&SteveC.

What are you crunching on the CPU side of things on these machines?

Tom M

Over the hill?  What hill?  I don't REMEMBER any hill....
Your are entitled to your own opinion but not your own Data. (Senator and Prof. Pat Moynihan).
In detail, I am a big picture person.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.