Gag Me With A Spoon
27 Apr 2013 22:27:58 UTC
Topic 196927
(moderation:
Four GPUs on one machine. Even my builder could not do it and I don't know how it happened. I THINK the GTX 660 is the master card, and the three GTS 650s are somehow slaves. I am ashamed to even estimate the number of ERROR tasks that got sent in managing to do this. I don't know yet the production, but the watt meter shows 320 watts at the plug.
That can't be to bad, but the PSU is only 350 watts so that needs an upgrade for efficiency.
Arecibo 19 Oct 2012
Just Because The Space Alien Is Green
Does Not Mean You Should Go
Gag Me With A Spoon
)
FIRST REPORT & ESTIMATES
Summary: About 80,000 cobblestones/day on 300 watts using seven slot gaming motherboard, two-core i3 3220 ivy bridge CPU, three GTX650s and one GTX66O. Solid State Boot drive 30 Gig, one four gig memory stick. PSU Seasonic 350 watt Gold.
DATA
-- Power draw at the plug is down to 300 from 320 at burn-in start.
-- All GPUs seem to be running at about 55% of power
-- GPU capacity utilization is low. Between 85 and 90 percent on all four
-- CPU core power draw is 15 watts
-- Core 0 is at 45-55 percent utilization and seems to be running one GTX 660 and one GTX 650 on pcie3.0 cluster one. Core 1 is at 30-40 percent utilization and may be running two GTX 650s in cluster two.
PRODUCTION
The single GTX 660 runs two tasks in 52 minutes (rounding seconds up to next highest minute). Three GTX 650s run all their six tasks in 85 minutes. By my calculation that comes to about just under 80,000 stones per day. Once or twice an hour all eight tasks stop for about 30 seconds then restart without error. I have not observed any errors since final configuration. Actual production will reveal itself in the next week or so. At 300 watts at the plug I calculate about 38 watts per 10k stones.
OBSERVATIONS
First and formost, this was not a project for the faint of heart. Here are my thoughts.
1) A good way to avoid reporting error tasks is to put the entire production into SUSPEND before shutting down the machine while fiddling with the GPU cards. AND DO NOT FORGET TO CHECKMARK [on] GPU ON YOUR INDIVIDUAL MACHINE. Tools, Computing Preferences.
2) I never could meet my goal of four GTX 650s running at one time. I managed to get three of them running as long as my GTX 660 was in slot number three. I never could get the GTX 660 to run in slot one. Slot 3 = 650; Slots 1, 5 and 7 = 650s.
3) I took a hint from one of the members [THANKS] and assumed Windows 7 might not recognize a card without monitor attached. My solution was to install various cards one at a time with a spare monitor attached, using the GTX660 and a video adaptor to feed my primary VGA monitor, then rebooted to do it again. Windows seems to remember cards once they have been seen.
4) After each GPU switcher-roo I visited the nvidia site and used their automatic detection button and let it do its thing. They had me jumping through various Java and other download hoops to accept my requests, then provided whatever drivers they sent. Also, on at least three ocasions Windows provided a new update upon shutdown.
5) Oviously you do not need a monster PSU for this. My Seasonic 350 Gold is now drawing 300w at the plug. But remember. The PSU itself is using about 10% of the power. That means the internals of the machine are drawing about 270 watts. By my rekoning that is just a tad below 80% utilization of 350 watts available to the innards. Probably a bit out of the sweet spot. Not by much though. But for the time being I am going to leave this beastie alone and treat it like an irritated rattlesnake for the next few days.
FINALLY
Do not, repeat, do NOT use a configuration like this for gaming. Both the CPU and PSU and other hardware are way to small. You will burn your house down, even with all five case fans running! They will simply spread the flames.
Arecibo 19 Oct 2012
Just Because The Space Alien Is Green
Does Not Mean You Should Go
Most gamers would advise you
)
Most gamers would advise you to use a substantially higher rated power supply in that configuration.
I used to think they grossly overstated the advisable capacity, but I learned something a few months ago, and am more humble.
The basic problem is that I was able to observe, with a UPS power consumption reporting application which updates a few times more frequently than my Kill-a-watt, that the at the wall socket power consumption of my PC containing a BOINC-running GTX460 varied more at that time scale than I expected.
As the UPS observation is at the socket, this variation is much less than seen internally, as it has already been low-pass filtered by the primary power supply capacitors (as well as being low-pass filtered at considerably higher frequency by filter capacitors internal to the PC).
David, the one saving grace of your system may be that 4 graphics cards have their power consumption spikes somewhat asynchronously, so peak may (usually) exceed average by a smaller percentage than in a single GPU system.
After my observation, I upgraded a supply which already exceeded the average consumption of my system by rather more than yours with another rated about 50% higher. I believe than some annoying but fairly infrequent error conditions got better, possibly as a consequence.
I'm an electrical engineer, a lifetime computer guy (since 1965) and long-ago microprocessor design engineer. It was no surprise to me that system power consumption would vary greatly at a time scale of nanoseconds, microseconds, and even milliseconds, but I was actually surprised to see a GPU system running an Einstein BRP load varying in load on the wall socket by many, many tens of watts at a time scale of hundreds of milliseconds.
Sadly, I can't suggest a rational way to size one's supply. But I'd be really intrigued if David would try swapping out the 350 watt Seasonic for one of higher capacity and report any observed differences. It might take a while for a near worst-case combination of usage from the multiple boards to coincide, possibly needing to also include something else being in an unusually voltage-sensitive state. So if this is done, it would be best done after running the system in a consistent state for some days, if not weeks.
Great work there David! Keep
)
Great work there David! Keep the reports coming. Let's hope you don't progress from the Bleeding Edge of discovery to the Burning Edge ... :-) :-)
I'd have to agree with Peter here -> always go long on PSU's. My preference for over five years now are Corsair's with a rating that well exceeds your addition of estimates of averages. That way you keep it in the sweet spot ( efficiency wise ) plus plenty of soak for variations. It costs more for sure but I think it is an area where the 'right' calculation is of the cost if you don't, not the cost if you do.
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
archae86 & Mike I sized
)
archae86 & Mike
I sized the PSU based on four pcie3.0 GTX560s which would drop the total draw by at least twenty watts over the current configuration. In addition, I was expecting an idle draw about twenty watts less. [More on that later] At the time I asked my builder about a Platinum PSU and he told me they are expensive [true] but also way bigger then I needed. Since then I found Seasonic makes a 400w fanless Platinum and ordered it for my second build.
Both build plans have maximum production to power consumption in mind, with a goal of 35-45 watts per 10k stones. This big mother board [and star-wars case] were needed to mount the four GTX650 configuration, but no one could get it to run all four. It now has three plus the GTX660. Preliminary calculations put it between 35 and 40w per 10k stones.
The second build will only run two GTX650s on a very basic motherboard. I ordered a couple of supper-clocked EVGA units, and the above mentioned Seasonic Platinum. One problem with the big game board is much higher overhead power consumption then expected. It idles naked [now GPUs etc] at about 70 watts, even with the little twin core Ivy Bridge CPU. That CPU draws 15 watts to run this etire rig, so there is considerable room for overhead improvement. I hope to get a production to power between 30 and 35 Watts per 10k stones which seems reasonable. I doubt it will do sub 30w production, but will keep a bottle of Milwaukees Best Ice in the fridge just in case.
This rig has been up for about two days now and has run flawlessly. Not one single error task I know of. All eight tasks do stop once or twice an hour 20 -60 seconds then resume without error. I don't know what the machine is doing, but I doubt it is heat related since power draw remains near 300 watts. Speaking of heat. I played with fan speeds quite a lot at first since one card was running a bit above 70 degrees. However upon restart the fan scheme reverted to default.
Late yesterday I buttioned the entire case back up, both top and bottom covers, and set it upright for the first time. The two case fans I have running [total of five] and all the GPU fans are near silent. I am afraid to check GPU temps, but don't want to go nuts on adjusting four separate card fan speeds. I will take my chances and let the cards run their own fans.
I will keep additional buletin board updates in mind. Right now I am happy not to fix something that is not broken [yet]. However, I am the kind of guy who put a possitive displacement Kenne Bell supercharger on top of a dead stock 5.0 ford in my favorite car of all time. A heavily suspension tweeked [subframe connectors, boxed links etc etc] 1991 Lincoln Mark VII. Ran 8 psi for an uneventful 110,000 miles. It would run NASCAR rock solid at 130mph at 3,000 rpm at which time it hit the speed limiter. That was with a 2.73 [two seventy three] axle ratio and a truck torgue converter. Needed these last two in order to tame the torgue from stoplight starts.
My downfall was upgrading to a 347 cubic inch stroker. Absolutely smoked the tranny practicing my torgue braking. Did not realize where all the smoke was coming from until I did it the second time. Two Late. So I am not going to open this computer case for fear I might just do something just like that.
Later,
Rapalyea
Arecibo 19 Oct 2012
Just Because The Space Alien Is Green
Does Not Mean You Should Go
archae86 PS I believe
)
archae86
PS
I believe you are correct in the power spike department. If anything kills this thing it will be the absolutely wild 'percent useage' spikes. While running two cards the two separate graphs were dead flat at about 93 percent each. Now all four look like osciloscopy graphs of Outer Space Radio Noise, mostly between 80 and 90 percent. But some down spikes to as low as 75 percent.
At that time I was experimenting with how to divy up the two GTX650s among the seven slots and two clusters. I believe the best perfomance came from running both in cluster one in the 16 lane slots one and three. I did not have the software to monitor the CPU cores at the time, so don't know how the work was spread out. However, I suspect these new gyrations are caused by the dual threading of each of the two Ivy Bridge cores requred by the additional two GPUs. One core separatly running two cards in each cluster. Just my speculation. But no smoke so far.
If this rig was not performing so well I might swap out for a quad core. Perhaps a quad core would flatten them all out and raise them up. But the Ivy Bridge is not even at 50 percent on either core. And no quarantee a quad would divy up the work differently. Probably I should padlock the case and toss the key. Sort of reminds me of my thinking when I 'upgraded' to the 347 stroker.
Any thoughts on this stuff?
Rapalyea
Arecibo 19 Oct 2012
Just Because The Space Alien Is Green
Does Not Mean You Should Go
There is something more than
)
There is something more than spikes when it comes to choose the right PSU, and it's their efficiency over the real consuptiom...
Most PSUs, dont give a clue about how they work in this aspect, but ussually the best efficiency is achieved when the PSU is supplying around half the nominal capacity... in this spot the power wasted inside the PSU is minimun so it should suck less from the plug... Better PSUs have a more flat line on efficiency over consuptiom, and it may be negligible but as your goal is to maximize the credits/power ratio it's worth to take this into account...
Horatio You are correct.
)
Horatio
You are correct. My builder showed me the advertised efficiency/power curve on the Seasonic Gold and explained one of the criteria to qualify for Bronze, Silver, Gold or Platinum rating was not just raw efficiency at the sweet spot, but how WIDE the sweet spot extended. My Gold has a rather broad sweet spot. I don't recall exactly, but 50-70 percent at least.
Arecibo 19 Oct 2012
Just Because The Space Alien Is Green
Does Not Mean You Should Go
RE: Sadly, I can't suggest
)
Both Asus and Corsair have Power Supply Calculators:
Recommended Power Supply Wattage Calculator
Psufinder
Claggy
Hi, just some remarks on the
)
Hi, just some remarks on the GTX 650: As far as I know they have no external 6pin PCIe power adapter. This is good when only one gpu is operating. When you have more than one then all the power has to provided by the mainboard and so over the ATX power line. If it is not beefy enough then some error can occur. Most mainboard have an additional power plug exactly for this case.
I had two 9800GX2 running (each with 6pin+8pin supply) and still got some errors or resets from time to time. Connect a molex line to the board was the solution.
Hi Hot. I specified to my
)
Hi Hot.
I specified to my builder the pcie needed to be 3.0, and have auxiliary power to run four GTX650s. He subsequently informed me even so there was not enough power to supply four GTX650s, but he was mistaken. The reason he [and me] originally could not get more then two running together had to do with WINDOWS video idiosyncrasies. Apparently each new card added needs to be added with a monitor load. Either resistors or an actual monitor. Then close Windows and do it again.
I have been running four GTX650s for a day now with good results. Errors tasks occur like the plague if you unplug and replug a monitor. Even if you shut down between times. I lost at least an hours worth of work while Windows, stupid windows, scratched its head and remembered that yes, yesterday I DID have a VGA monitor attached to this very same card. So now I have a $20 thrift shop flat screen permanently attached to GTX650 number four. Thinking about applying super glue.
.
Arecibo 19 Oct 2012
Just Because The Space Alien Is Green
Does Not Mean You Should Go