Troubleshooting Multiple gpu setups that use Riser cards

I prefer to use the PCIe 3.0 (shielded) ribbon cables.

Firstly because I contribute with a heavy focus on GPUGRID, which is one of the projects that greatly benefits from PCIe bandwidth (to a point. 3.0x1 will cause slowdowns, but 3.0x8 will not. the limit for GPUG seems to be around PCIe 3.0x4 depending on how fast the GPU is)

Secondly, because after I started using them, they've proven to be the most reliable and easier to manage. Less cable clutter. Less things to plug in, have loose connections, or be flaky, or fail.

and those expander cards are really geared towards mining, where PCIe speed/bandwidth makes no difference. you can run mining on PCIe 1.0 x1. most of the expander cards I've seen (unless they are proper, expensive, PLX-based units) can't handle more than PCIe 2.0. And the USB cable risers almost always need a USB cable upgrade to reliably run at PCIe 3.0 (and sometimes can still be flaky even with good cables).

Also a good idea to look up the block diagram for whatever board you're trying to use. figure out which PCIe lanes are coming direct from the CPU and which are being run through the chipset. then make sure you understand the link between the chipset and the CPU as there can be conflicts there if you try to do too much through the chipset as it has a bottleneck (for example Intel's common CPU<->chipset link on consumer boards is PCIe 3.0 x4, shared for ALL chipset devices). This consideration is largely why I moved away from consumer platform to the AMD EPYC board which provides 128 PCIe CPU based lanes and no chipset nonsense.

yeah, the PCIe 3.0 capable ribbons are more expensive, but you get what you pay for.

only two of my systems are using the risers though. the single GPU systems aren't hard to imagine being inside a case. but my 7x RTX2080 system actually has all GPUs plugged directly into the motherboard. they all were converted to watercooling with 1-slot waterblocks. the radiator is mounted externally though.

8x RTX 2070 on mining frame

7x RTX 2080Ti on mining frame

7x RTX 2080 direct to motherboard in case

_________________________________________________________________________

I can get your blue screen of death if there is only one GPU with a questionable BIOS installed, usually when I get a 'mining bios gpu' used from ebay. One such MSI rx570 4GB armor OC I have never found a good BIOS to correct the problem other than not using that GPU!  Nothing to do with O/C, drivers, CPU's, etc. Just one bad GPU causing it. All this with windows 10.

I'm so stubborn trying to solve it over and over with same results over and over. So the GPU finally ended up in a drawer to solve the blue screen problem and my anger. It seems to be when the gpu or memory clock gets stuck on full or on 0. Leave it long enough and TBSofD shows up. Sometimes a few minutes and sometimes before we fully boot from restart. Some solve the hangup on their own soon enough it keeps going.

It would be sooooo nice if the manufacturers were a little more giving of stock signed BIOS.

Sometimes this can be caused by a bad riser too. Not always. Rule out risers and look to which gpu was used with a beta mining bios. Easy to find with gpu-z. Also, those gpu's tend not to run with newer AMD drivers anyway. The last AMD driver the patcher app works on is PRO 20.q2. After that when running the patcher it says 'file size to large'. All solved if we could get signed BIOS from the Company.

Might be your issue Tom.

Tom, you probably already do this. I let MSI Afterburner run the monitoring screen with just a very few parameters. GPU usage and temp, sometimes power. CPU main temp and usage. I run that screen all the time stretched across the entire monitor. I can correlate gpu hanging up on gpu clock or memory clock, using only usage graph, directly with Blue screen death.

I also tried trimming the clocks to stop it. Seems to slow down the problem but never fully gets rid of it till I can find a proper gpu BIOS.

When an AMD rx 570 4gb was under a hundred bucks we could buy a couple, looking for a good bios. For now those days are over. I now have a rule. When buying a used card, the card must have dual bios or bust the sale. The large GPU firmware archive that will go unnamed, those bios on file are not signed stock bios and work only sometimes only so well.

I may loose 10'th place for a while. I selected the engineering task to do and they downloaded over 4,000 tasks to my 13 computers. By the time I clean up all of them, I may be out of sight!

For our BOINC crunching I also have found the same issues. I thought it may be due to use of H-81 and Z-97 system boards but it seems that it's more widespread. I read with interest about PCI-e lanes and pairing with chipsets and pairing with CPU. Interesting.

I've also been thinking of other 'good practice' habits that seem to help when running multiple gpu configuration. I strongly recommend checking to be sure your power is properly grounded. I use 4 wire 120VAC on all computers. Also often overlooked is when using power supplies at such high amperage continuous duty, the AC power cable should never be smaller than 14AWG. Running 700-1400W constant, the voltage drop using 18AWG can drop our output by 5-10% due to voltage drop between the wall and the PSU. A while back when I had several 5 card rigs I immediately saw several % output increase just changing every power cord to 14AWG. ( The smaller the gauge the larger the wire size).

4 wire 120AC means 1-positive wire, 2-neutral wire, 3-electronics ground or the third plug on the power cable, and 4-a separate green wire running direct to ground rod and to the screw that holds the PSU into the computer frame and to the mining frame where the gpu cards are bolted down. If the green wire ran to a ground fault circuit breaker it would be called a (PE).

PLEASE be careful running your multi gpu rigs so to reduce chance of electrical fire in your home from overloaded circuits. Spread out each rig to a different power supply circuit breaker or have dedicated circuit breakers only for 2-3 rigs each, with 10AWG wire supplying the room where crunching happens.