PCIe 2 vs 3 SAME RIG

5pot
5pot
Joined: 8 Apr 12
Posts: 107
Credit: 7,577,619
RAC: 0
Topic 196342

Runtime differences between PCIe 2 & 3 while running 3 tasks on the same GPU:

PCIe 3.0:

GPU sec

3,172.46
3,015.02
3,129.75
3,051.46
3,581.70
3,333.05
3,497.01
3,262.41
2,940.57
3,341.89
3,138.30
3,140.32
3,220.97
3,032.34
3,109.77
3,093.44
3,236.33
2,790.31
2,582.41
3,257.33 AVG TIME FOR GPU= 3146.34

PCIe 2.0

GPU

3,659.54
3,765.54
3,701.79
4,071.55
3,488.81
4,000.56
3,806.07
3,685.78
3,171.36
4,354.94
3,593.30
4,214.23
3,516.39
3,680.21
3,731.63
3,816.37
4,183.45
3,451.99
3,748.91
3,723.35 AVG GPU TIME= 3768.28

% Difference= 16.5%

Even giving it a little wiggle room, THATS HUGE!!

Here are my rig's specs:

CASE: Corsair 600T Mesh
CPU: i7
MOBO: x79 Sabertooth BIOS 0906
RAM: 16GB Corsair Vengeance 1866
GPU: GTX 680
COOLING: H70 CORE w/ Scythe Ultra Kaze in push/pull@ 2400/2200
Storage: Agility 3 2x60GB
PSU:Seasonic 750W Gold
OS:W7 Home Premium SP1

In order to get PCIe 3.0 working, I had to use a workaround which involved editing the registry. If you would care to know how to do this, send me a private message (only applies to x79 owners)

Sid
Sid
Joined: 17 Oct 10
Posts: 149
Credit: 565,548,027
RAC: 576,138

PCIe 2 vs 3 SAME RIG

Thank you. This is indeed interesting result. By the way, almost the same can be said about PCIe 2.0 - 16X and 8X has about 20% difference in runtimes .

5pot
5pot
Joined: 8 Apr 12
Posts: 107
Credit: 7,577,619
RAC: 0

Interesting. Given that I

Interesting. Given that I was running 7 WCG tasks while these 3 were running, I bet with 0 WCG tasks active or HT disabled, the % would increase to around that 20% mark. Thanks for the info

Jeroen
Jeroen
Joined: 25 Nov 05
Posts: 379
Credit: 738,816,536
RAC: 1,582

To my surprise, there is a

To my surprise, there is a performance difference from 3.0 x8 to 3.0 x16 as well. I tested out one of my cards on an x8 slot for future expansion planning and noticed the following difference:

2 Tasks Per GPU

PCI-E 3.0 x16: ~ 1950 seconds
PCI-E 3.0 x8: ~ 2280 seconds

For PCI-E 2.0, I have seen as much as a 30% difference in performance going from x8 to x16 while running 3-tasks per GPU. I need to dig up the numbers again but that was approximately the percentage I came up with when I did the comparison last year. This is one of the reasons I did not upgrade my dual 580 system to triple cards as the performance loss from running two cards at x8 did not seem worthwhile in consideration to significantly higher heat output and power draw.

5pot
5pot
Joined: 8 Apr 12
Posts: 107
Credit: 7,577,619
RAC: 0

Going to assume you have a 4

Going to assume you have a 4 slot mobo. That fact of most 4 slots automatically switch to 16 8 8 with three is exactly why I bought the 3 slot sabertooth. It has 2 dedicated to 16, while the third one is 8. This way, when I have 3 in, it uses the max 40 lanes.

The results with PCIe 3.0 @ x8 lanes is not surprising. Remember, 3@8 is the same as PCIe 2@16.

Since this is a linux OS you're using, do the drivers you have installed not force PCIe 3.0 slots to turn to 2? If so, I bet there are quite a few people at NVIDIA's forums that would "love" to hear this information.

Cheers

Jeroen
Jeroen
Joined: 25 Nov 05
Posts: 379
Credit: 738,816,536
RAC: 1,582

For my PCI-E 3.0 board, I am

For my PCI-E 3.0 board, I am running the Asus RIVE which supports x16/x8/x16. At the moment I have only the two x16 slots occupied. The 40-lane support of socket 2011 comes in handy for this project IMO.

When I was running a full Linux distro with X11, the GUI tool from NVIDIA called nvidia-settings reported a link speed of 8.0 GT/s with the GPU loaded. Since then, I switched to a very small Linux system about 35 MB in size that does not have X11 which gives a bit better performance without the overhead of X running. The driver and kernel version is otherwise the same.

Since I do not have the ability to run nvidia-settings any more, I did some digging around with the latest version of pciutils and the slots are showing as being enabled at 8.0 GT/s. In both cases, I ran driver 295.33 and PCI-E 3.0 support is enabled by default. I was a bit worried initially that NVIDIA would have done something in the driver code to disable 3.0 in Linux like they did for Windows but that has not been the case fortunately.

hotze33
hotze33
Joined: 10 Nov 04
Posts: 100
Credit: 367,079,550
RAC: 3,466

@ 5pot You have quite some

@ 5pot
You have quite some run time variations. In your other thread you mentioned that your gpu utilization varied a lot. You compensated this with a slight overclock. I see the same run time difference when running 7cpu+3gpu but hardly any when running 6cpu+3gpu. The RAC stays the same but the system is much smoother.

I had little bit higher hope for the new nvidia chips. PCIe 2.0 performance is quite the same as with my GTX470 (except the power draw). And we know that the einstein app is hungry for PCIe bandwidth.

5pot
5pot
Joined: 8 Apr 12
Posts: 107
Credit: 7,577,619
RAC: 0

Since this isn't one of my

Since this isn't one of my main projects I'm quite hesitant to drop CPU on wcg down to 6. Only on occasion does it get wild still. Guess it all has to do with what parts of the wus are being crunched for both Einstein and wcg. Sones it fluctuates and sometimes it stays steady. When I suspend wcg it for sure stays steady at around 90%. Will give it a try and see if it levels out today though. Always love an experiment.

hotze33
hotze33
Joined: 10 Nov 04
Posts: 100
Credit: 367,079,550
RAC: 3,466

I have always an eye on the

I have always an eye on the actually run time vs. the cpu time and make sure they are not to far apart (for the cpu only tasks).

Jeroen
Jeroen
Joined: 25 Nov 05
Posts: 379
Credit: 738,816,536
RAC: 1,582

For anyone running the GTX

For anyone running the GTX 680 with Linux on a PCI-E 3.0 x79 board, I would suggest avoiding driver 295.53. I just tried this driver and NVIDIA did something to limit the slot configuration to PCI-E 2.0 (5.0 GT/s). I reverted back to 295.33 and all looks well again with PCI-E 3.0 (8.0 GT/s). I plan to spend some time this weekend to see if I can figure out what NVIDIA changed in the kernel source to limit this with the 295.53 drivers as these drivers are needed for some of the newer 6xx cards that have come out.

5pot
5pot
Joined: 8 Apr 12
Posts: 107
Credit: 7,577,619
RAC: 0

Thought that was going to

Thought that was going to happen. Saw the 295.33 were the release drivers. Same as how 300.83 worked for PCIe 3 on x79.

Wish you the best of luck.

NVIDIA hasn't posted about how their "certification" was going yet on their forums. So, it would appear we will be in the dark for quite some time. Next driver release isn't scheduled til end of June. NVIDIA released anyways. Sometimes "leaked" ones pop up on Guru3D. Don't usually see too many Linux ones though.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.