Discussion Thread for the Continuous GW Search known as O2MD1 (now O2MDF - GPUs only)

Gary thanx for the input.  Now I better understand what is going on and I shall continue to run 2X until something changes, which it most certainly will. 

This would be at odds with a recent post by T-800 on the Navi 10 forum (https://einsteinathome.org/goto/comment/178293) saying that it is only when the GW app needs to reach into system memory that it bogs down.

Here's an (probably not original) idea: when a card's available VRAM is exceeded, long-run GW VelaJr tasks are not efficiently crunched.  Here are some numbers...

I can run my 4 GB RX 570 card at 1x with none of the extra-large task time increases that are seen for some VelaJr frequency bands, but at 2x those long-run tasks take much longer to complete. My 6 GB RX 5600xt, on the other hand, can run at 2x with no large task time increases, but at 3x the long-run tasks become problematic.

These differences between the cards can make sense when viewed by how much of the card's VRAM is used and assuming that "long-run" task time will take much longer when the card's VRAM capacity is exceeded. For example, the RX 5600 XT uses ~80%, or ~4.8 GB, of available VRAM when running 2x (https://einsteinathome.org/goto/comment/178290). Let's call this a requirement of 2.4 GB VRAM per task. But at 3x, it pegs out at ~100% VRAM; that is, it needs ~7.2 GB, but only has 6, so the app dips into system memory and doesn't do a very good job of it.

Now consider the RX 570 with 4 GB; assuming 2.4 GB VRAM is needed per task, it crunches fine at 1x, but at 2x it needs 4.8 GB VRAM and so must dip into system memory and, again, doesn't do a very good job of it.  This all assumes that GPU or memory load isn't limiting at any of those multiplicities.

By this reasoning, the 16 GB Radeon VII should be able to run all GW tasks equally well at 6x (14.4 GB VRAM needed), that is, without having the long-run tasks take an extraordinarily longer time. That is not to say it will crunch efficiently at 6x because GPU or memory loads will likely be exceeded and average individual task times will increase past a certain multiplicity (4x? 5x?). I'd predict, however, that the Radeon VII at 6x should see only about a 2-fold run time difference between short-run and long-run tasks, the same time differential seen with the RX570@1x and the RX5600xt@2x. Similarly, a 2 GB card would always have VRAM be limiting for long-run tasks.

As pointed out by T-800, and no doubt by others, this is not an AMD thing, and these VRAM crunching requirements would be expected to hold for NVIDIA cards.

I am using this system to exercise riser hardware (Pcie to usb3 board, cable, external PCIe adaptor for the card)

As a by-product, I am getting the following results under Windows 10 with an Nvidia Gtx 1660 Super.

~ 1,075 seconds to 1,383 seconds (GW gpu).

I don't know if it is useful or not but I thought I would offer the data.

I am testing the riser hardware, that I bought used before I re-sell it.  So I can vouch for it.

Tom M

I **think** I have an explanation for both the observed slow down when GPU's have to access main system RAM (for AMD and newer nVidia cards) and crashing completely (older nVidia cards).

Taking the Rx 5600XT as an example, for the reference AMD card at least, the maximum memory bandwidth is rated at 336 GB/s.
(From the specifications here: https://www.amd.com/en/products/graphics/amd-radeon-rx-5600-xt)

However the bandwidth of typical DDR4 main system memory is much lower , for example 38.4 GB/s for DDR4 2400 RAM running in dual channel.
(Source showing how this is calculated: https://www.pcsteps.com/7932-real-ram-speed-mhz-cas-latency/)

I would assume that the sort of workload that einstein is running involves a constant flow of data into and out of memory.
When this is VRAM on the graphics card, this can occur roughly 10x faster than when the card has to use main system memory.

Assuming I've understood this correctly, this would explain the tasks bogging down when the tasks require more memory than is available in VRAM.

Interestingly, this would mean using higher frequency main system memory (e.g. DDR4 3600 rather than the DDR4 2400 in my example above) would reduce this bottleneck, although maybe not by enough to make it worthwhile.
I wonder if memory speed is also a limiting factor for the rate at which CPU tasks are crunched?

As for older nVidia cards crashing completely when their VRAM is exceeded, I wonder if this has something to do with OpenCL / CUDA versions?
I know einstein's applications are written in the OpenCl programming language, which AMD cards support natively, whereas (as I understand it) nVidia cards only support nVidia's own CUDA language natively.
When an OpenCl application is run on an nVidia card it does some kind of 'emulation' that converts the OpenCl code into CUDA code before it runs.

I know different generations of nVidia cards use different CUDA versions, so maybe older versions could not properly handle emulating an OpenCl application that was trying to access memory outside of the cards VRAM, and that this has subsequently been added to later versions of CUDA found on newer nVidia cards, which is why they no longer crash?

Not hugely useful, but interesting (at least to me!) to explain the observed behaviour.

As Peter said above, experimenting with your system is probably the best way to find out what works best for you:

It's best I think for every individual to experiment with their own combination of GPU, VRAM, CPU, to find out what produces the most valid WUs in a given time period