Fun with run_harmonic_summing() on CUDA

Heya

I've been spending a bit of spare time learning CUDA and decided to try looking at one of the BRP4 CUDA functions to see if I could make any sense of it - and maybe even improve it a bit. The Einstein code is all quite well documented, so it is a nice place to pick up new tricks for learners like me.

The choice of function ended up being run_harmonic_summing() because it is a fairly simple function. As is visible from the image below, the function uploads some data to the GFX card device (HtoD), runs two little programs on the data (the blue-ish tinted blocks) and then downloads some of the results back to the host CPU memory (DtoH):

Both of the little programs (also known as kernels in CUDA-lingo) are available in the BRP4 source tree in the file "harmonic_summing_kernel.cuh" in src/cuda/app and contain the actual science code. What exactly this code does is a bit of a mystery to me still - it searches for harmonics in the power spectrum; which probably has something to do with detecting signals which are periodic but where you only have a certain chance of hearing each "blip". Ah well, someone else can probably chime in with some enlightening details here =)

One striking thing about the performance printout for the function call (above) is that it seems to spend relatively little time doing actual computation and a lot of time moving around data. This seems to mainly be because of the PCI-E bus latency involved when sending a command from the CPU to the GFX card.
A PCI-E bus is a bit like a network: in your software you make a "packet", the packet goes to a driver, the driver sends it to a controller in the CPU or on the motherboard, the controller puts it on the bus and the GFX card's controller picks it up from the bus and then informs the graphics processor chip about the new data. Then you wait for all of that to happen in reverse to get an answer to whether your command went fine.
Obviously, if the bus is fast then it will be able to make this round-trip much more quickly than a slow bus - and in turn it will be able to send more data in the same amount of time. The size of the packets also has something to say. These two things in unison is why a PCI-E 2.0 x16 lane is important for the Einstein CUDA app: it makes frequent round-trips and sometimes sends decently sized packages across the bus.

Taking a closer look at what happens:

Upload loop:
Data is uploaded (6 round-trips)

Computation
This is done as soon as the last piece of data is available, so it actually doesn't seem to require a round-trip

sumspec[0] download and border calculation
Power spectrum data is downloaded and used to pick interesting areas (1 round-trip)

Sync
The CPU asks the GPU to be done with everything it is doing and waits for it to finish (1 round-trip)

Download loop
Data is downloaded (5 round-trips)

So all in all it seems like it is doing 10-15 round-trips in order to get the results.
Luckily it is possible to:
1) Do more than one thing in parallel
2) Request that the GFX card accepts or delivers more than one piece of data in one round-trip, or performs a sequence of commands and only reports back once the entire sequence is done.
The GPU makers call this asynchronous transfers or asynchronous computation. They enable parallelism by using multiple "streams" and each stream accepts sequences of commands - which they will try to carry out simultaneously with the sequences running in the other streams. This is somewhat like multiprocessing on the CPU where multiple programs can run their code sequences at the same time.

So just for the fun of it here's another view of what the function could potentially look like if it used two streams performing asynchronous sequences of commands:

The key differences are:
a) All the input data is uploaded asynchronously together with the calls to start the kernels in one roundtrip - and all in one stream to make sure that they are run in sequence on the GFX card.
b) Assuming that the spectrum data isn't changed by the harmonic summing, another stream can be started to download the power spectrum while the first stream does its thing
c) Once all that is done all the relevant data can be downloaded asynchronously in one roundtrip
d) Remaining cleanup can be done after the data has been successfully downloaded.

Even with multiple asynchronous streams it seems like the function would still spend some time doing nothing but waiting for round-trips to complete.
This is why most people with modern GFX cards will find that it improves their total computing performance if they run 2 or more CUDA WUs on their card at the same time - WU2 would be able to take advantage of the "downtime" from WU1 by doing computations when WU1 is transfering data or transfering data when WU1 is computing and so on. As is clear from the plots it won't always be a 100% increase - it all depends on timing.

So, there you have it - a bit of fun with run_harmonic_summing().

__
ps. I'm clueless and I probably have half of this wrong and am missing the other half, so please correct flaws in the post or add your view to it. That is the best way to learn.
pps. Everything is somewhat photoshopped - and the curvy braces are all 10% additionally curvy; you get that for free ;)