BRP on GPUs: loop order

When Maxwell with its relatively large L2 cache launched we had a short but interesting discussion: the current app streams the entire data array for each operation. That's the usual way suitable for GPUs, as they have massive memory bandwidth, a massive amount of execution units with long latencies and small caches.

For CPUs one would do it the other way around: perform several calculations on a subset of data fitting into the cache. And only go through the entire array step-by-step once the previous block has finished.

Currently we have a nicely optimized GPU app, which uses almost all the GPU memory bandwidth it can get. And which shows strong signs of being limited by that bandwidth. With modern GPUs such as Maxwell moving to larger caches and generally focussing on keeping the execution units busy, the question arose: was the traditional scheme still the best option? We didn't pursue this thought any further, as the PCIe communication optimization had higher priority. I think it would be worth to give this a further look. Apart from Maxwell the AMD and Intel integrated GPUs could benefit especially, since they have limited bandwidth but comparably huge caches.

What do you guys think? Obviously I haven't seen the code, so I can only speculate. But from this comfortable distance it surely sounds worth a try.

Edit: the "current working set", i.e. the number of data points the chip would have to keep in flight, would not need to fit into the cache entirely. Even if it exceeds the cache size by a factor of 3, still 1/3 of all memory operations would be performed within the cache. Which should (to 1st approximation) reduce the memory bandwidth requirement by 1/3.

MrS