Possible resource starvation (?) with FGRPB1G tasks on RX 460 GPUs under Linux

SUMMARY

This is a report on a possible resources exhaustion problem that I seem to be seeing when running FGRP5/FGRPB1G tasks on a range of different generation hosts but all having AMD RX 460 GPUs.  The OS is Linux, but not one of those that are supported by specific AMDGPU-PRO driver/lib packages supplied directly by AMD.  By looking at what gets installed when using the --compute option in the AMD supplied install script for the Red Hat specific package, I have been able to get GPU tasks to run correctly and produce valid results on my Linux distro of choice.  The hosts run for significant lengths of time (up to ~30 days) before exhibiting symptoms of resource exhaustion.  I'm wanting to document the details as I have observed them, for the purposes of seeing if others are seeing similar behaviour or have alternative theories about what might be happening.

FULL REPORT

I have quite a few older CPU crunchers that have been upgraded with RX 460 GPUs from different manufacturers - Asus, Gigabyte, Sapphire, XFX, in the main.  I also have some new builds using the same GPU.  The CPU/motherboard combinations cover a very wide range, from 2008 to recent, and include Q6600 quads, Q8400 quads, E6300/6500 Pentium dual cores, various Ivy Bridge and Haswell CPUs right up to recent Kaby Lake G4560 Pentium dual cores.  There are DDR2,3,4 RAM variants in the mix.  There are various PSUs, from old and faithful to brand new, high efficiency.  The only common factor is the RX 460 GPU.  The machines all run PCLinuxOS - not all precisely the same version - but the OpenCL compute libs all come from the same source.

I have no desire to run Ubuntu or a derivative, or even Red Hat or a derivative, so there isn't a simple procedure to install a pre-packaged driver/OpenCL compute libs combination.  On installation, PCLOS installs the free amdgpu driver.  To get the GPUs to crunch, I looked at what gets installed using the --compute option in the Red Hat version of the AMDGPU-PRO package installer (version 16.60).  I did this back in February this year.  It's a long story, but the GPUs seem to be able to crunch fine by installing just the compute libs this way.  In truth, I'd stumbled on a procedure that works without really understanding why.

The performance was good and the hosts did run for quite a while with no apparent issues.  I tended to reboot from time to time and I did make hardware changes progressively over time so it wasn't initially apparent that there was any real problem.  Now there are signs that there may be some sort of resource exhaustion issue (memory leak?) with a GPU software component.  I'm not a programmer so that's just a guess to explain what I'm seeing recently.

With a workable arrangement for RX 460s, I've been steadily bringing retired CPU only machines back on line.  Over a period, these have been stopped and restarted multiple times for various reasons so that none of them had long uptimes that were in sync with each other.  The odd machine would have a particular type of issue from time to time.  A lot of the hardware is quite old so I expected problems.  I have replaced capacitors on motherboards and in PSUs and even installed brand new PSUs and this has (until very recently) seemed to cut down the incidence of machines having regular stoppages/crashes/lockups - whatever.

Just over a month ago, a car demolished a power pole in a neighbouring suburb and it took more than 6 hours for power to be restored.  So every machine in the fleet got a restart at much the same time - it took 2-3 days to get my hardware damage sorted and things back to normal.  So for the first time, all the upgraded RX 460 machines had pretty much the same uptime.  Whilst a couple have been restarted in the period since, all those that hadn't (around 20 machines) have had an almost identical problem.  A common factor is that these 20 machines all had uptimes in the range 27-30 days.  To me, this was the key bit of evidence for some sort of resource exhaustion.

To make it very clear, the problem concerns a group of around 20 machines where the only remotely common hardware component is an RX 460 GPU.  The problem is restricted to the GPU because in all cases, the machines continue to crunch CPU tasks and can be remotely accessed with ssh or even BOINC Manager running on a server machine.  The machines don't normally have a screen, keyboard, mouse attached.  I only hook these up as required.  When I hookup to a 'good' machine, everything just works normally.  If I hookup to a machine having a problem, the screen is black and the keyboard and mouse give no reaction.  The numlock light is usually on but not always and sometimes cannot be toggled.  I used to just do a hard reset but some time ago I noticed that a hard reset is not necessary as a machine would almost invariably respond to the Linux 'REISUB' trick.  Just google it if you want to know about that :-).

So even though a machine is seemingly quite unresponsive, it is still running.  I've recently started logging in remotely over ssh to a problem machine and checking with 'ps' for the status of running tasks.  BOINC is still running and clocking up time.  FGRP5 tasks are still running and clocking up time.  FGRPB1G tasks are listed as running but don't clock up time.  If I try to use boinccmd to get the client to exit, the machine will lock up.

I've also started connecting using BOINC Manager on another LAN host and examining the tasks list of the problem machine.  CPU tasks are running normally and results get reported and new work gets downloaded.  GPU tasks appear to be running because elapsed time is increasing - way beyond what it normally should take.  I've seen as much as 20+ hours.  However % done is not moving and there is no movement in CPU time as disclosed by 'ps'.  Apart from typing Alt+Sysreq+R E I S U B on the connected keyboard, I can also trigger a remote reboot over ssh to restore normal operation.  But it has to be a full reboot because I can't stop and restart the client only without locking up the machine in the attempt.  My guess is that the graphics card is in an inconsistent state and needs a full reset.  Sometimes, a reboot is not enough.  Everything is apparently working but GPU crunching is much slower than normal.  This happens in maybe 10-20% of cases.  A shutdown, remove all power, cold restart sequence always resolves this.

When a problem machine is restarted, there is almost invariably no particular issue visible.  Crunching always correctly restarts from saved checkpoints and there are no compute errors.  Very regularly, one or more tasks in progress (either CPU or GPU or both) will finish quickly and upload - perhaps within seconds to a minute or so after restarting.  This happens so regularly that it can't be just coincidence.  Such tasks are in the followup stage after normal crunching has finished.  My guess is that something is happening in that followup stage that contributes significantly to resource depletion - enough to tip it over the edge whilst in that stage.  Whilst both types of tasks are involved, it happens surprisingly often with CPU tasks.  If it were a random event, you would expect it to be quite rare for CPU tasks to just happen to be in the followup stage.

I'm documenting this now because the chance alignment of so many RX 460 hosts with ~30 day uptimes has seemingly clarified the issue.  It's hard to believe that this problem is not some sort of resource exhaustion.  The other main GPU types I use are the older Pitcairn series HD7850 and the newer (but still Pitcairn series) R7 370.   All these use the old fglrx driver and do tend to have much larger uptimes without an apparent resource exhaustion problem.  At the moment they have ~35 day uptimes but I've looked back before the power outage and in August, quite a few had uptimes of ~235 days which just happens to coincide with the previous full power outage (a storm) on 6th December 2016.  Many others of them had uptimes around 210 days and 180 days and there were storms in Jan - March which caused power glitches that crashed some machines on particular power circuits.  I have a 3-phase power connection with circuits on different phases so at storm time (December to March) only a subset of machines might be affected by a particular event.

I'm hoping one of the Devs or perhaps the program author(s) may be inclined to think about the above description and give a better informed opinion on whether or not there may be resource exhaustion happening.  If so, is it the driver/compute libs or the app itself or a combination of these?   Is there anything to be done to better isolate/identify the true cause?

If anyone has any thoughts, please feel free to share.  My next step will be to download the 17.40 version of the Red Hat AMDGPU-PRO package and take a look at what the --compute option installs for that.  If things look feasible, I might try to get things working with that version to see if there is any difference in behaviour.