GPU crunching on Linux - Fun with AMD fglrx drivers

I've just just achieved a working HD7850 GPU equipped system with an i3 3240 CPU and the Catalyst 13.12 set of drivers from the repo, not the AMD website.

First some background. Skip it if you're not interested.

The system was running catalyst 13.4 drivers and was commissioned originally by an OS install followed by the complete removal of the repo supplied fglrx drivers (3 separate packages were removed). This was followed by the fresh installation of catalyst 13.4 driver package from the AMD website and then the 2.8 AMD APP SDK with rebooting between each separate installation. I didn't bother running aticonfig. I just saved xorg.conf from the original working 13.4 system and moved it back into place after reinstalling the AMD website supplied catalyst 13.4 package. That installation ended up having a somewhat larger xorg.conf (which I also saved). I was keen to see if the previous xorg.conf would still work. It certainly did so I didn't worry about the second larger version.

After completing the above setup the machine set to work crunching BRP5 GPU tasks 4x and a single FGRP task since I had set the number of cores preference to 75%. It has been operating like this for nearly two months. The only change has come with the advent of FGRP3 GPU tasks.

The distro I use has a number of advantages for me. It follows the 'rolling release' model but it also puts out regular images covering a range of DEs. There is a KDE MiniMe version which is quite small and very fast to install. Every three months there is an updated .iso which I load onto a 2.5" external USB drive. On the same drive I also keep a complete mirror copy of the repo system. Currently it's about 18GB and I update it with rsync every fortnight. Yesterday's update took less than an hour for about a 1.2GB download.

So to bring a new system online, the OS install (due to the speed of the live USB) takes about 6 mins followed by a complete update from the local repo which takes maybe another 5 to 15 minutes depending on how many updates are needed. I've seen over 300 packages get downloaded in a couple of seconds with less than 15 mins for full installation. Sure as hell beats updating over the internet. I use synaptic which is a GUI front end for apt and the point and click interface is just so easy.

After updating, I install any extra libs for BOINC. At the moment it's just wxwidgets as used by BOINC Manager. Then (if there's 4GB or more of RAM) I install a PAE kernel since the OS I'm using is not 64bit.

The most time consuming part is configuration. I want each node on the LAN to have a static IP. I want all the important and often used utilities to have identical look and feel, etc. Eack node must know what file shares are available and a new node needs to get various stuff from these shares. Each desktop needs to look identical with the same array of icons for doing repetitive stuff. These sorts of things probably take an hour or so.

I never install BOINC by running an installer. I have templates on a file share that contain everything needed. So to get BOINC going I copy a template into position and that's it. I don't use BOINC from a repo. I create a template for each version and populate it with the files from Berkeley. Everything is located under /home/gary/BOINC/ which is on its own partition on (usually) a 20GB drive. I acquired a bunch of these many years ago and they rarely seem to want to die. There are 3 partitions - 6GB for the root partition, 2GB for swap and 10+GB for /home. I never have a problem with disk space. The templates from the file share contain all 'common' files that would otherwise get downloaded from project servers. There's even a template state file (client_state.xml) which contains the identity of the new machine. I like my new hosts to reuse the ID of one of my previous hosts that is now decommissioned. A simple edit of the state file template achieves this. The final step is to click an icon on the desktop which launches BOINC in daemon mode. Benchmarks are done and crunching starts very quickly after that since there's only task data to download.

So, now for the nitty gritty. I had planned to completely reinstall the OS but I tried the following instead. These are the precise steps.

1. Prepare AMD APP SDK 2.9 in case it was needed. Edit the perl script so that it would look for /usr/lib/fglrx-current/libamdocl32.so in order to detect a catalyst driver installation.

2. Stop BOINC and run synaptic to do a full update. It found 230 packages to update and around 20 new ones to install. Catalyst 13.12 drivers were not in this list so I added the 4 extra packages from the repo so that all fglrx-current stuff would get installed including a package that had opencl in its title. The full update took 10mins to complete with only info messages and no complaints in the output. These included messages about kernel module installation and a new version of KDE so a logout/login was required.

3. I rebooted the machine and there were no issues with that. I went into the control centre and checked the graphics hardware. It told me (as usual) that a proprietary driver was available and did I want to use it? Yes. I checked xorg.conf and the fglrx driver was listed there as usual. I rebooted again.

4. This time I started BOINC. It started cleanly but, as I half expected, missing GPU messages for all the tasks. At least it's smart enough not to trash the tasks. So on to plan B for which I had prepared.

5. I installed the SDK with the modified perl script. I had checked and found that libamdocl was in both /usr/lib and /usr/lib/fglrx-current and they were of different sizes. The one in fglrx-current must have come from the repo install so I was happy to let the perl script look for that one. The SDK installed without complaint so I immediately rebooted.

6. This time when I started BOINC there were no missing GPU messages and everything is now running again - both CPU and GPU.

While I've been composing this message I can see there has been a good speedup of the BRP5 tasks that were in progress so I reckon this host will now be on a par with the i3 4130 I repaired/upgraded previously. It looks like sub 17Ksecs for tasks running 4x. They were previously taking just under 19Ksecs.

Yes, of course, because the beta driver from the AMD website puts stuff in /usr/lib where the SDK install script can find it.

In my case, the ultimate aim is to use the 13.12 drivers in the repo and have them just work like the nvidia stuff does. I can now achieve most of that by ensuring the SDK install finds the repo installed stuff in /usr/lib/fglrx-current because I'm sure that's where future libs from future catalyst drivers will go for my distro. My next step is to work out exactly what is missing from the repo stuff (that the SDK provides) that allows BOINC to see the GPU. That way I should be able to do away with the SDK altogether by just convincing the package maintainers to provide it in their fglrx-current-opencl package.

I'm sitting in front of one of my 2GB 7850s at the moment. The card cost me $AU149 which is about $US134. I can see 1GB GTX750s for about the same money or 2GB GTX750Tis for $30-50 more. My machine is plugged into a kill-a-watt meter and it's drawing 155 watts from the wall. It's on track for a RAC of 70K probably more if the catalyst 13.12 speedup (not yet applied to this host) works here as well. I'll be doing it in the morning and it'll be interesting to see if there is any change in power consumption.

In the '... Maxwell and such' thread that I imagine you're referring to, most of the discussion seems to be about 750Tis. Because of the memory requirements of FGRP3, I don't think I'll ever buy a 1GB card again. Out of interest, I might buy a 2GB 750Ti and run it against a HD7850. I'm not in any hurry as I don't think I'd see much benefit particularly when you take the extra capital cost into account. Also, according to this report in that thread, a machine getting a RAC estimated to be 39K is drawing the same power from the wall as mine. Of course, it's hard to draw comparisons based of other people's setups.

Yes and Yes :-).

The first GPUs I bought were nvidia - a 550Ti and (much later) 650s and a single 650Ti. The 550Ti went into a Windows machine at the time when the first GPU app at this project was being improved. I decided to see if it could do better under Linux. The install went smoothly but the GPU wasn't detected by BOINC. I decided, using the synaptic package manager, to search the repo for anything with 'cuda' in the name and immediately found an 'nvidia-current-cuda' package with a description that said something along the lines of "Not installed by default. You only need this if you want to do high performance computing on your GPU." So I installed that and everything just worked, with a significant performance gain over what I had been getting from Windows.

I started using PCL in around 2007 and now I run it on virtually everything I own. Originally it was a fork of Mandrake Linux (Mandriva) but it's quite separate now. I've never used Ubuntu or any of its derivatives so I know nothing about their repo system or what "PPAs" are. The only other distro I have any experience with is openSUSE. If I hadn't been using PCL, I'd probably be using that now, simply because it has a repo for AMD GPU drivers that immediately work for GPU crunching. After I had some 650s working on PCL, I decided to buy a HD7770 to compare with. I couldn't get it to work on PCL so I looked around for a suitable distro and found SUSE. You need to add a special repo but once you do that it's very straight forward. That machine with the 7770 is still running SUSE and the drivers have been upgraded a few times without problems. Initially it performed around 20% worse than a GTX650. With better drivers it's now outperforming the 650s by quite a bit. It's the reason I've become hooked on 7850s at the moment. At the time of the driver improvement (around catalyst 12.10 or just after that I think) I bought a 7850 and installed SUSE. I was quite blown away by the performance.

SUSE is a very polished distro but it's not for me. It's hard to explain but I tend to feel that it is more complex than I'm comfortable with and that it hides things away in order to protect the user from himself. Now what other product on the market does that to extreme :-). I like things to be simple, open, and easy to find. I always like to understand how things work and I'm actually quite averse to change - particularly cosmetic change or change for change's sake. The more I delve into the innards of PCL, the more I feel it suits my style of doing things. It may well be regarded as being 'way behind the times' to the younger generation :-).

So, to anyone reading this and wanting to use Linux, just be aware that you really do need to find what suits you from the vast array of choices out there! That's probably not going to be a quick or easy thing to do.