Budget Builds: An E@H Top 10 Computer for about $1200 USD

To celebrate my host reaching the Top 10 (named users) for the first time, I wanted to share a budget build guide. Many of the top hosts here at E@H are quite impressive machines, boasting a fleet of very expensive GPUs, and some of them would cost more than my car to build. I wanted to start contributing in a more substantial manner to E@H, but as a graduate student I do not have the means to build such a machine. Yet, I have managed to (however briefly) get a spot on the Top 10. So here is my guide to building the most bang-for-your-buck E@H computer.

Using a second hand website like Craigslist/Ebay/Kijiji etc,

1.) Find an old workstation PC with as many PCIe slots as possible, preferably dual socket.
2.) Find the highest core count CPUs that exist for that platform.
3.) At least 32-64GB of memory, the speed is not so important.
4.) Get a bunch of natively single-slot GPUs

The hardware does not need to be new. My version of this has hardware over 10 years old that still works great. The point of this build is to not spend a lot of money, so GPU waterblocks/riser cables/mining rigs are to be avoided, hence the single slot GPUs. In fact, you might be surprised that 2 single slot cards often have better performance and less power draw than a double slot card. I will assume this to be a dedicated E@H machine, so we will aim to use all of the CPUs and GPUs we can muster. Right now, workstations based on the LGA2011 Intel Xeon socket are pretty cheap. Examples of workstation platforms for LGA 2011-v3 socket:

HP Z840     (dual socket LGA2011, 5 directly useable PCIe, one half length PCIe, one 1x PCIe)
Lenovo P710 (dual socket LGA2011, 5 directly useable PCIe, one PCI slot)
Dell T7810  (dual socket LGA2011, 4 directly useable PCIe, one PCI slot, one 1x PCIe)

Some of these PCIe slots are 16x, some 8x, some even 4x, but that hardly matters to us as the All-Sky GW apps don't really care about PCIe bandwidth, and workstations usually have open-ended slots, so any card will theoretically fit. Even 1x slots can be used with the addition of a 1x to 16x PCIe mining riser, assuming you can fit that in the case somewhere, but that might be pushing the limit on PCIe bandwidth. PCI to PCIe 1x adapters also exist. Sometimes a full length GPU won't fit due to offset CPU sockets, so keep that in mind. You also need to be very careful that the power supply that comes in a system can indeed handle the total power draw of all the devices you plan to use.

The best CPU you can find for this socket is the Intel Xeon E5 2699A v4, which is easy to find second hand for a reasonable price in my experience, since they were used in datacenters years ago. With dual sockets, this gives us 44 total cores to feed the GPU apps as well as Gamma Ray CPU tasks.

As for the GPUs, Nvidia has a line of workstation grade single-slotable GPUs, the Quadro X4000 series. These are the Quadro M4000, P4000, RTX 4000, and A4000, etc. The highest I would dare to go on this list from a price standpoint is the P4000. Due to GPU boost 3.0, the P4000 gets a core clock uplift of 2.5x from the M4000, so it makes a lot of sense not to go lower on the list either.

I spent a while shopping around and bartering, and with quite a bit of luck, here is what I picked up:

1.) An HP Z840 workstation (enough DDR4 memory included) ($750 CAD, ~$550 USD)
2.) Intel Xeon E5-2699a v4 CPUs ($175 CAD, ~130 USD) each (knocked off $30CAD by trading the CPU the workstation came with)
3.) 5 Nvidia Quadro P4000 GPUs ($100CAD, ~$70USD) each

Together with some accessories like some PCIe 6pin to dual PCIe 6pin adapters I spent around $1650 CAD or around $1200 USD on my machine. In this HP Z840 chassis, 5 of these GPUs fit in the 3 16x and 2 8x PCIe slots, and the power supply can handle all of them and more according to the spec sheet.

My HP Z840 came with an Nvidia K620 GPU that I have fit in the top half length slot for display output (which is why BOINC reports 6 GPUs). It can't do much with the 2GB of VRAM except the BRP7 app. I find that I can keep all of the temps below 70C at less than 100% fans. It pulls a total of about 800 Watts during crunching, good thing I run it on university electricity ;)
 
I did get lucky with the seller on the P4000 GPUs, who accepted my offer of a bulk discount. These GPUs can be found for $200-300 CAD on ebay, so your milage may vary. I currently run this machine with 3x All-Sky GW tasks each (the most tasks I can fit on the 8GB of VRAM), which uses 15 of the cores and the rest are dedicated to Gamma-Ray CPU tasks. I find that running as many GPU tasks at once is crucial to overcome the CPU bottleneck in the Recalc region of the app, so that the GPUs stay almost always busy. If you want to replicate this build, be sure you get a CPU that can at least handle the GPU tasks you plan to run, as those have the best credit. Make sure to use Linux so that you get the CUDA version of the app. I haven't experimented with the GPU-only CUDA version yet.

Do you have any better ideas or suggestions? Feel free to discuss your budget builds!