GPU Crunching - What setup works 'best', what does it cost, what comparison metric is best to use?
Language
Copyright © 2024 Einstein@Home. All rights reserved.
Copyright © 2024 Einstein@Home. All rights reserved.
Gary Roberts wrote:If you
)
Gary this is great stuff. The concern I have is that other users may be daunted by the complexity and effort to give the full detail you gave, and not spot the loophole that I think you welcome somewhat less thorough contributions.
In particular I think the section which compares productivity and power for varying application loads and with and without the GPU is informative but adds considerable effort, and is not directly needed for what I consider the main goal of benchmarking total system productivity, cost, and power consumption.
Both to support your effort and to "test the waters" for a less fully populated contribution, I intend to take a stab at instrumenting my simplest and most modern host, intending to produce a post in your general format but omitting such tasks as removing the GPU just to allow a GPU-less power measurement.
I do have a comment on power, aimed at the general audience and not suggesting a change in your format--for people who own a meter which accumulates power consumption (all Kill-a-Watt models, and many other meters), time-varying power consumption is often better averaged by measuring the energy consumption over a known period, and doing the math, rather that looking at a rapidly changing watt display and trying to do an eyeball average. For meters with a reset button, such as the more expensive kill-a-watt model, this is easiest, but for others use of a watch, pencil and paper, and a bit of subtraction and division do the job.
In support of Gary's
)
In support of Gary's exploratory phase, here is a posting for my Stoll8 host.
I've started with the quoted text of Gary's Configuration 1, over-writing with my own data, and omitting portions I deem appropriate. I'll color any sections where I have intentionally differed from Gary's template, and think my change might be an improvement.
Configuration 1 - Hardware
[pre]
CPU: Intel Haswell Core i3 4130 $120
Motherboard: Asrock Z87 Extreme 3 124
GPU: EVGA GTX 970 Super Clocked 4GB GDDR5 337
RAM: 2x4GB DDR3 1600(PC3 12800) (G.Skill) 63
Hard Disk: 2 TByte Seagate hybrid drive ST2000DX001 100
PSU: 430W Nexus Value 430 75
[/pre]
Configuration 1 - Software
[pre]
OS: Microsoft Windows 7 Home Premium x64 Edition, Service Pack 1, (06.01.7601.00)
Driver: 344.60
BOINC Version: 7.4.36 64bit
[/pre]
Power Consumption and Productivity
[pre]
Power Usage Task Mix Av. Secs per Task Tasks per Day
Hardware Setup and Running State Watts kWH/day CPU + GPU CPU GPU CPU GPU Credit/Day Calculated Credit/kWH Comments
================================ ===== ======= ========= ================= ============= ============ ===================== ==========
System at idle - GPU installed: 55 1.32 Idle power with GPU
System crunching 2xFGRP4 + 3xBRP6: 212 5.09 2 + 3 23,400 10,095 7.38 25.68 5118 + 112975 119092/5.09 = 23,209 CPU + GPU cruncher
[/pre]
Comments
* While the CPU and RAM are operating at stock frequency and clock counts, the GTX 970 is operating at a moderate GPU clock overclock (1427 MHz as reported by GPU-Z) and a very substantial GPU memory clock overclock (1950 MHz). Stock clocked GTX 970s will not approach these performance results with current software.
* I have not tinkered with productivity vs number of tasks on the current BRP6 release, and don't plan to unless the long-promised higher CUDA release is indefinitely delayed--as that seems a better tuning base.
* I use Process Lasso affinity to assure that the two allowed CPU tasks never share the same physical core (i.e. I prevent "unluckly assignment").
* I use Process Lasso priority affinity and priority adjustment to try to give low latency CPU service to the GPU tasks.
* This machine is my wife's daily use machine. That and various misadventures mean the reported RAC has never quite reached the 119,092 theoretical shown, though it has spent over two consecutive weeks over 110,000, despite the short time 1.52 has been available.
*
Locally I pay about $0.15/kWHr for incremental use on top of my base. Rounding to exactly 15 cents for simplicity, that means I pay 76 cents/day for total system power on this system
* Assuming a 3 year replacement cycle for the agreed priced components (CPU, RAM, motherboard, GPU, boot drive, power supply) I've paid 75 cents/day for assumed replaced hardware.
* Not assuming anything about life cycle, the priced components total cost divided by daily credit output is US $.006935 /cobblestone/day. This is a somewhat unwieldy measure, so perhaps (looking forward a bit), we should state hardware price per million cobblestones/day, in which case this system would score US$6,935.25/MCobblestone/day.
RE: ... The concern I have
)
A very valid concern. I'd like all who are perhaps wondering about the level of detail I gave in my post, to understand one particular point. I did what I did because I was personally interested in seeing exactly where the power gets used and how crunching is affected by disabling certain types of tasks and noting the consequences. Having gone to the trouble of satisfying my curiosity, I decided others might be interested too. I'm certainly not expecting anyone else to do the same. I just found it interesting so I wrote it all down for anyone who cares to look at it.
Well, I'm really glad you spotted it :-). I was quite amused at your choice of the term 'loophole' to describe it. A bit like a 'get out of jail free' card - to save you from otherwise dire consequences :-). I'll be grateful to anyone who makes any contribution that documents just the normal operating conditions, just like you have done in your follow-up message - absolutely perfect. I'll comment further about that in a direct response to that message.
I'm hoping that we get coverage for a range of different GPUs, particularly the new releases. I have lots of GPU hosts, but essentially just two 'different' GPU types, both of which are starting to be 'out-of-date' now, let alone when the next round of the latest and greatest is released.
Thanks very much for the general comments about the ability of power meters to accumulate usage over a time period to get a 'better' average. In my case, I tried both a visual average of the fluctuating watts value and comparing that to allowing the kWH display to accumulate for several hours. I found that both methods gave pretty much the same answer to withing an acceptable degree of accuracy. I wasn't too happy with my particular meter's kWH system as the accuracy was only 1 decimal place at all stages but I found that if I took the time reading precisely at the point where the kWH display incremented, the average of the fluctuating watts reading gave fairly precise agreement with the accumulated kWH figure.
I've noticed other factors that can affect the accuracy of power measurement, quite apart from the basic accuracy of the instrument itself. Firstly, power use may change slightly from task to task. Some tasks seem to create a greater load than others of exactly the same type. The variation seems to be of the order of say 1-2% or thereabouts. I've noticed this by recording the limits of fluctuation every couple of hours or so over a period of a couple of days and noting that the 'average' of the range moves up and down by a small amount over time. I don't see this change while the same tasks are in play. The change seems to be related to one set of tasks finishing and a fresh set taking over.
Secondly, with CPU tasks, the power being used seems to increase slightly when the last checkpoint has been written and the final followup calculations are being done (that period where no further checkpoints occur). I happened to notice that with no change in the actual tasks being crunched, there was a small increase in the fluctuating range of values at a time when both CPU tasks were one step away from 100% completion. It was a single observation so I can't say for sure it was due to those follow-up calculations creating more load but it seems possible.
Of course, both of the above factors could be compensated for by measuring the consumption over a period of several days so that a large number of tasks would be contributing to the final average. I'm not at all suggesting that this is necessary. I consider that these variations are small enough to be neglected and that much shorter duration measurements are good enough for the purposes of this exercise.
Cheers,
Gary.
RE: ... I'll color any
)
Thanks very much for taking the trouble to publish the figures for your host. I'm very pleased to see the results for a GTX 970. Locally, your card costs $AUD500 which translates to close to $US400. I tend to baulk at paying any more than about $AUD200 for a GPU, preferably rather less :-).
I was relieved to see that your green section wasn't so much a 'difference' as an 'addition' to what I'd produced. I completely dodged the issue of how to account for capital cost. I just didn't know how best to treat it. I like what you have done. For the benefit of newer recruits who may not be familiar with the term 'cobblestone', just think 'credit'. The only change I would suggest to what you wrote would be to use thousands of credits (cobblestones)/day rather than millions. Somehow, having a capital cost of $US 6,935.25 to produce some amount of credit looks rather scary :-). We could define a kstone as a production rate of 1000 cobblestones/day. So the capital cost of your host is $6.94/kstone, and the running cost (for you) is $0.76/day.
My electricity cost is about $US0.24 per unit. If I look at the final entry of my second configuration (1xFGRP4 + 4xBRP6), and I assign a new component 'cost' of say $US150 for disk and PSU, the capital cost becomes $US520 which gives a figure of $US5.90/kstone and my running cost is $US1.01/day.
Cheers,
Gary.
I wrote the comment about the
)
I wrote the comment about the use of green when I thought I'd likely be changing some of the internals. As you can see, I did not, actually. I was running for the door knowing that by the time I got back the editing window would have closed, so it was a bit of a muddle.
I don't think the way I formatted the crucial capital cost points in my comments section was the right way, but do think that coming up with a single summary number for capital cost per unit productivity is needed, just as you came up with a single number for power consumption per unit productivity.
On the other hand, the final combination of capital and power cost to a single number is, as I argued before, problematic fundamentally, because it requires one to settle on an assumed replacement cycle time, AND, requires an estimate of power cost, both of which will be specific to a particular user.
In summary, for the final format, I think there should be a single overall capital cost per unit productivity number in the main, standard format section, and, perhaps, that users should be encouraged to populate the comment section with estimates of their replacement cycle time and local power cost per kWHr. Those assumptions would in turn enable the final step, specific to their situation of overall daily cost per unit productivity, and daily running cost. It may also be a good idea to encourage people to expressly figure the power and capital portions of daily running cost. I confess this last is part of my long-running campaign to get people to think of power as a real cost component. Because the GTX970 is an expensive card (and quite uncompetitive until then recent code changes improved the position somewhat) one might be surprised to find that for my situation power cost matches capital cost.
RE: METHODOLOGY For each
)
Thanks Gary and archae86, this thread has made me think about efficiencies and long term costs. Power costs are the interest for me mainly, over time they outweigh all others.
That said, if the thread is going to compare costs (amongst other metrics), (to build a cruncher from scratch, and run for its lifetime) then the following component costs are not being included and should be(?). Operating System, heat sinks for CPUs, PSU efficiency and Case costs (includes fans). Monitors, Keyboards and mice are not zero cost either to purchase or run, but are needed (by most) to crunch.
Time spent in setting up, and final shutdown are hidden costs, i guess to buy, assemble and install and clean down and dispose probably would take say two days. So a system which lasts 6 years will crunch two days longer than two 3 year systems back to back. Not to mention you get a free weekend, and less bits to throw away!
Cases and case fans.
I don´t think the case cost makes much difference to performance, other than the obvious, like a HDD, you probably need one that works (for crunching).
Overheating does affect performance either directly or indirectly (summertime schedules etc), and component lifetime.
Multiple GPUs share the same case, and so the incremental cost to crunch more is less than a second cruncher.
Operating Systems
Like a HDD, you need one. Some are not free, and different setups need different OSs (number CPU cores, 32/64bit etc).
CPU Heat sinks
Sometimes you need one, the stock comedy ones from Intel are not really up to crunching, especially in warm weather next to a big GPU baking cobblestone pies. An efficient one over a long time will pay back fan power costs.
Component lifetime
I would be very disappointed if my Gigabyte/nVidia build only lasted 5 years and to be honest I expect it to crunch 24x7 for 7years, touch wood. I have only opened the case a few times, once to add a second GPU a few years ago, another recently to see what space i had for a third. It is approaching the 5 yr mark now and putting out ~95K RAC for around 320-350W, the GPUs usually 60C and rarely they reach 70C and the CPU temps are 45C approximately although not all cores are running 100%, typically averaging around 50%.
My recently deceased, Acer Aspire which had a very easy life doing nothing, seemed to give trouble in its fourth year and died in its 7th year, the last few years it simply would hang after a while. I should have put it down two years ago, as the time spent trying to fix it and get it crunching, for meagre returns, was a waste.
Change the lifetime from 3 to 7 years makes a big difference in the calculated results and conclusions.
PSU efficiency
I know i go on on about this...
Small power savings 7x24 over a long time, make a difference. Using the USD 0.24USD figure per KWhr below, a single watt saved is 2.1 USD / year or 14.7 USD over 7 years. One other thing is certain, power costs will rise over time.
PSUs old and cheap, probably fall into 75% efficiency, latest Platinum into 90+ efficiency so it is important to know the PSU efficiency at the load you expect to run. There are many uncertified PSUs which are very efficient and will pay for themselves if replacing an old PSU. archae86´s 430W Nexus Value 430 below is a perfect example of a PSU that is 80-85% plus, but not certified.
I will try to put something in about my new Gigabyte/HD 7990 build shortly.
I think it is quite a chore to complete all the information, i had thought a little bit about - could i write an awk script to gather it, with help from aticonfig, dmidecode, hwinfo and others.
RE: PSU efficiency I know
)
One other thing about PSUs, the more efficient they are the LESS heat they put out, so a better quality heat sink is ALSO cooler to run than a cheapy one. If I had known this way back when my basement would not now be 86F when the outside temps are in the mid 50'sF!! AND I would not be looking at ways to cool it down as much either! ALL my newer PSUs are of the higher quality ones, no more cheapys for me.
RE: On the other hand, the
)
This is all very true so I guess there are at least two separate ways to go forward from here. The first would be to attempt to identify what it actually costs to volunteer a given computer for crunching purposes, and for that, the problems you identify make the exercise quite difficult. You would also need to account for the 'costs' of maintenance and ongoing management, etc, and these too would be dependent on the commitment and skills of individuals and how they valued their time. I've come to the view that this is not really feasible for us to attempt.
The second way forward would be to look at doing a simple comparison by making certain assumptions. The aim with this would be to come up with some sort of relative value that suggests (all other things being equal) that system A is 'better' or 'worse' than system B by some relative amount, based on specific assumptions which are laid down at the outset. The fundamental assumption would be that systems being compared are likely to have similar lives, similar rates of hardware failure, similar maintenance and management costs, similar peripherals costs and similar residual values at the end of an agreed lifespan. With that assumption, we put the onus back on the individual for how they budget for all these issues and whether or not they wish to vary the assumed numbers.
As a compromise, we could choose a 4 year lifespan and an electricity cost of $US0.25/unit. US prices may well be currently cheaper but the rest of the world is probably dearer already and prices are just going to rise further over the 4 year lifespan anyway.
If you have the time and are prepared to devote it, I'm sure many would be interested in seeing your thoughts developed further. I always find it difficult to visualise what is going to be most useful until I set it out and can actually see it and have a good think about it. Along the lines of making certain assumptions in order to simplify things, here is a further offering of what the format could look like. As usual, this is quite open for comment and criticism. I've chosen to do this on a host whose CPU should be rather more power efficient that what was used in the very first config I posted. I've changed the format of the table to suit the assumptions.
Configuration 3 - Hardware
[pre]
CPU: Intel i5 3570K (3.4GHz) @ 4.0 GHz $220
Motherboard: Asrock Z77M 90
GPU: AMD HD7850 2GB (2014) 190
RAM: 2 x 4GB DDR3 1333MHz 60
Hard Disk: old HDD used - replacement cost (new 1TB) is 55
PSU: old PSU used - replacement cost (new 450W gold) is 75
----
Total hardware cost for the agreed items is $690
----
[/pre]
Configuration 3 - Software
[pre]
OS: PCLinuxOS 2014.04 64bit - kernel 3.12.16-pclos3 0
Driver: fglrx driver and OpenCL libs from Catalyst 13.12
BOINC Version: 7.2.42 64bit
Total software cost is $ 0
[/pre]
Daily hardware and software capital cost
For the assumed 4 year life of the components, the figure is (690+0)/1461 = $US0.472/day.
Power Consumption, Productivity and Running costs - Power is assumed at $US0.25/unit (kWH)
[pre]
Daily Power Use Task Mix Av Secs per Task Tasks per Day Credit/Day
Hardware Setup and Running State W kWH $US CPU + GPU CPU GPU CPU GPU CPU GPU Calculated Credit/kWH Calculated Credit/$US
================================ === ===== ===== ========= ================ ============= ============ ===================== =====================
System at idle - GPU installed: 60 1.440 0.360 -
System crunching 2xFGRP4 + 4xBRP6: 211 5.064 1.266 2 + 4 16,450 17,733 10.50 19.49 7280 + 85752 93032/5.064 = 18,371 93032/(0.47+1.27) = 53,528
System crunching 3xFGRP4 + 4xBRP6: 226 5.424 1.356 3 + 4 17,139 17,830 15.12 19.38 10481 + 85285 95766/5.524 = 17,336 95766/(0.47+1.36) = 52,388
[/pre]
Notes
1. The system at idle has been included above just to inform of the power used to have a system on standby.
2. The 2nd config is the standard config I have been using on this machine since the HD7850 GPU was first installed (probably around 18 months ago).
3. Now that CPU support for GPU tasks is a lot less, I thought I'd try the 3rd config to see how well 1 free core could support 4 GPU tasks.
Comments
Capital cost = $US0.47/day and Running cost = $US1.27/day
* This config produces 53k credits/$US. I would suggest that if people wanted some sort of single performance metric, this might be suitable. I would also suggest the individual costs/day should also be highlighted.
* Despite having to make assumptions, the above calculations show very clearly the importance of running cost in the overall cost equation.
* If you use the above results together with those for Config 2 from an earlier message, you can appreciate just how much extra power is used for crunching extra CPU tasks. The chosen mix of CPU tasks alongside GPU tasks will have a significant effect on running cost.
* Whilst I fully understand the effect of PSU efficiency, the above figures were obtained using PSUs in the 6-12 year age bracket. They were carefully chosen at the time for high 12V output and the best possible efficiency commensurate with a reasonable price. I believe they are operating at close to 80% or even above. 80+ certified units were either unheard of or prohibitively expensive at the time. I've had extremely good luck with PSUs. Very few have failed, despite 24/7 use for many years.
* In order to keep the above table simple, I haven't published any power data for configs having either 0 or 1 CPU tasks. The figures are 174W and 196W respectively. There is a rather more uniform increase in power used for each extra crunching core than there was for Config 1 in the earlier message.
* I was pleasantly surprised to see that GPU crunch time essentially remained constant and CPU crunch time increased very little when going from 2 to 3 CPU tasks on this host.
Cheers,
Gary.
Gary, On further
)
Gary,
On further reflection my thinking has moved toward your current position on more than one point. Specifically:
1. While I think it important to prepare separately the purchase cost total and cost productivity, versus the daily power consumption, I now agree that it is best to prepare a final overall cost productivity number which combines the two, using an agreed standard assumed lifetime and assumed power cost. So long as the two primary input variables are displayed prominently, an interested user can easily adjust the lifetime and power cost assumptions to their liking and get new answers with the simplest of arithmetic.
2. I had not thought about including software cost, but for those of us building (legal) Windows systems, the Windows license is just as much a part of new system cost as is the motherboard. I would resist, however, adding in MS office, and less BOINC-essential things.
3. While I suggested 3 years assumed lifetime (and others like 6 or more), your 4 is quite reasonable. I may have been unduly influenced by the non-characteristic power savings of the most recent nvidia offerings, which have made the economics of continuing operation of older power hogs less favorable than usual recently.
4. While formally and historically I like the official term cobblestone, I agree that it is currently so little used in discussion forums that it would be a point of confusion--better just to say "credit" as you do.
Some other points:
1. While capital cost means the same thing to you and to me, I suggest the terminology "purchase cost" as more likely to be similarly interpreted across the user base.
2. I don't spot in your latest format a summary number for purchase cost productivity. While it does go as a component into your final combined productivity number, I think it should be exposed separately. While personally I favor the daily credit/cost form, you have chosen cost/credit, so for consistency this should, I think be present as purchase cost productivity in the form of (purchase cost)/(kilo-credit/day), and be presented prominently.
Within a day or two, I'll try to collect my thoughts and prepare a revised format listing. Just to complicated matters, I'll use one of my dual dissimlar GPU systems. Nobody would emulate mine for a new build, as they include out-of-date GTX 660s, but it may be worthwhile to present a multi-GPU case for formatting or policy concerns.
Peter, Thanks very much
)
Peter,
Thanks very much for continuing to give your thoughts. I have no problem with most of your points but I'm a little puzzled about your second 'other' point:-
I've tried playing around with this and I've split out the 'purchase cost productivity' from the 'running cost productivity' and produced the following example. I'm not sure if this is what you were looking for. To me it seems to add confusion rather than clarity to the information being conveyed.
[pre]
Credit/$Purchase Credit/$Running Credit/$Overall
================ =============== ================
93032/0.47 = 197940 93032/1.27 = 73254 93032/(0.47+1.27) = 53,528
[/pre]
Actually, I don't believe I have :-). Perhaps you really meant that you preferred cost/credit because, in your contribution of the data for Stoll8, you quoted the figure of:-
The figure I came up with was definitely credit/cost since the final column in the table was "Calculated credit/$US". I highlighted the overall figure of 53k credits/$US in the 2nd bullet point in the comments section. I definitely want to inform people of how many credits they are likely to be awarded for every dollar they spend. It's good to know the split between the purchase and the running cost components of their contribution but I don't really see the value of apportioning a specific number of credits to each component. Maybe I'm not understanding exactly what you are asking for.
It's just dawned on me that your point 2. may actually be a typo and you really want cost/credit rather than the other way around. If so, the above example would appear as (using units of million credits - MCredit):-
[pre]
$Purchase/MCredit $Running/MCredit $Overall/MCredit
================= ================ ================
0.47/.093032 = $5.05 1.27/.093032 = $13.65 1.74/.093032 = $18.70
[/pre]
This is certainly much less likely to confuse anybody. Now that I can see it, I quite like it :-).
In summary, I very much like being able to state the assumptions clearly and give a final single figure result (for my example) of 53k credits per dollar spent. I also see the usefulness of telling people that earning a million credits will cost $18.70 and specifying the actual split. It would be very simple to include both methods of presenting the data.
Hopefully, your post did contain a typo and this is what you really wanted.
Cheers,
Gary.