GPU Crunching - What setup works 'best', what does it cost, what comparison metric is best to use?
Language
Copyright © 2024 Einstein@Home. All rights reserved.
Copyright © 2024 Einstein@Home. All rights reserved.
RE: RE: .... Please feel
)
Thanks for joining in. I value your input, even if I'm going to disagree with some things :-).
No. I believe that for most average people, that motivation comes a lot later, if ever. Most newcomers are attracted by some particular WOW factor. "Wow, this Seti thing is supposed to find aliens; wouldn't that just be soooo cool!!!" Or, "Hey, did you know there's this place where you can sign up your computer to look for pulsar thingamabobs? Don't know what the heck they are or why anybody wants to find them, but if ya do bag one, they gave ya a special signed certificate - wouldn't that look awesome to show off!!!"
Of course, I'm exaggerating but people respond to exotic and unusual circumstances by joining in to see what the deal is. A good fraction tend to lose interest rather quickly and a lot may drop out rather quickly, but quite a few will leave their computers on 'auto-pilot' until something changes to break the connection.
My impression is that a smallish fraction of newcomers eventually become long term contributors and nowhere near all of those add to or upgrade their hardware just to "generate the maximum RAC". Many are content just to support the science being done and are happy to do that with whatever systems they currently have without ever spending time and money chasing RAC.
I believe that very few people here are chasing credits just for the sake of amassing a huge number. Any that were, have long since gone elsewhere to those projects that pay ridiculous amounts. Most people who stay the course here are genuinely interested in the science and simply use credits as a measure of performance, of their contribution, or other 'fun' purposes like competition with others.
I also believe that some spend time any money improving the efficiency of their hosts, just because they can. The motivation isn't just a bigger RAC. Most of the motivation comes from the 'challenge' to make something perform better - produce more output in a shorter time. That's why optimised apps create a buzz around a project. Suddenly, you have a new tool to explore and play with that performs better than the old one.
I don't think these restrictions are really necessary. It's still quite possible to come up with a figure of merit for the Einstein component of multiple project production. Ditto for machines that aren't always on. Ditto for machines that have more than one GPU. If people want to report such machines, they would just need to do some calculations to figure out the daily production and power use as if they were running only Einstein or for 24 hours per day operation.
Sure, but the RAC numbers would have to be long term averages rather than single value read off at the time of posting. You should also have column(s) for power consumption under specific conditions with units of say kWH/day.
I would decline to choose without knowing capital cost and power consumption. I doubt that very many people at Einstein have a 'RAC at any cost" mentality.
Yes, it's straightforward but it's misleading without comparative power consumptions. Is your AMD GPU a 7850 or a 7870? There's a fair bit of performance and power difference between the two. I have no idea of how a GTX770 stacks up against either of the AMDs.
If it doesn't already, it absolutely should. The whole point of the exercise is to help people understand what it really costs to run a GPU crunching box. It's certainly not just to say, "buy this because it gives the biggest RAC."
People will read what they want to read. The object is to provide information that might take time and effort to calculate but can be very simple to digest. For example, if the summary said, "Setup A costs $B and uses C kWK/day and produces D credits/day, whereas setup W costs $X and uses Y kWH/day and produces Z credits/day", don't you think people might actually read and think about that?
None of the above should be seen in any way as a criticism. I really do value your input and it's really good to have other people's perspectives. I certainly do take all your comments seriously, particularly the prospect of losing readers. I've dusted off a power meter and I'm going to pick a host and experiment with a couple of different GPUs and see if I can come up with a presentation of results that informs but doesn't alienate the readership. If others are willing to do the same it would be great. Having different approaches open for comment is bound to produce a better final product.
Cheers,
Gary.
RE: RE: Now if power
)
The problem with including it is that the power used is exactly the same here in Northern Virgina USA where I live and in Germany and in Russia and, and, as long as the same cpu or gpu is being used. The costs will be radically different depending on physical location, but the power used will be the same. So if you have a GTX770 in Germany and I have the same one here in NO VA then while our costs will be different, our RAC should be similar.
Even the power costs are different for me in NO VA then for someone in Maryland or Florida or Ohio or Oregon or California. What I'm trying to say is that YES it's an important consideration, I just don't know how you could fit it into the chart without lots of explanations. In some States here in the US you can even do a 'co-op' type power thing where you could pay a totally different amount, per kwh, then your next door neighbor does for example. My power company charges me one rate for the first x amount of kwh's I use, then charges a different rate for the next x amount of kwh's I use, how do I figure out which part my gpu was in?
RE: If it doesn't already,
)
Gary,
I think there is a disconnect here. When I joined E@H I was/am interested in helping E@H achieve its goal(s). E@H like other distributed projects has enormous amounts of data to evaluate and this requires help from outside sources. I was/am happy to contribute my computers and electricity. From the outset credits were interesting but have nothing to do with why I participate. They do however provide a metric by how well my machine is doing against someone else's machine with similar hardware and circumstances. Maybe my logic is flawed but is it not true that the higher credits your machine generates the more WUs of a given type its chewing through? And the more WUs you can chew through the more likely you are going to find that pulsar or GW? When I started I did not know how much more of an electrical load a GPU would impose because I did not have one - games do not interest me. Cost did not concern me either because by joining a distributed project I knew I would incur expenses and was/am willing to assume them providing that they are reasonable. My interest in daily RAC is because it provides me with a way to evaluate the performance of my machine(s), not because I am going to win a toaster. My assumption was/is when you join a project you accept that its going to co$t you.
When I joined if I had seen that an AMD Pitcairn (an R9 270x) could generate ~100k daily credits I would have probably bought it over the current NVIDIA units I have on other machines. Also I have noted that the AMD I have runs cooler than do the NVIDIAS. With NVIDIA GPUs I had to install the "fan slider" on my Linux boxes to maintain an RPM to hold 65C in the summer. Again another factor to consider.
I am not going to purchase wall monitors to evaluate my electrical consumption because 1. I do not want to spend the money and 2. my electric bill will dictate if there is a need to retire some machines during the hotter months to cut expenses. It will be my average daily RAC for each machine that determines which machines go off line and which stay on line during the dreaded summer should I need to conserve.
I have seen some other members running Pitcairn GPUs and their daily RAC differs from mine. My questions to them would be: 1. are you dedicated to E@H or are you time slicing with another project(s). 2. are you running 24/7?, etc. It would be nice to know why/how they are achieving a higher/lower daily RAC then me when running the same GPU. Is is their MB, BIOs seetings, etc?
Are these not the questions we are seeking answers to?
RE: My interest in daily
)
Exactly.
RE: RE: RE: Now if
)
There is no problem with including "it" - "it" being the power consumption in kWH/day. If people don't have to pay for their power use they may choose to disregard "it". If people want to know the dollar cost, they will multiply "it" by their local cost per unit. The whole purpose is to highlight the power requirements of different GPUs in case that is an important consideration in the choice of GPU. I imagine it will be to most people.
You don't even try. You show the power consumption only and let people work out the cost for themselves.
You split your consumption into two categories - necessary consumption and discretionary consumption. Whatever you deem to be absolutely necessary is costed against the first x kWHs limit and then (if that is exceeded) the balance is apportioned at the different rate. Once all your necessary consumption is accounted for, your discretionary consumption can easily be costed. So how you cost your crunching consumption depends on whether you regard it as 'necessary' or 'discretionary'.
Cheers,
Gary.
RE: I think there is a
)
I don't think so. After reading this message, I'm comfortable that we are not really in major disagreement. Everyone will have their own particular way of assessing things and working out what's best for them and that's as it should be. At the end of the day we are both interested in supporting the project the way we think is best.
No disagreement there.
Again, no disagreement. But that isn't the point. Let's say that GPU A will crunch 1000 tasks in the time GPU B takes to do 600. Lets also say that GPU A uses 100kWH to crunch the 1000 but GPU B uses 50 kWH to crunch 600. The question is then which GPU would it be best to use? The answer will be different depending on what is important to the individual making the assessment. If you are just maximising your support for the project you may choose GPU A. If you want to support the project in the most cost effective way, you would place a cost on the power consumption, as well as the capital costs of each set of components and then make a decision. It could go either way, depending on circumstances, as well as other factors not even mentioned. It's very much a personal choice - made with the benefit of at least understanding the power consumption, even if you ultimately decide it's unimportant to you. I want to include that information because it will be important to some.
Neither you nor anybody else needs to purchase anything if they don't want to. I reckon there will be enough people who have power meters, or decide to purchase one, for there to be power consumption information about quite a wide range of GPUs. This thread is not about coercing people into spending money - it's about discussing what information should be included in any attempt to produce a comparison of GPUs suitable for crunching. If we don't get sufficient coverage, I'd be quite prepared to purchase particular GPUs just to fill in some gaps.
Just a couple of suggestions. You have left out one of the biggest factors that make this a 'hard to do' comparison - GPU utilization factor. Also, you shouldn't be comparing on RAC, for all the reasons you allude to. Check the tasks list for such hosts and get a feel for the average run time per task. You may be able to guess the GPU task concurrency from that. If you can, then you may be able to do a useful comparison. You will probably find that most hosts you look at are producing less than you simply because their owners don't have the same level of commitment and degree of technical skills that you have. If you do find one that is producing more than you (shorter run times for the same concurrency) then you should look deeper into why that might be.
We may well be wanting answers to those questions - but not in this thread. This thread was supposed to be about how to evaluate different GPUs for crunching purposes - what GPUs work 'best' for some agreed definition of 'best'. I suggested some sort of TCO figure but archae86 pointed out the difficulty of assigning a dollar value to power consumption so I was very glad to accept that and look towards having a metric based on capital cost and daily output together with a power consumption figure. If the daily output is based on average crunch time (not RAC) it doesn't matter if a machine is not crunching 24/7 or is working on multiple projects.
Cheers,
Gary.
I've shifted the discussion
)
I've shifted the discussion of the usefulness, or otherwise, of RAC to this new thread. I trust that nobody is inconvenienced by this.
Cheers,
Gary.
Hi Gary, RE: I've
)
Hi Gary,
Fine by me:-)
Cheers,
Cliff,
Been there, Done that, Still no damm T Shirt.
I found this on PG this
)
I found this on PG this morning, it gives the performance of alot of gpu's:
http://www.primegrid.com/forum_thread.php?id=6113
Specifically it points out the hit that dual precision takes on both Nvidia and AMD cards, but it also gives the FLOPS for both 32bit and 64bit applications for each card.
I've now had some time to set
)
I've now had some time to set up a couple of machines and to do some power consumption measurements under a range of conditions. I've chosen two particular configurations to play with. The first is a six year old Intel Q8400 machine that has spent most of its life crunching CPU tasks only. I've added a HD7850 GPU and I want to document the costs and effectiveness of extending the life of an older machine in this way. The second configuration is a brand new 'budget' style crunching box. What does it cost and what do you get if you just want to add a pure crunching machine.
Of course there are many other scenarios that could be chosen. For example, a person needing a new daily workhorse who is prepared to donate the 'spare' cycles is quite unlikely to buy a bottom of the range machine like the second configuration above. That person will most likely have a higher spec machine (probably with a higher spec GPU) suitable for its intended primary purpose. I don't really have such machines, certainly not with higher end GPUs. I'm hoping to encourage people who own such machines to contribute their numbers so we can get a range of comparisons.
METHODOLOGY
I wish to document the various steps I am using to do the measurements. As is always the case, people tend to come up with ideas for improvement so the methods used and the presentation of data could easily change as suggestions are received. This is all up for discussion.
For each configuration, I intend to state the true retail cost (eg something like newegg prices) for a standard set of components in the machine - motherboard, CPU, RAM, HDD, GPU, PSU - the things you actually need to crunch. You obviously need other peripherals, etc, but there's a wide range of personal choices (and prices) so people can work these out for themselves. I also intend to measure the power consumption in kWH per day, using a kill-a-watt style power meter. To show what happens when going from 'idle' to full bore crunching, I intend to produce figures at each of these stages.
* The machine booted to the desktop, but at idle with GPU installed.
* The machine with no GPU installed, crunching CPU tasks (FGRP4) only (all cores).
* The machine with GPU installed, crunching CPU tasks only on all cores - GPU idle.
* The machine with GPU installed, crunching GPU tasks only - all CPU cores free.
* The machine at full bore, crunching a suitable mix of CPU and GPU tasks.
I intend to calculate the theoretical task output per day by averaging the elapsed times in seconds over a significant number of completed tasks. If N tasks of a given type are crunching simultaneously and if the average elapsed time to crunch such tasks is S seconds, the task output per day (TOPD) for each task type will be calculated using the formula, TOPD=86400xN/S. I will calculate this figure separately for CPU and GPU tasks when both are running.
From tasks per day, I will calculate credit per day on the assumption that all tasks eventually validate. The power consumption in watts will be measured for each of the above six states. This will be converted to a daily consumption in kWH. If the credit per day is divided by kWH per day, the resulting figure will be the credit per kWH that the machine is capable of producing. So, at the end of the exercise, the reader will be able to see the cost of the specified components at the time of purchase and the credit/kWH that the machine is capable of producing under various conditions.
I think it will be quite interesting to see how the numbers fall :-).
Below are two configurations of mine. I've done the basic power measurements but it takes quite a bit of time to gather reliable average elapsed times for both CPU and GPU tasks, separately and combined. To get an initial draft output, I'm basing some numbers on very low counts of completed tasks. I'd like comments on the format so that when I have more reliable data, I can change both format and content if necessary. I'm not expecting that others willing to contribute will want to test all the setups that I'm listing here. I'm just hoping that people might be inspired to contribute something - perhaps just what they consider to be their most productive setup.
For each config, there is a list of hardware with the approximate retail price, when known, in $US. For non-current hardware, I have used the local retail price at the time of purchase, converted to $US, using my recollection of the exchange rate at the time. For current hardware, I am using the Newegg website. These seem to be cheaper than local prices in a lot of cases. The first config was built in 2009 as a CPU only crunching box. A year ago, the GPU, plus an extra PSU to power it were added. The second config is a recent addition so the prices were as of about 5 months ago, except for the GPU (12 months ago).
As I said earlier, all this is just a draft submission and everything is up for discussion. If you think it can be done 'better', please make suggestions.
Configuration 1 - Hardware
[pre]
CPU: Intel Core 2 Quad (2.66GHz) @ 3.16 GHz $180
Motherboard: Asus P5KPL AM/PS 50
GPU: AMD HD7850 2GB (2014) 190
RAM: 2 x 2GB DDR2 800MHz 50
Hard Disk: 20GB Seagate ST320014A (circa 2002) IDE ?
Original PSU: 300W OEM SS300-SFD (Seasonic) - 270W @ 12V ?
Extra PSU (for GPU): 175W Delta DPS-175HB A - 100W @ 12V ?
[/pre]
Configuration 1 - Software
[pre]
OS: PCLinuxOS 2014.04 64bit - kernel 3.12.16-pclos3
Driver: fglrx driver and OpenCL libs from Catalyst 13.12
BOINC Version: 7.2.42 64bit
[/pre]
Power Consumption and Productivity
[pre]
Power Usage Task Mix Av. Secs per Task Tasks per Day
Hardware Setup and Running State Watts kWH/day CPU + GPU CPU GPU CPU GPU Credit/Day Calculated Credit/kWH Comments
================================ ===== ======= ========= ================= ============= ============ ===================== ==========
System at idle - no GPU installed: 62 1.488 Idle power - no GPU
System at idle - GPU installed: 81 1.944 Idle power with GPU
System crunching 4xFGRP4 - no GPU: 123 2.952 4 53,713 6.434 0.000 4459 + 0 4459/2.952 = 1,510 Pure CPU cruncher
System crunching 4xFGRP4 idle GPU: 136 3.264 Just to measure watts
System crunching 0xFGRP4 + 4xBRP6: 172 4.128 0 + 4 - 17,699 - 19.527 0 + 85917 85917/4.128 = 20,813 Pure GPU cruncher
System crunching 2xFGRP4 + 4xBRP6: 217 5.208 2 + 4 33,576 18,788 5.147 18.395 3567 + 80937 84504/5.208 = 16,226 CPU + GPU cruncher
[/pre]
Comments
* Another puzzling thing that I hadn't really noticed previously (although it's been there all along) is the big change in CPU crunch time when changing from 4xCPU tasks to 2xCPU tasks (53,713sec -> 33,576sec). I have quite a few Q8400 based hosts. All of the ones with no GPU run all 4 cores and take over 50,000 secs per task when crunching 4xFGRP4. All the others with a HD7850 GPU have 2 free cores and crunch the 2xFGRP4 tasks around 20,000secs faster. It seems to me that once you try to run more than 2 CPU tasks, the CPU is reaching some sort of internal limit and probably lowering both frequency and voltage to cap the power dissipation. This would explain both the above power 'plateau' when going above 2 tasks and the big slowdown in crunch time. The only way I know of to check CPU frequency under Linux is to examine /proc/cpuinfo which shows 3160 MHz irrespective of the number of running tasks. Maybe this info is only sampled at boot time and not updated as the load changes.
* To test whether the above behaviour is temperature related, I replaced the stock CPU heat sink and fan with an after-market cooler (Coolermaster Hyper TX3) and moved the whole machine to a cooler environment (~8 C cooler). I don't have a utility to see core temperatures at full load but I'm sure the machine is running a lot cooler than it was. This has made absolutely no difference to the measured power usage. I can't say for sure about elapsed time per task yet (much more time needed), but there is still a very obvious slowdown once two CPU tasks is exceeded. It would appear that the above listed times will not change appreciably as a result of the improved cooling and so are not temperature related.
* The credit/day figures show that running with all CPU cores unloaded gives a slightly higher credit result at a significantly lower power use.
= = = = = = = = = = = = = = = = = = = = =
Configuration 2 - Hardware
[pre]
CPU: G3258 Pentium Dual Core (3.2GHz) @ 3.8 GHz $ 70
Motherboard: Asrock H81M-DGS 50
GPU: AMD HD7850 2GB (2014) 190
RAM: 2 x 4GB DDR3 1333MHz 60
Hard Disk: 40GB Seagate ST340014AS (circa 2005) SATA ?
Mainboard PSU: 175W Delta DPS-175HB A - 100W @ 12V ?
Extra PSU (for GPU): 175W Delta DPS-175HB A - 100W @ 12V ?
[/pre]
Configuration 2 - Software
[pre]
OS: PCLinuxOS 2014.04 64bit - kernel 3.12.16-pclos3
Driver: fglrx driver and OpenCL libs from Catalyst 13.12
BOINC Version: 7.2.42 64bit
[/pre]
Power Consumption and Productivity
[pre]
Power Usage Task Mix Av. Secs per Task Tasks per Day
Hardware Setup and Running State Watts kWH/day CPU + GPU CPU GPU CPU GPU Credit/Day Calculated Credit/kWH Comments
================================ ===== ======= ========= ================= ============= ============ ===================== ==========
System at idle - no GPU installed: 34 0.816 Idle power - no GPU
System at idle - GPU installed: 60 1.440 Idle power with GPU
System crunching 2xFGRP4 - no GPU: 72 1.728 2 15,333 11.27 0.00 7804 + 0 7804/1.728 = 4,516 Pure CPU cruncher
System crunching 2xFGRP4 idle GPU: 98 2.352 Just to measure watts
System crunching 0xFGRP4 + 4xBRP6: 156 3.744 0 + 4 - 17,944 0.00 19.26 0 + 84744 84744/3.744 = 22,635 Pure GPU cruncher
System crunching 1xFGRP4 + 4xBRP6: 176 4.224 1 + 4 15,452 18,036 5.59 19.16 3875 + 84311 88186/4.224 = 20,877 CPU + GPU cruncher
[/pre]
Comments
* Also (unlike the previous one) freeing up all CPU cores has virtually no impact on GPU crunch time.
* Crunching a CPU task with the 4 GPU tasks makes a reasonable improvement in the credit/day output, with only a quite small drop in credit/kWH for the extra power being used.
* I was quite surprised to see that the old Q8400, when used as a pure GPU cruncher, is quite able to maintain a very competitive credit/kWH figure, despite its age.
Cheers,
Gary.