Fun with Benchmarks
16 Jan 2011 16:38:44 UTC
Topic 195584
(moderation:
I know there are a lot of issues with benchmarks for computer systems but the stats available here at E@H give us the same programs run on almost 2,000,000 systems so I've been looking into what I can learn.
I've been searching here and at boinc.berkeley.edu for the page that defined them. I know I read it but forgot to bookmark it. If you can, would you please provide the link.
The fun part so far has been parsing the info returned as vendor and model. I've been using the field credit_cpu_per_sec which is a combination of integer and floating point ops per second. In the graphs below I express it as credit per cpu hour just for a more convenient range of numbers.
There is an interesting phenomenon that I can't explain. The main cluster of benchmarks is between 30-50 credits/cpu-hour but many processors have outliers that go up to 250 credits/cpu-hour. Can anyone explain that? It's not that common with only 2900 cpus with a value of 180 credits/hr (0.05 credits/sec) or more.
The following graphs are histograms expressing the number of hosts with that credit value.
This one shows the whole range of values for 4 of the newer processors. Multiple processor models are combined and clock speed is ignored.
This one is the same data with an expanded x-axis looking only at the main peak.
I am not making any claims of insight or significance of this data. I am interested in what you are interested in and how I can better divide and present this.
In a fairly short order (this is a low priority project) I will have an interactive website to overlay graphs and individual computers.
Joe
ps. The raw data for this came from http://einstein.phys.uwm.edu/stats/host_id.gz using a 2010-01-15 download.
Fun with Benchmarks
)
For a finer look at all the processors that I could identify (p_vendor, and p_model are not exactly in the form most conducive to this analysis) you may download this CSV file.
It provides mean, sd, min, max by vendor, processor, family, model, stepping.
It only looks at entries with cpu_credit_per_sec > 0 and < 0.05 (180/hr). That covers 1,969,114 out of 1,973,036 records in the stats file.
Joe
Hi Joe, thanks for the
)
Hi Joe,
thanks for the work and effort!
Is the "credit_cpu_per_sec" field derived from the BOINC benchmark? It does sound so when you say "combination of integer and floating point ops per second" (side note: integer is not relevant for E@H). If that's true then these numbers are.. well, I'd say almost completely useless. The problem is that the real-world E@H performance is quite unrelated to the BOINC benchmark number (which is true for virtually any project).
To get meaningful numbers I think we'd need to measure the E@H performance. Next side note: the high-speed outliers could be peolpe running modified BOINC versions giving out higher benchmark numbers to gain in projects which still base their credits on the BOINC benchmark.
Would also be nice if you could factor HT in for the i7. It's an almost free "get 2 logical cores for the price of 1 physical core", so if you base your numbers on a "per logical core" basis the HT-enalbed chips will look about 30% worse than they actually are (throughput in E@H is typically improved by this amount due to HT).
MrS
Scanning for our furry friends since Jan 2002
RE: Hi Joe, thanks for the
)
You're welcome. It's a little embarrassing but this is the one of the things I do for fun. I used to make a decent living at it.
This is an excellent question that I'm still looking into. credit_per_cpu_sec is a field in the host record that I download from the E@H site. I BELIEVE that it is calculated from the 2 benchmarks run but I can't find the WIKI page that led me to that conclusion. (more ^D's in my future). I plan to summarize the number of fp and int ops in the same way.
While I agree benchmarks themselves have low information content but 2 million of them present some interesting comparisons.
I agree, I would love to get my hands on the list of completed tasks and work units that are on the "my account" page but that is not one of the downloadable datasets. A direct comparison of 2 cpu's running the same calculation would be the most meaningful. IMHO.
I was wondering how to get a factor of 5 or so improvement out of my machine. I also suspected inaccurate cpu clock readings.
I haven't found anything in the data I have that lets me know if hyperthreading is available or in use. I also haven't seen good numbers on the relative speed of a HT context switch vs non-HT let alone how many context switches are done. I MIGHT be able to deduce something from the processor type and number of cpus reported, but I felt it best to work with data presented before I try to "fix" it.
Just for reference this is the data from one cpu that I'm working with:
3820808 93231 251.070000 19.336934 1294945378.630030 AuthenticAMD AMD Phenom(tm) II X4 955 Processor [Family 16 Model 4 Stepping 2] Microsoft Windows 7 x64 Edition, (06.01.7600.00) 1294671038 1294715574 -18000 4 3236560612.587980 7501223923.691910 250000000.000000 8580251648.000000 524288.000000 21447479296.000000 419323445248.000000 213103149056.000000 26958.990343 826931.493947 44536.000000 0.013788 d604080cf6bf475dcae07fa7e3eef10aSuggestions on what else to present or how to better present it are more than welcome.
Joe
On the "my computers" page
)
On the "my computers" page the information is quite similar to the one you've got. So I think it's almost safe to assume they're the same - in which case and refer to the BOINC benchmark. Not totally sure about though.
Regarding which CPUs have HT: currently it's still easy for the i7 as all of them have 2-way HT. It's blurred for i5 and i3 though. And then there are a ton of Xeons using the same chips with or without HT and some larger chips using the same architecture but 8 physical cores (in the 7000 series).
If you'd "only" look at i7 then you could be quite sure that if it reports = 8 then it's got HT on, whereas 4 means off.
And another side note: on of the beautiful things about HT is that it doesn't requires task switches. Tasks are really run in parallel and share some ressources, in order to better exploit them. The regular way using task switches can be done by any single core. Regarding the performance of HT vs. non-HT at Einstein we recently had a nice discussion link.
MrS
Scanning for our furry friends since Jan 2002
RE: On the "my computers"
)
My [poor] memory says that cred_per_cpu_sec is a weighted average of the 2 benchmarks but I can't find the reference. It would be nice if it were calculated from awarded credits but I don't think so. Back to scouring the web.
Joe
RE: I haven't found
)
You might want to get the info from an Intel chip instead of an AMD one for your HT stuff. I am NOT sure it would be there but I am pretty sure it will not be for an AMD chip.
RE: You might want to get
)
Good point but the fields are the same regardless of processor.
The only way I can think of is as MrS suggested is to know the number of cores and compare it with the number of cpus reported.
Joe
My Opteron 1210 has two cores
)
My Opteron 1210 has two cores and only AQUA@home uses both since it is a multithreading app.
Tullio
RE: My Opteron 1210 has two
)
You mean it uses both cores when working on the same unit, correct? Because I have several pc's and all the cores are being used in all of them, just each core works on its own unit.
RE: RE: My Opteron 1210
)
Correct.
Tullio