Fun with Benchmarks

joe areeda
joe areeda
Joined: 13 Dec 10
Posts: 285
Credit: 320,378,898
RAC: 0
Topic 195584

I know there are a lot of issues with benchmarks for computer systems but the stats available here at E@H give us the same programs run on almost 2,000,000 systems so I've been looking into what I can learn.

I've been searching here and at boinc.berkeley.edu for the page that defined them. I know I read it but forgot to bookmark it. If you can, would you please provide the link.

The fun part so far has been parsing the info returned as vendor and model. I've been using the field credit_cpu_per_sec which is a combination of integer and floating point ops per second. In the graphs below I express it as credit per cpu hour just for a more convenient range of numbers.

There is an interesting phenomenon that I can't explain. The main cluster of benchmarks is between 30-50 credits/cpu-hour but many processors have outliers that go up to 250 credits/cpu-hour. Can anyone explain that? It's not that common with only 2900 cpus with a value of 180 credits/hr (0.05 credits/sec) or more.

The following graphs are histograms expressing the number of hosts with that credit value.

This one shows the whole range of values for 4 of the newer processors. Multiple processor models are combined and clock speed is ignored.

This one is the same data with an expanded x-axis looking only at the main peak.

I am not making any claims of insight or significance of this data. I am interested in what you are interested in and how I can better divide and present this.

In a fairly short order (this is a low priority project) I will have an interactive website to overlay graphs and individual computers.

Joe

ps. The raw data for this came from http://einstein.phys.uwm.edu/stats/host_id.gz using a 2010-01-15 download.

joe areeda
joe areeda
Joined: 13 Dec 10
Posts: 285
Credit: 320,378,898
RAC: 0

Fun with Benchmarks

For a finer look at all the processors that I could identify (p_vendor, and p_model are not exactly in the form most conducive to this analysis) you may download this CSV file.

It provides mean, sd, min, max by vendor, processor, family, model, stepping.

It only looks at entries with cpu_credit_per_sec > 0 and < 0.05 (180/hr). That covers 1,969,114 out of 1,973,036 records in the stats file.

Joe

ExtraTerrestrial Apes
ExtraTerrestria...
Joined: 10 Nov 04
Posts: 770
Credit: 446,568,221
RAC: 589,648

Hi Joe, thanks for the

Hi Joe,

thanks for the work and effort!
Is the "credit_cpu_per_sec" field derived from the BOINC benchmark? It does sound so when you say "combination of integer and floating point ops per second" (side note: integer is not relevant for E@H). If that's true then these numbers are.. well, I'd say almost completely useless. The problem is that the real-world E@H performance is quite unrelated to the BOINC benchmark number (which is true for virtually any project).
To get meaningful numbers I think we'd need to measure the E@H performance. Next side note: the high-speed outliers could be peolpe running modified BOINC versions giving out higher benchmark numbers to gain in projects which still base their credits on the BOINC benchmark.
Would also be nice if you could factor HT in for the i7. It's an almost free "get 2 logical cores for the price of 1 physical core", so if you base your numbers on a "per logical core" basis the HT-enalbed chips will look about 30% worse than they actually are (throughput in E@H is typically improved by this amount due to HT).

MrS

Scanning for our furry friends since Jan 2002

joe areeda
joe areeda
Joined: 13 Dec 10
Posts: 285
Credit: 320,378,898
RAC: 0

RE: Hi Joe, thanks for the

Quote:

Hi Joe,

thanks for the work and effort!


You're welcome. It's a little embarrassing but this is the one of the things I do for fun. I used to make a decent living at it.

Quote:
Is the "credit_cpu_per_sec" field derived from the BOINC benchmark? It does sound so when you say "combination of integer and floating point ops per second" (side note: integer is not relevant for E@H). If that's true then these numbers are.. well, I'd say almost completely useless. The problem is that the real-world E@H performance is quite unrelated to the BOINC benchmark number (which is true for virtually any project).


This is an excellent question that I'm still looking into. credit_per_cpu_sec is a field in the host record that I download from the E@H site. I BELIEVE that it is calculated from the 2 benchmarks run but I can't find the WIKI page that led me to that conclusion. (more ^D's in my future). I plan to summarize the number of fp and int ops in the same way.

While I agree benchmarks themselves have low information content but 2 million of them present some interesting comparisons.

Quote:
To get meaningful numbers I think we'd need to measure the E@H performance.


I agree, I would love to get my hands on the list of completed tasks and work units that are on the "my account" page but that is not one of the downloadable datasets. A direct comparison of 2 cpu's running the same calculation would be the most meaningful. IMHO.

Quote:
Next side note: the high-speed outliers could be peolpe running modified BOINC versions giving out higher benchmark numbers to gain in projects which still base their credits on the BOINC benchmark.


I was wondering how to get a factor of 5 or so improvement out of my machine. I also suspected inaccurate cpu clock readings.

Quote:

Would also be nice if you could factor HT in for the i7. It's an almost free "get 2 logical cores for the price of 1 physical core", so if you base your numbers on a "per logical core" basis the HT-enalbed chips will look about 30% worse than they actually are (throughput in E@H is typically improved by this amount due to HT).

MrS


I haven't found anything in the data I have that lets me know if hyperthreading is available or in use. I also haven't seen good numbers on the relative speed of a HT context switch vs non-HT let alone how many context switches are done. I MIGHT be able to deduce something from the processor type and number of cpus reported, but I felt it best to work with data presented before I try to "fix" it.

Just for reference this is the data from one cpu that I'm working with:

    3820808
    93231
    251.070000
    19.336934
    1294945378.630030
    AuthenticAMD
    AMD Phenom(tm) II X4 955 Processor [Family 16 Model 4 Stepping 2]
    Microsoft Windows 7
    x64 Edition, (06.01.7600.00)
  1294671038
  1294715574
  -18000
  4
  3236560612.587980
  7501223923.691910
  250000000.000000
  8580251648.000000
  524288.000000
  21447479296.000000
  419323445248.000000
  213103149056.000000
  26958.990343
  826931.493947
  44536.000000
  0.013788
  d604080cf6bf475dcae07fa7e3eef10a

Suggestions on what else to present or how to better present it are more than welcome.

Joe

ExtraTerrestrial Apes
ExtraTerrestria...
Joined: 10 Nov 04
Posts: 770
Credit: 446,568,221
RAC: 589,648

On the "my computers" page

On the "my computers" page the information is quite similar to the one you've got. So I think it's almost safe to assume they're the same - in which case and refer to the BOINC benchmark. Not totally sure about though.

Regarding which CPUs have HT: currently it's still easy for the i7 as all of them have 2-way HT. It's blurred for i5 and i3 though. And then there are a ton of Xeons using the same chips with or without HT and some larger chips using the same architecture but 8 physical cores (in the 7000 series).

If you'd "only" look at i7 then you could be quite sure that if it reports = 8 then it's got HT on, whereas 4 means off.

And another side note: on of the beautiful things about HT is that it doesn't requires task switches. Tasks are really run in parallel and share some ressources, in order to better exploit them. The regular way using task switches can be done by any single core. Regarding the performance of HT vs. non-HT at Einstein we recently had a nice discussion link.

MrS

Scanning for our furry friends since Jan 2002

joe areeda
joe areeda
Joined: 13 Dec 10
Posts: 285
Credit: 320,378,898
RAC: 0

RE: On the "my computers"

Quote:
On the "my computers" page the information is quite similar to the one you've got. So I think it's almost safe to assume they're the same - in which case and refer to the BOINC benchmark. Not totally sure about though.


My [poor] memory says that cred_per_cpu_sec is a weighted average of the 2 benchmarks but I can't find the reference. It would be nice if it were calculated from awarded credits but I don't think so. Back to scouring the web.

Joe

mikey
mikey
Joined: 22 Jan 05
Posts: 8,758
Credit: 922,438,366
RAC: 5,660,661

RE: I haven't found

Quote:


I haven't found anything in the data I have that lets me know if hyperthreading is available or in use. I also haven't seen good numbers on the relative speed of a HT context switch vs non-HT let alone how many context switches are done. I MIGHT be able to deduce something from the processor type and number of cpus reported, but I felt it best to work with data presented before I try to "fix" it.

Just for reference this is the data from one cpu that I'm working with:

    3820808
    93231
    251.070000
    19.336934
    1294945378.630030
    AuthenticAMD
    AMD Phenom(tm) II X4 955 Processor [Family 16 Model 4 Stepping 2]
    Microsoft Windows 7
    x64 Edition, (06.01.7600.00)
  1294671038
  1294715574
  -18000
  4
  3236560612.587980
  7501223923.691910
  250000000.000000
  8580251648.000000
  524288.000000
  21447479296.000000
  419323445248.000000
  213103149056.000000
  26958.990343
  826931.493947
  44536.000000
  0.013788
  d604080cf6bf475dcae07fa7e3eef10a

Suggestions on what else to present or how to better present it are more than welcome.

Joe

You might want to get the info from an Intel chip instead of an AMD one for your HT stuff. I am NOT sure it would be there but I am pretty sure it will not be for an AMD chip.

joe areeda
joe areeda
Joined: 13 Dec 10
Posts: 285
Credit: 320,378,898
RAC: 0

RE: You might want to get

Quote:
You might want to get the info from an Intel chip instead of an AMD one for your HT stuff. I am NOT sure it would be there but I am pretty sure it will not be for an AMD chip.


Good point but the fields are the same regardless of processor.

The only way I can think of is as MrS suggested is to know the number of cores and compare it with the number of cpus reported.

Joe

tullio
tullio
Joined: 22 Jan 05
Posts: 2,087
Credit: 54,801,654
RAC: 33,131

My Opteron 1210 has two cores

My Opteron 1210 has two cores and only AQUA@home uses both since it is a multithreading app.
Tullio

mikey
mikey
Joined: 22 Jan 05
Posts: 8,758
Credit: 922,438,366
RAC: 5,660,661

RE: My Opteron 1210 has two

Quote:
My Opteron 1210 has two cores and only AQUA@home uses both since it is a multithreading app.
Tullio

You mean it uses both cores when working on the same unit, correct? Because I have several pc's and all the cores are being used in all of them, just each core works on its own unit.

tullio
tullio
Joined: 22 Jan 05
Posts: 2,087
Credit: 54,801,654
RAC: 33,131

RE: RE: My Opteron 1210

Quote:
Quote:
My Opteron 1210 has two cores and only AQUA@home uses both since it is a multithreading app.
Tullio

You mean it uses both cores when working on the same unit, correct? Because I have several pc's and all the cores are being used in all of them, just each core works on its own unit.


Correct.
Tullio

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.