[S5R3/R4] How to check Performance when Testing new Apps

Mike,

This is the most marvelous thing since sliced bread!! :).

I've just tried it out for an AMD64 box runnung Linux. I have 8 (or more) data points for the 4.14 app and the same number for the new 4.27 app. If you want the data I could send it but basically it was a good spread of sequence numbers from close to a peak to close to a trough and quite well spaced out. The frequency was 724.60 for all the "old app" data and 724.70 to 724.85 for the "new app" data. I used 724.75 as an average for the "new" data.

The output for the old app using 8 data points is:-

Average runtime = 42869 secs
Peak runtime = 53010 secs
Runtime variance = 0.300
Error = 1.29%

The output for the new app using 8 data points is:-

Average runtime = 41090 secs
Peak runtime = 52952 secs
Runtime variance = 0.351
Error = 1.89%

It was interesting to observe that there was only a marginal improvement in the peak runtime but a much higher (~8%) improvement in the trough runtime. The actual "old/new" times very close to the trough were 37119/34235 respectively. These compare quite favourably with the model values of 37107/34366 right at the trough.

So as people were suggesting, the displacement is certainly not uniform over the cycle.

I have data from at least 20 machines covering a range of CPU types and OSes to work through and your tool will make my life considerably easier than it would have been otherwise. After I play with the tool for a bit I'll send any suggestions that strike me as being useful. One that immediately comes to mind is a "Print" button to record the vital inputs and outputs (all of step 5 in particular) for posterity. As I have no idea of how difficult that might be, it certainly isn't a "request" unless it is trivial to implement :).

Another, probably considerably easier to implement is an output box to list the estimated trough runtime. I know you can get it from A and B but why not save reaching for the calculator? :).

Many thanks for all your efforts - it is highly appreciated.

EDIT:
Let me rephrase the "Print" button bit. How about an "Inputs and Outputs Summary" button which creates a single page of useful information - just the data points used and the model values calculated - that would fit nicely on a single sheet of paper. Perhaps even the ability to save the "old app" set while creating the "new app" set and printing them both together for posterity.

Please feel free to ignore all this if it's not trivial to do. I'm very happy with what exists already.

EDIT2:
In step 5, it would be very useful to have a button which allowed you to save all the inputs - say to a CSV text file for later use. Of course you would then need a "retrieve data" button to allow you to read back a set of inputs from a saved file somewhere .... :).

I think you're going to regret saying that all feedback was welcome :).

A couple of cosmetic items:-
1. Change the "Use Estimates" button to read "Use these values in Step 6"
2. Transfer the words "units at the given frequency" from RHS to LHS so LHS would be two lines which read:-

"Average runtime for all
units at the given frequency"
-- consider changing the word "units" to "tasks"

The space freed on RHS could be used for a "minimum runtine" output suggested earlier.

EDIT3:
At Mike's request, I'm publishing here the 2 datasets (8 points per set) that were used to test out his "ready reckoner V3". There is a small correction to be made in that the "old" app was 4.21 (as shown below) and not 4.14 as originally stated.

Taskname__ Seq# AppV Crunchtime
h1_0724.60 _231 4.21 46,945
h1_0724.60 __57 4.21 37,213
h1_0724.60 __49 4.21 37,119
h1_0724.60 __40 4.21 38,297
h1_0724.60 __32 4.21 40,154
h1_0724.60 __23 4.21 42,965
h1_0724.60 __12 4.21 47,478
h1_0724.60 ___4 4.21 50,806

h1_0724.65 __80 4.27 38,987
h1_0724.70 __18 4.27 43,731
h1_0724.70 ___5 4.27 49,625
h1_0724.75 __73 4.27 36,818
h1_0724.85 _132 4.27 41,156
h1_0724.75 __49 4.27 34,541
h1_0724.75 __25 4.27 40,767
h1_0724.85 _127 4.27 43,555

Since the 8 "new app" data points were recorded, another task has completed. Its sequence number was 122 (ie 5 later than the 127 value listed above. The ready reckoner estimated 45,705 secs for this task. The actual time was 45,449 secs, which is 256 secs lower than the estimate. The error is only 0.56%.