[S5R3/R4] How to check Performance when Testing new Apps

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6,143
Credit: 129,218,220
RAC: 25,772

RE: Looks quiet good. But I

Message 77744 in response to message 77743

Quote:
Looks quiet good. But I have a problem at step 5. If I intput my sequence number and runtimes, the fields "Average runtime", "Error ~" and "A" and "B" will stay at "0". Don´t know what happens there.


How many work units inputted at step 5?
Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter. Blaise Pascal

Svenie25
Svenie25
Joined: 21 Mar 05
Posts: 139
Credit: 2,436,862
RAC: 0

Tried with only 2 and 8. Both

Tried with only 2 and 8. Both the same result.

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6,143
Credit: 129,218,220
RAC: 25,772

RE: Tried with only 2 and

Message 77746 in response to message 77745

Quote:
Tried with only 2 and 8. Both the same result.


Could you give us the frequency and say two work unit details ( time & sequence number for each )?
Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter. Blaise Pascal

Svenie25
Svenie25
Joined: 21 Mar 05
Posts: 139
Credit: 2,436,862
RAC: 0

Okay, frequency is

Okay, frequency is 754.85

Runtime: 27884 Sequencenumber: 83
29168 88
29754 96
32378 105
33089 109
33893 112
35308 117
34011 121

I wanted to get the estimated runtime for sequence 73.

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6,143
Credit: 129,218,220
RAC: 25,772

RE: Okay, frequency is

Message 77748 in response to message 77747

Quote:

Okay, frequency is 754.85

Runtime: 27884 Sequencenumber: 83
29168 88
29754 96
32378 105
33089 109
33893 112
35308 117
34011 121

I wanted to get the estimated runtime for sequence 73.


Uh oh, let me guess .... you're using MS IE?

Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter. Blaise Pascal

Svenie25
Svenie25
Joined: 21 Mar 05
Posts: 139
Credit: 2,436,862
RAC: 0

RE: Uh oh, let me guess

Message 77749 in response to message 77748

Quote:


Uh oh, let me guess .... you're using MS IE?

Cheers, Mike.

Yes I do...

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6,143
Credit: 129,218,220
RAC: 25,772

RE: Yes I do... OK,

Message 77750 in response to message 77749

Quote:
Yes I do...


OK, changes made - should be good now.
[ MS IE has an issue in mode of declaring globals ]
Please re-download ( same link ) and retry.

Please let me know how it is .... :-)

Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter. Blaise Pascal

Svenie25
Svenie25
Joined: 21 Mar 05
Posts: 139
Credit: 2,436,862
RAC: 0

Thanks a lot. Works pretty

Thanks a lot. Works pretty well now. Good job!

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6,143
Credit: 129,218,220
RAC: 25,772

RE: Thanks a lot. Works

Message 77752 in response to message 77751

Quote:
Thanks a lot. Works pretty well now. Good job!


Terrific! :-)
I've also noted the layout in places looks a bit crappy on IE too ....
Next version:
- I'll do a workaround the fact that MS doesn't agree with the rest of the world on what the definition of the width of a box should be ..... sigh

Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter. Blaise Pascal

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 5,354
Credit: 48,525,448,757
RAC: 57,668,913

Mike, This is the most

Mike,

This is the most marvelous thing since sliced bread!! :).

I've just tried it out for an AMD64 box runnung Linux. I have 8 (or more) data points for the 4.14 app and the same number for the new 4.27 app. If you want the data I could send it but basically it was a good spread of sequence numbers from close to a peak to close to a trough and quite well spaced out. The frequency was 724.60 for all the "old app" data and 724.70 to 724.85 for the "new app" data. I used 724.75 as an average for the "new" data.

The output for the old app using 8 data points is:-

Average runtime = 42869 secs
Peak runtime = 53010 secs
Runtime variance = 0.300
Error = 1.29%

The output for the new app using 8 data points is:-

Average runtime = 41090 secs
Peak runtime = 52952 secs
Runtime variance = 0.351
Error = 1.89%

It was interesting to observe that there was only a marginal improvement in the peak runtime but a much higher (~8%) improvement in the trough runtime. The actual "old/new" times very close to the trough were 37119/34235 respectively. These compare quite favourably with the model values of 37107/34366 right at the trough.

So as people were suggesting, the displacement is certainly not uniform over the cycle.

I have data from at least 20 machines covering a range of CPU types and OSes to work through and your tool will make my life considerably easier than it would have been otherwise. After I play with the tool for a bit I'll send any suggestions that strike me as being useful. One that immediately comes to mind is a "Print" button to record the vital inputs and outputs (all of step 5 in particular) for posterity. As I have no idea of how difficult that might be, it certainly isn't a "request" unless it is trivial to implement :).

Another, probably considerably easier to implement is an output box to list the estimated trough runtime. I know you can get it from A and B but why not save reaching for the calculator? :).

Many thanks for all your efforts - it is highly appreciated.

EDIT:
Let me rephrase the "Print" button bit. How about an "Inputs and Outputs Summary" button which creates a single page of useful information - just the data points used and the model values calculated - that would fit nicely on a single sheet of paper. Perhaps even the ability to save the "old app" set while creating the "new app" set and printing them both together for posterity.

Please feel free to ignore all this if it's not trivial to do. I'm very happy with what exists already.

EDIT2:
In step 5, it would be very useful to have a button which allowed you to save all the inputs - say to a CSV text file for later use. Of course you would then need a "retrieve data" button to allow you to read back a set of inputs from a saved file somewhere .... :).

I think you're going to regret saying that all feedback was welcome :).

A couple of cosmetic items:-
1. Change the "Use Estimates" button to read "Use these values in Step 6"
2. Transfer the words "units at the given frequency" from RHS to LHS so LHS would be two lines which read:-

"Average runtime for all
units at the given frequency"
-- consider changing the word "units" to "tasks"

The space freed on RHS could be used for a "minimum runtine" output suggested earlier.

EDIT3:
At Mike's request, I'm publishing here the 2 datasets (8 points per set) that were used to test out his "ready reckoner V3". There is a small correction to be made in that the "old" app was 4.21 (as shown below) and not 4.14 as originally stated.

Taskname__ Seq# AppV Crunchtime
h1_0724.60 _231 4.21 46,945
h1_0724.60 __57 4.21 37,213
h1_0724.60 __49 4.21 37,119
h1_0724.60 __40 4.21 38,297
h1_0724.60 __32 4.21 40,154
h1_0724.60 __23 4.21 42,965
h1_0724.60 __12 4.21 47,478
h1_0724.60 ___4 4.21 50,806

h1_0724.65 __80 4.27 38,987
h1_0724.70 __18 4.27 43,731
h1_0724.70 ___5 4.27 49,625
h1_0724.75 __73 4.27 36,818
h1_0724.85 _132 4.27 41,156
h1_0724.75 __49 4.27 34,541
h1_0724.75 __25 4.27 40,767
h1_0724.85 _127 4.27 43,555

Since the 8 "new app" data points were recorded, another task has completed. Its sequence number was 122 (ie 5 later than the 127 value listed above. The ready reckoner estimated 45,705 secs for this task. The actual time was 45,449 secs, which is 256 secs lower than the estimate. The error is only 0.56%.

Cheers,
Gary.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.