I think this is due to a higher than usual number of pending tasks that are awaiting validation. One of my hosts used to have approximately 140 pending tasks but more recently has in the range of 230-260 pending tasks and as a result the host RAC has dropped. The number of pending tasks do vary but I would say in my case has gone up around 100 over what I used to see.
Welcome to the Einstein project. Thank you for participating here!
RAC on individual machines always fluctuates for a whole range of reasons. However in this case, there may be something unusual going on.
I found this machine in your list of hosts that the graph belongs to (nice looking machine) and here is its full list of tasks. In the types of science runs listed (top and bottom) you can see the numbers of tasks for each different science run (sub-project). If you select each one in turn and select the completed tasks for each, you can get some idea of how your machine is performing with each particular science run. Here are some examples I've taken from just the Gamma-ray binary pulsar search (FGRPB1) on that machine. The other science runs show similar patterns:-
[pre]
Date Status Elapsed Time CPU Time
---- ------ ------------ --------
25 Feb 2016, 3:28:10 UTC Validated 27,727.71 26,530.47
29 Feb 2016, 16:38:44 UTC Validated 50,085.97 18,669.70
4 Mar 2016, 20:59:29 UTC Validated 27,687.16 27,165.22
[/pre]
Prior to around 28/29 Feb, the CPU time component was only marginally less than the full elapsed time. This is the normal behaviour I'm used to seeing. There are a number of results similar to this. After that date there is a complete change where the CPU component shrinks significantly but the elapsed time virtually doubles. I would normally interpret this as some sort of competition for resources from something else that's compute intensive but the reduction in CPU time is puzzling. That seems to suggest perhaps a fairly dramatic increase in CPU frequency as well as competition for resources. If both times had dropped to around 18K you could attribute that to a big frequency increase. The computer is listed as 2.2GHz. Can it go a lot faster?
The third entry in the table shows the situation has reverted to its previous state in the most recent results. The other science runs seemed to show similar patterns but you have no in-progress tasks for the other two so there are no recently completed results to confirm this.
Perhaps you can re-think about changes that might have caused the time variations, things like CPU frequency, changes to any settings, thermal throttling, HT, etc. Something must be doing this and it's not a variability of one task to the next. There is a fairly constant amount of work for tasks of each type.
With a fairly substantial block of work completed with slow elapsed times, it's not surprising there is a fall in RAC. If normal times are maintained in future, your RAC will progressively recover. It would be interesting to know the reason for the behaviour if you manage to track it down :-).
I will investigate, TTBOMK I did not make any changes around 26th/27th Feb.
I will check and see if any other things are running with high CPU usage.
The H/W is more or less stock builds, but a couple do have good CPU's
They are not customised or tweaked, and CPU settings are not tweaked.
The only thing I do in settings is to set %CPU usage to a value where
they are running at a temp of around high 50's, Low 60's degree C
I'm only using stock cooling apart from 1 where I changed fans to Noctua's
RAC Falling on ALL PC's
)
I think this is due to a higher than usual number of pending tasks that are awaiting validation. One of my hosts used to have approximately 140 pending tasks but more recently has in the range of 230-260 pending tasks and as a result the host RAC has dropped. The number of pending tasks do vary but I would say in my case has gone up around 100 over what I used to see.
Seti team Challenge? Started
)
Seti team Challenge? Started Feb 28.....
As you know, I've come back
)
As you know, I've come back from over there, Zalster.
My Pendings have risen to 1100.
Hi wirezfree, Welcome to
)
Hi wirezfree,
Welcome to the Einstein project. Thank you for participating here!
RAC on individual machines always fluctuates for a whole range of reasons. However in this case, there may be something unusual going on.
I found this machine in your list of hosts that the graph belongs to (nice looking machine) and here is its full list of tasks. In the types of science runs listed (top and bottom) you can see the numbers of tasks for each different science run (sub-project). If you select each one in turn and select the completed tasks for each, you can get some idea of how your machine is performing with each particular science run. Here are some examples I've taken from just the Gamma-ray binary pulsar search (FGRPB1) on that machine. The other science runs show similar patterns:-
[pre]
Date Status Elapsed Time CPU Time
---- ------ ------------ --------
25 Feb 2016, 3:28:10 UTC Validated 27,727.71 26,530.47
29 Feb 2016, 16:38:44 UTC Validated 50,085.97 18,669.70
4 Mar 2016, 20:59:29 UTC Validated 27,687.16 27,165.22
[/pre]
Prior to around 28/29 Feb, the CPU time component was only marginally less than the full elapsed time. This is the normal behaviour I'm used to seeing. There are a number of results similar to this. After that date there is a complete change where the CPU component shrinks significantly but the elapsed time virtually doubles. I would normally interpret this as some sort of competition for resources from something else that's compute intensive but the reduction in CPU time is puzzling. That seems to suggest perhaps a fairly dramatic increase in CPU frequency as well as competition for resources. If both times had dropped to around 18K you could attribute that to a big frequency increase. The computer is listed as 2.2GHz. Can it go a lot faster?
The third entry in the table shows the situation has reverted to its previous state in the most recent results. The other science runs seemed to show similar patterns but you have no in-progress tasks for the other two so there are no recently completed results to confirm this.
Perhaps you can re-think about changes that might have caused the time variations, things like CPU frequency, changes to any settings, thermal throttling, HT, etc. Something must be doing this and it's not a variability of one task to the next. There is a fairly constant amount of work for tasks of each type.
With a fairly substantial block of work completed with slow elapsed times, it's not surprising there is a fall in RAC. If normal times are maintained in future, your RAC will progressively recover. It would be interesting to know the reason for the behaviour if you manage to track it down :-).
Cheers,
Gary.
Hello All, Many thanks for
)
Hello All,
Many thanks for the responses, much appreciated.
@ Gary,
I will investigate, TTBOMK I did not make any changes around 26th/27th Feb.
I will check and see if any other things are running with high CPU usage.
The H/W is more or less stock builds, but a couple do have good CPU's
They are not customised or tweaked, and CPU settings are not tweaked.
The only thing I do in settings is to set %CPU usage to a value where
they are running at a temp of around high 50's, Low 60's degree C
I'm only using stock cooling apart from 1 where I changed fans to Noctua's
Many Thanks...