RAC ? - is it of any use? Why bother to look at it or record it?

Thanks very much for posting this. It's amazing how blind one can be, even when the obvious is staring you in the face. Your code snippets have prompted me to see a very simple but quite beneficial change to my own control procedures. For people with no experience of unix (Linux) or shell scripting, let me just explain in plain English what the above code does.

Line 1 - set a variable (thetime) to contain the time right now in unix format (number of seconds since the 'epoch' - 1 Jan 1970 00:00:00).

Line 2 - decrement 'thetime' by exactly one day.

Line 3 - a series (pipeline) of unix utilities, each one sending its standard output to the standard input of the next one in the pipeline. The vertical bar (|) symbol is what 'connects' each utility to the next in the pipeline. Let's explain each stage.

Line 3a - tail is a utility that selects a number of lines at the end (tail) of a file - in this case 800 lines from the Einstein job log file. You just want a big enough number to be sure you are getting all results returned to the project in the last 24 hours. It doesn't matter if the number is way too big, as it is in this case :-).

Line 3b - awk has many text processing and data extraction uses. Here, it reads the lines sent to it by 'tail', breaks each line into a series of items ('tokens' - $1, $2, $3, etc) and selects only those lines whose very first item ($1) is greater than (>) the value stored in the variable 'thetime'. This guarantees that only those results that were returned to the project in the last 24 hours make it through to the next stage.

Line 3c - grep is a pattern matching utility. It looks at what awk is sending and only allows through those lines which contain 'PM' somewhere in the line. 'PM' is the identifier of the Parkes PMPS XT tasks. So at this point we are left with just those lines that represent BRP6 tasks sent back in the last 24 hours. If you've never done it before, go and find your own Einstein job log file in the BOINC data directory and browse it with a simple text editor (don't make or save any changes). Look at the two letter 'names' each of which describes the following 'field' - eg a 'name and field' of 'et 14688.924797' would be representing an elapsed time of 14689 seconds. The name 'ct' would represent CPU time, and so on.

Line 3d - cut is a utility for 'cutting' out and passing through certain parts of each line fed into it. the -d flag tells cut what delimiter is being used to designate individual fields, in this case a 'space' character - if you look closely there is a space following the backslash in "-d\ " used above. It could have been written as " -d' ' " to make the space more obvious. The "-f11" flag tells cut to just cut out of each line the 11th field which is the elapsed time field.

Line 3e - sh is the unix shell itself - in Linux it's usually the 'bash' shell. So the output of cut (just the list of elapsed times of relevant tasks) is given to the shell to feed into a further script called 'colstatsd2.sh' which exists in the current directory. Obviously this further script is a set of standard routines for calculating the statistics of a set of numbers fed to it.

So that simple 3 line script starts with the last 1000 entries in the job log file and ends producing a list of selected elapsed times for further analysis.

The bit that really struck home to me was that I'm already visiting each host in my fleet every 8 hours and doing a number of monitoring and control activities on each host. A by-product for me is the ability to flag 'suspicious' changes in RAC, which is working very well but only kicks in after there has been a suspicious decline. It does give a number of false positives where there is a decline but the host is not having a problem. It's a trailing indicator and doesn't show up until 15-20 hours (at best) after the 'event'. By counting the number of returned tasks, the problem that really interests me (host appears normal but has stopped returning GPU tasks) should be 'seen' within the 8 hour cycle time, in most cases, and certainly well within 16 hours at worst. It's a pretty trivial addition to make to the control script so once again thanks very much to AgentB for sharing the details of what he does and prompting me to think about it.