[S5R3/R4] How to check Performance when Testing new Apps

If there are any people out there using either Bikeman's awk script or my shell script which relies on the awk script, there is a small change you will need to make to the awk script as a result of the new >800 frequency tasks. This line from the script

match($0,"h1_[0-9]+\\\\.[0-9]+_S5R2__[0-9]+_S5R3[a-z]_[0-9]+"); needs to change to

match($0,"h1_[0-9]+\\\\.[0-9]+_S5R[23]__[0-9]+_S5R3[a-z]_[0-9]+"); This is needed because old data was referred to as S5R2 data and the new >800 data is referred to as S5R3 data.

Having got that out of the way, I'd like to report on some enhancements that I've made to my data gathering shell script. I wrote it this way because I don't really have any experience with other scripting languages and the unix shell seems to be able to handle what I wanted to do without too much of a re-learning curve. Obviously you need a unix (linux) machine to run this script.

My goals were to automate the process of data gathering for an unlimited number of machines and to create result files that could be dropped straight into Mike Hewson's RR. The data gathering process should be smart enough to handle things like frequency changes, app version changes, recognition of crunching of a single task by multiple app versions, etc, without requiring user intervention.

I've been testing my latest version of the script for a few days now and it seems to be working OK. It's gathering data for around 100 boxes and a random check on a few data files seems to show that everything is in order. For each successful task that a host completes, four CSV data values are recorded per line in the results file - Frequency, Sequence#, Runtime, App_Version.

Here are some of the features of the script

• * Separate results files for each host which are appended to at a user settable interval with no limit on the number of hosts.
  * Automatic recording of both HostID and Hostname in a file called hostids every time you give a new host to the script. Future runs can simply reuse some or all stored hosts, or can add new ones.
  * Manual operation on just a couple of selected hosts if needed.
  * Each time the script runs, any data that has previously been gathered will not be duplicated in the ongoing results file.
  * The correct app version used for crunching will always be recorded even if the task is "branded" differently in your Boinc Manager list of tasks. If more than one app version was used for a single task the output will show 'mixed'.
  * All website task data for each host will be examined irrespective of how many pages this might involve (ie one or many).
  * The script will attempt to minimise the number of website pages consulted in updating the results file for each host.
  * A rudimentary progress indicator - very useful when auto collecting for 100 hosts :).
  

Here is the current version (unfortunately indenting is lost) of the script. It is extensively commented if anyone is actually trying to make sense of it, so that partly accounts for the size. Undoubtedly, better ways to do things may emerge so there's a good chance it might shrink in future. I'll find a place to host this so you will get the full indentation if you wait a bit. I'll also post a separate message with proper instructions for use. The only thing you will need is access to a unix/linux machine that understands bash, awk, sed, grep, find, sort, paste, and a few other standard unix utilities.

#!/bin/sh
#
# grabdata.sh - Version 3.0
#
# Script to retrieve successful task data from the Einstein@Home online database for
# one or many hostIDs (either entered singly or read from a file) and to parse it to
# produce a tabulated set of information (frequency, seq#, cpu_time, app_version) as
# a CSV list for each host of interest.
#
# Hosts entered manually may be added automatically to the HostIDs file for future
# reuse.  Extracted results data are added to individual host results files to provide
# an ongoing, updating log of host results statistics.
#
# This script relies on an awk script written by Bikeman.
#
echo
echo "Program to grab stats for successful EAH tasks and write them to a results file."
echo "Data is collected from the EAH website for hosts of interest, using the HostID."
echo "The IDs can be entered manually or read from an existing file named hostids."
echo "HostIDs manually entered will be added to the hostids file if not already there."
echo "The lines in this file contain the HostID and a HostName separated by a space."
hidfile=hostids
mindays=+2
echo
echo "Integer days of enforced wait from previous update before a new one is allowed."
echo -n "(Signed ints like +0 +1 +2 - see -mtime flag on 'find' manpage) (default=+2) : "
read ans
echo
if [ "X$ans" != "X" ]
then
	mindays=$ans
fi
#
# Find all results files older than $mindays & store in $hids - ignore newer files
#
hids=`find results.* -mtime $mindays | sed 's/^results\\.//'`
#
# If there is no file of HostIDs (ie 1st time run) create a new one
#
if [ ! -f  $hidfile ]
then
	touch $hidfile
fi
#
# See if we are doing an auto update of all out-of-date host or perhaps a manual run
#
echo
echo -n "Using hosts from hostids file? (y or n - use  if entering manually) : "
read ans
echo
if [ "$ans" == "y" ]
then
	if [ "`wc -l $hidfile`" == "0" ]
#
# Can't autorun if there are no hosts filed.  Suggest manual host entry
#
	then
		echo "HostIDs file $hidfile does not contain hosts - use manual entry ..."
		echo
		exit
	else
		echo "Any outdated results for hosts in $hidfile file will be retrieved."
		echo
	fi
else
#
# Manual run.  Collect HostIDs.  Check if HostID is on the list of out-of-date results
# Only accept it if it is.  If it's not, check if we have a results file for it.
# Offer to update HostIDs file if we don't already have this host filed.
#
	mhids=""
	allhids=`cat $hidfile | sed 's/\\ \\ .*//'`
	while true
	do
		echo -n "Next hostID to use for results (  only when finished ) : "
		read hid
		if [ "X$hid" != "X" ]
		then
			inhids=n
			for i in $hids
			do
				if [ "$hid" == "$i" ]
				then
					inhids=y
					mhids="$mhids$hid "
					break
				fi
			done
			if [ "$inhids" == "n" ]
			then
				if [ -f results.$hid ]
				then
					inhids=r
					echo "HostID $hid was recently updated -- ignoring this HostID ..."
				else
					mhids="$mhids$hid "
				fi
				inallhids=n
				for j in $allhids
				do
					if [ "$j" == "$hid" ]
					then
					  inallhids=y
					  break
					fi
				done
				if [ "$inallhids" == "n" ]
				then
					echo -n "Hostname for updating HostIDs file $hidfile ( = Dont update) : "
					read host
					if [ "X$host" != "X" ]
					then
						echo "$hid $host" >> $hidfile
					fi
				else
					echo "HostID $hid already in $hidfile - no need to update that file ..."
				fi
			else
				echo "HostID $hid already in $hidfile - no need to update that file ..."
			fi
		else
			break
		fi
	done
	echo "Manually entered HostIDs requiring an update are :-"
	echo "$mhids"
	echo
	echo -n "If these are OK hit  to continue or q to quit : "
	read tmp
	if [ "X$tmp" != "X" ]
	then
		exit
	else
		hids=$mhids
	fi
fi
#
# Variable $hids now contains just those hosts (either auto-determined or manually
# entered) that require updating - if there are any.
#
if [ "X$hids" == "X" ]
then
	echo "There are no out-of-date results files that need updating -- exiting ... "
	echo
	exit
fi
#
# Grab the data from the website for each valid host
#
for hid in $hids
do
#
# Get the first page of (up to 20) results   Test if there is a "Next" page
#
	curl -s "http://einstein.phys.uwm.edu/results.php?hostid=$hid" > results.curl
	while true
	do
		tmp=`grep Next results.curl | tail -1 | sed -e 's/^.*offset.//' -e 's/.Next.*//'`
		if [ "X$tmp" != "X" ]
		then
#
# There is a next page - keep grabbing until no more
#
			while true
			do
				curl -s "http://einstein.phys.uwm.edu/results.php?hostid=$hid&offset=$tmp" > results.ext
				tmp1=`grep Next results.ext | tail -1 | sed -e 's/^.*offset.//' -e 's/.Next.*//'`
				cat results.ext >> results.curl
				tmp=$tmp1
				if [ "X$tmp1" == "X" ]
				then
					break 2
				fi
			done
		else
			break
		fi
	done
#
# All grabbed pages for a host have been concatenated.  Pass through Bikeman's awk script
# Massage to create a CSV list of Freq,Seq#,Runtime and a list of task IDs.  Use the TIDs
# to grab the page with stderr.out so that the app version can be obtained.
#
	awk -f parser.awk results.curl > results.raw
	cut -c-9 results.raw > results.tid
	tids=`cat results.tid`
	cut -c10- results.raw > results.cut
	sed -e s/\\ /,/g -e /,[0-9][0-9],/s/,/,0/ -e /,[0-9],/s/,/,00/ -e s/...$// results.cut > results.tmp
	paste -d, results.tid results.tmp | sed s/\\ // > results.csv
#
# Progress indicator.  Each dot means another set of raw host results has been parsed
#
	echo -n .
	for tid in $tids
	do
#
# For each TID obtained from the website, check to see if those details are already recorded
# in the results file,  Regard a match of seq# and runtime as sufficient to prove a match.
# Do not grab the page with stderr.out if we already have the data recorded
#
		runtime=`grep $tid results.csv | sed 's/^.*,//'`
		seqno=`grep $tid results.csv | cut -c18- | sed s/,.*//`
		tmp=""
		if [ -f results.$hid ]
		then
			tmp=`grep $runtime results.$hid | grep $seqno`
		fi
		if [ "X$tmp" == "X" ]
		then
#
# No match for seq# and runtime for this TID so we need to grab the page.
#
			freq=`grep $tid results.csv | sed -e 's/,[0-9][0-9][0-9],.*//' -e 's/^.*,//'`
			ver=`curl -s "http://einstein.phys.uwm.edu/result.php?resultid=$tid" | grep einstein_S5R3 | sed -e 's/^.*S5R3_//' -e 's/_[iwp][6io].*//'`
			numv=`echo $ver | wc -w`
#
# Check if more than one app version was used to crunch the data - if so record as 'mixed'
#
			if [ $numv != 1 ]
			then
				flag=1
				ver1=`echo $ver | sed 's/\\ .*//'`
				for i in $ver
				do
					if [ $i != $ver1 ]
					then
						flag=2
						break
					fi
				done
				if [ $flag == 1 ]
				then
					ver=$ver1
				else
					ver=mixed
				fi
			fi
#
# Assemble a line of new data and store it in a temporary results file
#
			echo $freq,$seqno,$runtime,$ver >> results.new
		fi
	done
#
# Progress indicator.  Each plus means another host's new results have been assembled.
#
	echo -n +
	if [ ! -f results.new ]
	then
#
# There are actually no new results at this time so create an empty file as a placeholder
#
		touch results.new
	fi
	if [ -f results.$hid ]
	then
#
# If there are existing results for this host add in the new ones.  For no existing results
# any new ones will become the future existing ones.  Clean up any temp files
#
		mv results.$hid results.sav
		cat results.sav results.new | sort > results.$hid
		rm -f results.sav results.new
	else
		mv results.new results.$hid
	fi
	rm -f results.raw results.c* results.t* results.ext
done
echo

This script has been tested a few times but I wouldn't call it "extensive" by any means. During testing, I would often think of new features that would be nice so some of the more recent additions haven't been tested much at all. I'll be very surprised if there aren't at least a few logic bugs and probably more logic deficiencies. The most recent test involving about 100 hosts took less than 10 minutes to update the results files of all hosts. Most hosts had an average of about 10-15 results listed on the website, with just a few going past 20.

I'll be interested to see if anyone is kind (?foolish?) enough to give it a trial :). There is one known deficiency which I've just remembered and will fix before I host it somewhere. The script only understands results crunched on Windows or Linux. I've got to look at what other platforms say about the app version and then make a few minor adjustments to suit.

EDIT:
A very small change has been made to the above script to allow it to handle results for MacOS PPC. Windows, Linux and MacOS X Intel were already correctly handled.

To actually trial the script, all you need to do is create a directory and in it place this script and Bikeman's awk script with the latest mod mentioned at the start of this post. Make the script executable and from a console window (I use an xterm) 'cd' to the directory and just run the script. It will ask you for whatever it needs.

I don't anticipate having much time to add new features but I'm certainly interested in bug reports and particularly in suggestions of smarter ways to do things. I'm quite limited in my knowledge of tools outside of what was available in unix 15-25 years ago :).

Thanks for all the details as I'm sure others will now appreciate more fully, just how your tool works. For my purposes "Principal Value = |sin(phase)|" would have done nicely :). It's obvious now but I couldn't see it at the time I asked.

Also, whilst it's nice to see all the cycles in the data (which gives a wide graph), it might also be useful to see just one period. Would it be possible to have a tick box or similar that would "compress" the multiple periods of a normal plot into just one period? Using the "Next"/"Prev" buttons for seq# would then move to the appropriate data point and show the true seq# in the numeric display but the plotted point would be at its converted position in the single period. This would effectively allow the relationship of the actual data points to the model line to be seen more accurately, I think.

Yes, indeed. I have around 100 data files accumulating so as I start looking at them and find anything interesting, I'll just pass you the CSV data. Too easy!!

Thanks very much!

I haven't plotted the points but I can see exactly what you are referring to :).

Something happened probably during or around seq# 40 which gave a step improvement of close to 10% in crunching performance. Could it have been an app change to a faster app or could it have been hardware improvements like upping the overclock a bit :). Maybe Peter might have some thoughts on this.