Automating changes to task multiples

I would not have guessed you would be able to do that, but it seems a really good goal.  Good luck.

I've grabbed your script and used it on my test machine - currently 418 tasks on board, running at x3.  All tasks were nicely listed in the output.  All 3 states, 6 uploaded and ready to report, 3 scheduled (running) and the balance listed as uninitialized, showed in the output.   Surely, the 'task-on-deck' is just the topmost uninitialized task, immediately following the 'scheduled' tasks?  I've looked down my list and that's exactly the order in which the uninitialized tasks will be started.  Perhaps I'm misunderstanding what you mean by that term :-).

I imagine the easiest approach might be to construct arrays in the main script for however many of task name, DF, task_state and sched_state that you wanted to keep track of.  You could manipulate the data in those arrays as current tasks get crunched and new ones are added through work fetch events.  That implies a "run forever" script that keeps everything up to date.

When first launched, the script would scan the complete list of tasks and store (ignoring the completed and running ones, as mentioned above) just the 'uninitialized' ones and identify the topmost DF as belonging to the 'next-to-start' task.  To know when to move on to the next DF and completely remove the topmost entry whenever there is a task completion, your script could monitor the PIDs (Process IDs) of all currently running O2MDF app instances and detect when any one of them disappears.  At that point, the topmost task in the arrays has started running, replacing the completed task, so this top entry should be removed, immediately exposing the next available DF for further decision making purposes.

Detecting when a PID disappears is relatively straight forward.  Below is a little script that shows a way to do it and I've added comments so that anyone following along might better understand what's going on.  I've tested it on my test host which is continuing to run at x3 and it works as expected.  Of course, there's a lot more work to detect work fetch events and to add the new task data to the bottom of the arrays.  The script below does none of that but creating and updating arrays shouldn't be too complicated.

#!/bin/bash
# pid_test - find current O2MDF PIDs and detect when any one of them finishes.
#
all_pids=`ps -A | grep O2MDF | sed 's/^\ *//' | cut -d' ' -f1`
echo
echo -n "Here are the current PIDs -> " ; echo $all_pids
echo
echo "When a task is at 99% hit <enter> to start detecting its completion."
echo "If you start too early you will get an awful lot of dots ........."
echo -n "Ready? (y or n) [y] : "
read ans            # If no answer is supplied, set the expected default
ans=${ans:=y}
if [ $ans != y ]; then
    exit 0
fi
echo
p_done=0                            # This will hold the PID that actually exits.
while true ; do                     # Repeat all this for as long as it takes for
    for p in `echo $all_pids` ; do  # any of the concurrent tasks to finish.
        msg=`kill -0 $p 2>&1`       # The output in $msg is redirected error output.
        if [ "X$msg" != "X" ]; then # This test only succeeds when a PID being
            p_done=$p               # tested has gone and there is some output.
            break                   # At that point we need to break out of
        fi                          # this inner loop looking at all PIDs.
    done
    if [ $p_done -eq 0 ]; then      # If $p_done is not set, the previous loop
        echo -n .                   # exited normally with out a break, so
        sleep 1                     # sleep for a bit and then keep going
    else                            # otherwise, a task has finished
        echo " - the $p_done PID has now disappeared."
        break                       # so break out of the outer (run forever) loop.
    fi
done
echo "This test has concluded successfully with a finished task detected."

I hope some of this might be of some use to you.

One complication would arise when a running task gets suspended.  I imagine its PID disappears and it gets a different one when it resumes.  That task would be liable to restart when a different running task finished so the 'next available DF' would need to be reconsidered.

I can confirm that there is no code (yet) that frees GPU resources if the app is terminated by a signal, e.g. when it is "suspended" by the client (and preferences are set to "remove the app from memory when suspended"). Some drivers will free these resources automatically, but some might not. We'll definitely fix that in the next version of the App.