Add automatic monitoring/instrumentation of underlying job

There have been many requests over time to provide data on CPU, memory, and disk usage.

In an ideal world this would be a rich time series of data, in a less ideal world this would be max with a couple of periodic data points, and in a much more practical world it’d just be max usage. The last one is the definition of done here but if one wants to get ambitious …

The PAPI backend already allows for a custom script to be attached, we should be putting in something on our own for any backend which runs things via unix command lines (ie spark jobs and such likely don’t make sense)

At least one group is using the following in production, this is likely a good start if not completely AOK

echo ==================================
echo =========== MONITORING ===========
echo ==================================
echo --- General Information ---
echo \#CPU: $(nproc)
echo Total Memory: $(free -h | grep Mem | awk '{ print $2 }')
echo Total Disk space: $(df -h | grep cromwell_root | awk '{ print $2}')
echo 
echo --- Runtime Information ---

function runtimeInfo() {
        echo [$(date)]
        echo \* CPU usage: $(top -bn 2 -d 0.01 | grep '^%Cpu' | tail -n 1 | awk '{print $2}')%
        echo \* Memory usage: $(free -m | grep Mem | awk '{ OFMT="%.0f"; print ($3/$2)*100; }')%
        echo \* Disk usage: $(df | grep cromwell_root | awk '{ print $5 }')
}

while true; do runtimeInfo; sleep 300; done```

[Horneth]

ToL: Instead of pretty printing the values like in this script, it shouldn’t be too hard to output them in some kind of tsv format with timestamps that would hopefully be easily parseable to a timeseries.

[Horneth]

@TedBrookings this seems like a really good idea. docker stats would of course only work for tasks running in a docker container but that’s hopefully the majority of them. It would not have been possible to do it in PAPIv1 but PAPIv2 should be flexible enough to allow for something like that.