All nodes in pool in state `starttaskfailed`: "Docker root dir $rootdir not within $USER_MOUNTPOINT"

Problem Description

I have been using Azure Batch Shipyard with VMs of type STANDARD_NC6 succesfully for a while. Usually, I create a pool, submit some jobs (with several tasks) and kill the pool again, all over the course of at most a couple of days.

As of today, when creating the pool and submitting a job, all nodes enter the “starttaskfailed” state. I have deleted and recreated the pool and job several times. Using the Azure Batch Explorer I have checked the node startup logs and find the following text at the bottom of stdout.txt:

Client: Docker Engine - Community
 Version:           19.03.0
 API version:       1.39 (downgraded from 1.40)
 Go version:        go1.12.5
 Git commit:        aeac949
 Built:             Wed Jul 17 18:16:07 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 03:42:13 2019
  OS/Arch:          linux/amd64
  Experimental:     false
2019-07-23T11:11:50,787210380+00:00 - ERROR - Docker root dir Dir: not within /mnt

This seems to originate from shipyard_nodeprep.sh line 730-737:

    local rootdir
    rootdir=$(docker info | grep "Docker Root Dir" | cut -d' ' -f 4)
    if echo "$rootdir" | grep "$USER_MOUNTPOINT" > /dev/null; then
        log DEBUG "Docker root dir: $rootdir"
    else
        log ERROR "Docker root dir $rootdir not within $USER_MOUNTPOINT"
        exit 1
    fi

It looks like the cut command does not properly extract the “Docker Root Dir” from the output of docker info (note that $rootdir = "Dir:" !).

Batch Shipyard Version

3.7.0

Steps to Reproduce

Create pool, then create job.

Expected Results

The pool gets created and the job + tasks start properly.

Actual Results

All nodes in the pool get stuck in “starttaskfailed”

Redacted Configuration

pool:

pool_specification:
  id: my-pool
  vm_configuration:
    platform_image:
      offer: UbuntuServer
      publisher: Canonical
      sku: 16.04-LTS
  vm_count:
    dedicated: 0
    low_priority: 10
  vm_size: STANDARD_NC6

Additional Logs

Header part from stdout.txt:

Configuration:
--------------
Custom image: 0
Native mode: 0
OS Distribution: ubuntu 16.04
Batch Shipyard version: 3.7.0
Blobxfer version: 1.7.0
Singularity version: 
User mountpoint: /mnt
Mount path: /mnt/batch/tasks/mounts
Batch Insights: 0
Prometheus: NE=, CA=,
Network optimization: 1
Encryption cert thumbprint: 
Install Kata Containers: 0
Default container runtime: runc
Install BeeGFS BeeOND: 0
Storage cluster mount: 
Custom mount: 
Install LIS: 
GPU: False:nvidia-driver_cc37.run
Azure Blob: 1
Azure File: 0
GlusterFS on compute: 0
HPN-SSH: 0
Enable Azure Batch group for Docker access: 
Fallback registry: 
Docker image preload delay: 0
Cascade via container: 1
P2P: 0
Block on images: REDACTED#

Additonal Comments

[canoas]

Thank you @alfpark ! workaround is working

rootdir=$(awk -F' ' '{print $NF}' <<< $(docker info | grep "Docker Root Dir"))

but we had to recreate our pool from a clean state using a recompiled version of shipyard

[elemakil]

@alfpark Thanks a ton for providing such a quick workaround! Using native: true has indeed resolved this issue for me.