Deployment of siammask works, but CVAT throws 500 Error when using tracking function

My actions before raising this issue

• [ x] https://github.com/openvinotoolkit/cvat/blob/develop/cvat/apps/documentation/installation_automatic_annotation.md
• Searched https://github.com/openvinotoolkit/cvat/issues/2641 (and actually more on this thread)
• https://github.com/nuclio/nuclio/issues/1821 in order to get NUCTL with my OS working

SUMMARY

• Deployment and Installation of CVAT was alright.
• I use Linux nuctl, because windows nuctl didn’t work. Weird docker linux windows combination is the reason why.
• Installation of siammask also works, and Nuclio dashbord says its running and cvat model tab also.
• nuctl invoke siammask doesn’t work. (most likely wrong call, though the error message of connection refused was the worrying part).
• I think error might be due to docker network issue, though I didn’t deviate in terms of installing CVAT and deploying siammask via nuctl. Maybe someone knows what I did wrong when installing it.

Expected Behaviour

When using the tracking model in cvat the tracking just “works”. And doesn’t get a http 500

Current Behaviour

When calling the tracking model in CVAT tool the server return a 500. I have narrowed down the problem to a network problem I believe, though I’d still like to confirm whether my installation steps were correct or not.

Steps to Reproduce (for bugs)

1. Install CVAT as https://github.com/openvinotoolkit/cvat/blob/develop/cvat/apps/documentation/installation.md (without the "wrong docker-compose command)
2. Follow https://github.com/openvinotoolkit/cvat/blob/develop/cvat/apps/documentation/installation_automatic_annotation.md
3. docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d
4. For nuctl i used ubuntu + linux version, because the Windows nuctl didn’t work. Installation Command for siammask works fine (log below) . ./nuctl deploy --project-name cvat --path serverless/pytorch/foolwood/siammask/nuclio/ --volume pwd/serverless/common:/opt/nuclio/common --platform local
5. Login to CVAT and use Tracking functionality

Error description

When calling the functionality i get the Error: Nuclio Log

21.04.02 07:20:28.490 ashboard.platform.invoker (I) Executing function {"method": "POST", "url": "http://172.17.0.1:0", "headers": {"Accept":["*/*"],
"Accept-Encoding":["gzip, deflate"],"Connection":["close"],"Content-Length":["1362188"],"Content-Type":["application/json"],"User-Agent":["python-req
uests/2.24.0"],"X-Nuclio-Function-Name":["pth-foolwood-siammask"],"X-Nuclio-Function-Namespace":["nuclio"],"X-Nuclio-Log-Level":[""],"X-Nuclio-Path":
["/"],"X-Nuclio-Project-Name":["cvat"],"X-Nuclio-Target":["pth-foolwood-siammask"]}}
21.04.02 07:20:28.490 .api/function_invocations (W) Failed to invoke function {"err": "Failed to send HTTP request", "errVerbose": "\nError - Post \"
http://172.17.0.1:0\": dial tcp 172.17.0.1:0: connect: connection refused\n    /nuclio/pkg/platform/abstract/invoker.go:119\n\nCall stack:\nFailed to
 send HTTP request\n    /nuclio/pkg/platform/abstract/invoker.go:119\nFailed to send HTTP request", "errCauses": [{"error": "Post \"http://172.17.0.1
:0\": dial tcp **172.17.0.1:0: connect: connection refused**"}]}

My container network overview:

Bridge.json

{
    "Name": "bridge",
    "Id": "a918296c39e23a624cc9f79da173a9adc331487438d903543fef9c16b57f0f8a",
    "Created": "2021-04-02T06:58:32.094464Z",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv6": false,
    "IPAM": {
        "Driver": "default",
        "Options": null,
        "Config": [
            {
                "Subnet": "172.17.0.0/16",
                "Gateway": "**172.17.0.1**"
            }
        ]
    } ...............

So my reasoning based on the error and the network config was, that the Nuclio / cvat application, can’t access the siammask function. When the nuclio server tries to connect to the siammask func via the gate 172.17.0.1, he gets an connection refused.

I also tried to connect the nuclio via “docker network connect bridge nuclio”, but that also didn’t change the error message. But I am pretty much “just trying” and not 100% sure. Given my few experiences with nuclio and docker.

I was also inside the nuclio container and tried ping the gateway, which worked no problem, I mabe will try using getting telnet on the container and try the Port 0, but honestly even if I don’t have an idea in term of the fix.

I am also not sure whether the deployment was correct, when the siammask model container is not in the same network as the cvat containers. The guide also does not mention any necessary network configuration after using the nuctl deploy command so i hope u guys might be able to help.

Context

I just like to use the Siammask model for easier annotation for my dataset.

Your Environment

• Githash: commit a60b4895c2f6fda292cf890c34e24df6535dd2e2
• Docker version docker version (e.g. Docker 17.0.05): 20.10.5
• Plain Docker on Windows 10 (home)
• Windows 10
• Ubuntu 20
• nuctl 1.5.16 (LINUX)

Logs

NUCLIO Log installation

`CVAT$ ./nuctl deploy --project-name cvat   --path serverless/pytorch/foolwood/siammask/nuclio/   --volume `pwd`/serverless/common:/opt/nuclio/common   --platform local
21.04.02 11:01:06.059                     nuctl (I) Deploying function {"name": ""}
21.04.02 11:01:06.060                     nuctl (I) Building {"versionInfo": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""}
21.04.02 11:01:06.554                     nuctl (I) Cleaning up before deployment {"functionName": "pth-foolwood-siammask"}
21.04.02 11:01:06.673                     nuctl (I) Staging files and preparing base images
21.04.02 11:01:06.706                     nuctl (I) Building processor image {"imageName": "cvat/pth.foolwood.siammask:latest"}
21.04.02 11:01:06.706     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.5.16-amd64"}
21.04.02 11:01:09.785     nuctl.platform.docker (W) Docker command outputted to stderr - this may result in errors {"workingDir": "/tmp/nuclio-build-500909153/staging", "cmd": "docker build --network host --force-rm -t nuclio-onbuild-c1jdqleo8vutvqjbsu50 -f /tmp/nuclio-build-500909153/staging/Dockerfile.onbuild  --build-arg NUCLIO_LABEL=1.5.16 --build-arg NUCLIO_ARCH=amd64 --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler  .", "stderr": "#1 [internal] load build definition from Dockerfile.onbuild\n#1 sha256:648c58c5c30185132d66115694bd004636c8c289cc5cb000d59d5c4ad01a2aaa\n#1 transferring dockerfile: 148B done\n#1 DONE 0.0s\n\n#2 [internal] load .dockerignore\n#2 sha256:33410abe26f6f76776652199c89200845bb835bdcc3571095dff03b392194346\n#2 transferring context: 2B done\n#2 DONE 0.0s\n\n#3 [internal] load metadata for quay.io/nuclio/handler-builder-python-onbuild:1.5.16-amd64\n#3 sha256:3d4c217b8dd4735b204a58a813d2ff5a52b7e584fa30653b3b5b38b4c94539f0\n#3 DONE 0.0s\n\n#4 [1/1] FROM quay.io/nuclio/handler-builder-python-onbuild:1.5.16-amd64\n#4 sha256:a3bb79ea4dc1e3f9ab4814a1d071094b6d7a3946ba3c32c40ce3f16b9f75d0fd\n#4 CACHED\n\n#5 exporting to image\n#5 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00\n#5 exporting layers done\n#5 writing image sha256:708a12fe13b56186c5cc413457939deb77bd05f4cea06e23899aee91ce2adb48 0.0s done\n#5 naming to docker.io/library/nuclio-onbuild-c1jdqleo8vutvqjbsu50 done\n#5 DONE 0.0s\n"}
21.04.02 11:01:10.647     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"}
21.04.02 11:01:14.760     nuctl.platform.docker (W) Docker command outputted to stderr - this may result in errors {"workingDir": "/tmp/nuclio-build-500909153/staging", "cmd": "docker build --network host --force-rm -t nuclio-onbuild-c1jdqmmo8vutvqjbsu5g -f /tmp/nuclio-build-500909153/staging/Dockerfile.onbuild  --build-arg NUCLIO_LABEL=1.5.16 --build-arg NUCLIO_ARCH=amd64 --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler  .", "stderr": "#1 [internal] load build definition from Dockerfile.onbuild\n#1 sha256:e2e57bcc592bba69fa9c9c3a77a59c203a1a1da8b0ab544d18dcbfe139c4033a\n#1 transferring dockerfile: 123B done\n#1 DONE 0.0s\n\n#2 [internal] load .dockerignore\n#2 sha256:e3e5f0dfc082b002f09f530239d325e172df22ac371b0dd9fbea5603afefab53\n#2 transferring context: 2B done\n#2 DONE 0.0s\n\n#3 [internal] load metadata for quay.io/nuclio/uhttpc:0.0.1-amd64\n#3 sha256:4aba9ea98350709bc44db6d4b0f46352caf4a7f2c8d89585e87a03a7a14007b3\n#3 DONE 0.0s\n\n#4 [1/1] FROM quay.io/nuclio/uhttpc:0.0.1-amd64\n#4 sha256:92bc73b1ee90a814263ae3231095a19e72949aebd7bf4477b119cea12b042ff9\n#4 CACHED\n\n#5 exporting to image\n#5 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00\n#5 exporting layers done\n#5 writing image sha256:1d7a80c3d68af5b05150876d5d966124a5b38181a67443addc1eef553b11de08 done\n#5 naming to docker.io/library/nuclio-onbuild-c1jdqmmo8vutvqjbsu5g done\n#5 DONE 0.0s\n"}
21.04.02 11:01:15.355            nuctl.platform (I) Building docker image {"image": "cvat/pth.foolwood.siammask:latest"}21.04.02 11:01:20.435     nuctl.platform.docker (W) Docker command outputted to stderr - this may result in errors {"workingDir": "/tmp/nuclio-build-500909153/staging", "cmd": "docker build --network host --force-rm -t cvat/pth.foolwood.siammask:latest -f /tmp/nuclio-build-500909153/staging/Dockerfile.processor  --build-arg NUCLIO_ARCH=amd64 --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler --build-arg NUCLIO_LABEL=1.5.16  .", "stderr": "#1 [internal] load build definition from Dockerfile.processor\n#1 sha256:6bc90684e2a4de2dca13ca9f486800e8d26d9336f907dccfea62e331d9c878e6\n#1 transferring dockerfile: 1.27kB done\n#1 DONE 0.0s\n\n#2 [internal] load .dockerignore\n#2 sha256:f494734f7f1c9d9768433eac33c6cc8aa7b54f8e547d0d323d0d6256f37ddcb9\n#2 transferring context: 2B done\n#2 DONE 0.0s\n\n#3 [internal] load metadata for docker.io/continuumio/miniconda3:latest\n#3 sha256:70c01be2623751a8f6df5d5b392c0445964bc863c94f2ca91c3dab835106ad61\n#3 ...\n\n#4 [auth] continuumio/miniconda3:pull token for registry-1.docker.io\n#4 sha256:0390eda111941c2996ddae26e2f0d980ff5ab141c2a8d9fe30b6188b5f0ae14d\n#4 DONE 0.0s\n\n#3 [internal] load metadata for docker.io/continuumio/miniconda3:latest\n#3 sha256:70c01be2623751a8f6df5d5b392c0445964bc863c94f2ca91c3dab835106ad61\n#3 DONE 2.2s\n\n#5 [ 1/13] FROM docker.io/continuumio/miniconda3@sha256:7838d0ce65783b0d944c19d193e2e6232196bada9e5f3762dc7a9f07dc271179\n#5 sha256:7bbac0936a110b8a8ca725fe434ec8895db2c031cc9af6aa28c99aac0eac8e19\n#5 DONE 0.0s\n\n#13 [internal] load build context\n#13 sha256:dd1baf5dccc7645d93b5a4c021e8a486e43e55d5836809550f1bdb9df5c90248\n#13 transferring context: 5.93kB done\n#13 DONE 0.0s\n\n#15 [10/13] COPY artifacts/py /opt/nuclio/\n#15 sha256:fbdbce4cecad9f62b43016a017e34c74cfd8d0be6bfc5fdce1cf45cae4e61e62\n#15 CACHED\n\n#9 [ 5/13] RUN pip install -r SiamMask/requirements.txt jsonpickle\n#9 sha256:f482b5396355675ba774c53d7c9bfc382bc2a6368796106be0fa4b16c794b671\n#9 CACHED\n\n#8 [ 4/13] RUN git clone https://github.com/foolwood/SiamMask.git\n#8 sha256:da31d6c92ef72a13018aea23d986ce589522decfa2f919a4f54629a6caef2555\n#8 CACHED\n\n#11 [ 7/13] RUN cd SiamMask && bash make.sh && cd -\n#11 sha256:2d4779852f85266d0734bd65bf8d09f770185d1dbe92041fdf95679cf76f74fc\n#11 CACHED\n\n#14 [ 9/13] COPY artifacts/processor /usr/local/bin/processor\n#14 sha256:48846f5f851b4e97140c2657c353a7616d0ea19930991e903a7be88d9b0c7a7e\n#14 CACHED\n\n#10 [ 6/13] RUN conda install -y gcc_linux-64\n#10 sha256:33e9c59d4b504a544888a4f382ed11db96f093ee403d81836f585230e94db494\n#10 CACHED\n\n#12 [ 8/13] RUN wget -P SiamMask/experiments/siammask_sharp http://www.robots.ox.ac.uk/~qwang/SiamMask_DAVIS.pth\n#12 sha256:8867b20110e23c2f1dd8b56df1afe191098ec5b98f0a52eedfe5bd6ec73aed00\n#12 CACHED\n\n#7 [ 3/13] RUN conda create -y -n siammask python=3.6\n#7 sha256:71d34445ee3de048535d6bc4b3f3791b26e9ec87440e2863920f03e85d691ea6\n#7 CACHED\n\n#6 [ 2/13] WORKDIR /opt/nuclio\n#6 sha256:534bf8720542c9a93e74b83f409cfa944841d8aaf34d3217e8900753ccb8ae82\n#6 CACHED\n\n#16 [11/13] COPY artifacts/uhttpc /usr/local/bin/uhttpc\n#16 sha256:5f070d3b45c3912e0aaca3bed9c2560dddeb5be7d9f4d21a5a5327fbc54f96b5\n#16 CACHED\n\n#17 [12/13] COPY handler /opt/nuclio\n#17 sha256:ae2f8afcdccf2466c5045e360497a0bbae25af3ebad7d2d395d98e910ce6cb73\n#17 DONE 0.0s\n\n#18 [13/13] RUN pip install nuclio-sdk msgpack --no-index --find-links /opt/nuclio/whl\n#18 sha256:690b7b72407b6f729cf579e4412f6a9dd98d9dcc06f89230f38afa347e3ba5da\n#18 2.279 Looking in links: /opt/nuclio/whl\n#18 2.279 Processing ./whl/nuclio_sdk-0.2.0-py2.py3-none-any.whl\n#18 2.279 Processing ./whl/msgpack-0.6.1.tar.gz\n#18 2.279 Building wheels for collected packages: msgpack\n#18 2.279   Building wheel for msgpack (setup.py): started\n#18 2.279   Building wheel for msgpack (setup.py): finished with status 'done'\n#18 2.279   Created wheel for msgpack: filename=msgpack-0.6.1-cp36-cp36m-linux_x86_64.whl size=14582 sha256=e60589a657f19305ce555e0d0cbb6ac40fd5ff87226ec4f48fcc68534aada855\n#18 2.279   Stored in directory: /root/.cache/pip/wheels/19/df/c3/08424fe5285667aff8a58a69c995c723ffa81c4208138d6d0f\n#18 2.279 Successfully built msgpack\n#18 2.279 Installing collected packages: nuclio-sdk, msgpack\n#18 2.279 Successfully installed msgpack-0.6.1 nuclio-sdk-0.2.0\n#18 2.279 \n#18 DONE 2.3s\n\n#19 exporting to image\n#19 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00\n#19 exporting layers 0.1s done\n#19 writing image sha256:89aa330960833cdc588b6aadef2cc5dbe91d8c14327bed4e74cc0236bdf4a225 done\n#19 naming to docker.io/cvat/pth.foolwood.siammask:latest done\n#19 DONE 0.1s\n"}
21.04.02 11:01:20.436            nuctl.platform (I) Pushing docker image into registry {"image": "cvat/pth.foolwood.siammask:latest", "registry": ""}
21.04.02 11:01:20.436            nuctl.platform (I) Docker image was successfully built and pushed into docker registry {"image": "cvat/pth.foolwood.siammask:latest"}
21.04.02 11:01:20.436                     nuctl (I) Build complete {"result": {"Image":"cvat/pth.foolwood.siammask:latest","UpdatedFunctionConfig":{"metadata":{"name":"pth-foolwood-siammask","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"framework":"pytorch","name":"SiamMask","spec":"","type":"tracker"}},"spec":{"description":"Fast Online Object Tracking and Segmentation","handler":"main:handler","runtime":"python:3.6","env":[{"name":"PYTHONPATH","value":"/opt/nuclio/SiamMask:/opt/nuclio/SiamMask/experiments/siammask_sharp"}],"resources":{},"image":"cvat/pth.foolwood.siammask:latest","targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"myHttpTrigger","maxWorkers":2,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"ingresses":{"0":{"host":"172.28.0.88","paths":["/"]}},"maxRequestBodySize":33554432,"port":0}}},"volumes":[{"volume":{"name":"volume-1","hostPath":{"path":"/mnt/c/Users/Nam/Documents/Implementierung/CVAT/serverless/common"}},"volumeMount":{"name":"volume-1","mountPath":"/opt/nuclio/common"}}],"build":{"image":"cvat/pth.foolwood.siammask","baseImage":"continuumio/miniconda3","directives":{"preCopy":[{"kind":"WORKDIR","value":"/opt/nuclio"},{"kind":"RUN","value":"conda create -y -n siammask python=3.6"},{"kind":"SHELL","value":"[\"conda\", \"run\", \"-n\", \"siammask\", \"/bin/bash\", \"-c\"]"},{"kind":"RUN","value":"git clone https://github.com/foolwood/SiamMask.git"},{"kind":"RUN","value":"pip install -r SiamMask/requirements.txt jsonpickle"},{"kind":"RUN","value":"conda install -y gcc_linux-64"},{"kind":"RUN","value":"cd SiamMask \u0026\u0026 bash make.sh \u0026\u0026 cd -"},{"kind":"RUN","value":"wget -P SiamMask/experiments/siammask_sharp http://www.robots.ox.ac.uk/~qwang/SiamMask_DAVIS.pth"},{"kind":"ENTRYPOINT","value":"[\"conda\", \"run\", \"-n\", \"siammask\"]"}]},"codeEntryType":"image"},"platform":{"attributes":{"mountMode":"volume","restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":60,"securityContext":{},"eventTimeout":"30s"}}}}
21.04.02 11:01:22.327            nuctl.platform (I) Waiting for function to be ready {"timeout": 60}
21.04.02 11:01:24.214                     nuctl (I) Function deploy complete {"functionName": "pth-foolwood-siammask", "httpPort": 0}

NUCTL Function

 NAMESPACE |         NAME          | PROJECT | STATE | NODE PORT | REPLICAS
 nuclio    | pth-foolwood-siammask | cvat    | ready |         0 | 1/1

Next steps

You may join our Gitter channel for community support.

[Zailux]

Hi @nmanovic,

thanks for looking into it, but I found the solution to my problem. Just forgot to answer here. The issue was, that the Services ran under the Port 0 --> big hint. Everything that runs below Port 1000 is kinda suspicious, if it isn’t something “Standard” like DNS Server or the like.

I changed the function.yaml for the respective models in the /serverless directory, and now the models can be accessed properly.

  triggers:
    myHttpTrigger:
      maxWorkers: 2
      kind: 'http'
      workerAvailabilityTimeoutMilliseconds: 10000
      attributes:
        maxRequestBodySize: 33554432 # 32MB
        ** here
        port: 8000
        ingresses:
          "8000":
            paths:
              - /
            host: 172.28.0.88
        **

The host ip adress is in the 172.28.0.* i chose based on the docker network config “cvat_default”, so the network in which the CVAT apps runs 😃

Cheers 😃

maybe you could add this part in your read.md or the like. Or maybe you already did but I am too dumb to find it in your docs x((

[mallyagirish]

@nmanovic Thank you. That fixed it indeed. I downloaded nuctl-1.6.4 and also modified my docker-compose.serverless.yml to use the matching Nuclio dashboard version.

...
image: quay.io/nuclio/dashboard:1.6.4-amd64
...