Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

DEXTR - Semi-Automatic Annotation - 504 Gateway Timeout

See original GitHub issue

My actions before raising this issue

Read/searched the docs
Searched past issues

I have searched the docs and past issues - could not figure it out, kindly help.

Expected Behaviour

Automatic instance segmentation annotation

Current Behaviour

Getting below error:

Gateway Timeout
The gateway did not receive a timely response from the upstream server or application.

Possible Solution

From logs, it looks like connection not reaching the docker instance - may be providing some advice on configuring proxy/firewall will help us to understand the issue better.

Steps to Reproduce (for bugs)

Followed the steps provided in the installation guide:

https://openvinotoolkit.github.io/cvat/docs/for-users/installation/

Note: Installation is successful, I will share the logs and details below

Context

When trying to use AI Tool getting timeout error

Your Environment

Git hash commit (git log -1):

commit 3afd7e827878112e6d6bf9a435e7b96d38baaa26 (HEAD -> develop, origin/develop, origin/HEAD)
Author: Dmitry Kruchinin <33020454+dvkruchinin@users.noreply.github.com>
Date:   Mon Jun 7 11:59:10 2021 +0300
    Cypress test. Create an object, save, undo, save, redo, save. (#3299)

Docker version docker version (e.g. Docker 17.0.05):

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 12:00:45 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:58:56 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Are you using Docker Swarm or Kubernetes? No
Operating System and version (e.g. Linux, Windows, MacOS):

NAME="Ubuntu"
VERSION="21.04 (Hirsute Hippo)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 21.04"
VERSION_ID="21.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=hirsute
UBUNTU_CODENAME=hirsute

Code example or link to GitHub repo or gist to reproduce problem: Same as official repo
Other logs:

docker ps:

CONTAINER ID   IMAGE                                   COMMAND                  CREATED        STATUS                  PORTS                                                                                
                             NAMES
3e43c3e7b553   alpine:3.11                             "/bin/sh -c '/bin/sl…"   5 hours ago    Up 5 hours                                                                                                   
                             nuclio-local-storage-reader
296d755cc856   cvat/openvino.dextr:latest              "processor"              11 hours ago   Up 11 hours (healthy)   0.0.0.0:49153->8080/tcp, :::49153->8080/tcp                                          
                             nuclio-nuclio-openvino-dextr
8e4e3f6f04ae   nginx:stable-alpine                     "/docker-entrypoint.…"   11 hours ago   Up 11 hours             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:808
0->80/tcp, :::8080->80/tcp   cvat_proxy
f1ec02907177   openvino/cvat_ui                        "/docker-entrypoint.…"   11 hours ago   Up 11 hours             80/tcp                                                                               
                             cvat_ui
1b32376d2c98   openvino/cvat_server                    "/usr/bin/supervisord"   11 hours ago   Up 11 hours             8080/tcp                                                                             
                             cvat
19675add9961   postgres:10-alpine                      "docker-entrypoint.s…"   11 hours ago   Up 11 hours             5432/tcp                                                                             
                             cvat_db
f52bcf45003b   redis:4.0-alpine                        "docker-entrypoint.s…"   11 hours ago   Up 11 hours             6379/tcp                                                                             
                             cvat_redis
0090c6624c5d   quay.io/nuclio/dashboard:1.5.16-amd64   "/docker-entrypoint.…"   11 hours ago   Up 11 hours (healthy)   80/tcp, 0.0.0.0:8070->8070/tcp, :::8070->8070/tcp                                    
                             nuclio

docker logs cvat: (only error part)

[Mon Jun 07 18:55:17.009057 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:34580] [2021-06-07 18:55:17,008] INFO cvat.server.task_2: get repository request
2021-06-07 18:55:17,010 DEBG 'runserver' stderr output:
[Mon Jun 07 18:55:17.010097 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:34580] INFO - 2021-06-07 18:55:17,008 - views - get repository request
2021-06-07 18:55:20,822 DEBG 'runserver' stderr output:
[Mon Jun 07 18:55:20.822152 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:34608] INFO - 2021-06-07 18:55:20,822 - views - {"client_id":582112,"name":"Send user activity","time":"2021
-06-07T18:55:20.723000Z","payload":{"working_time":50033},"is_active":true,"username":"admin"}
2021-06-07 18:56:35,536 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:35.536331 2021] [wsgi:error] [pid 322:tid 140652568270592] [client 172.28.0.7:34694] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/a
pi, referer: https://my-server-ip/tasks/2/jobs/2
2021-06-07 18:56:35,637 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:35.637400 2021] [wsgi:error] [pid 255:tid 140652446222080] [remote 172.28.0.7:34694] [2021-06-07 18:56:35,637] ERROR django.request: Gateway Timeout: /api/v1/lambda/functions/openvino-de
xtr
2021-06-07 18:56:35,637 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:35.637517 2021] [wsgi:error] [pid 255:tid 140652446222080] [remote 172.28.0.7:34694] ERROR - 2021-06-07 18:56:35,637 - log - Gateway Timeout: /api/v1/lambda/functions/openvino-dextr
2021-06-07 18:56:51,575 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:51.575198 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:35000] INFO - 2021-06-07 18:56:51,575 - views - {"job_id":2,"task_id":2,"client_id":582112,"name":"Load job"
,"time":"2021-06-07T18:55:20.723000Z","payload":{"duration":1450,"frame count":7,"track count":0,"object count":0,"box count":0,"polygon count":0,"polyline count":0,"points count":0,"cuboids count":0,"tag
 count":0},"is_active":true,"username":"admin"}
2021-06-07 18:56:51,575 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:51.575359 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:35000] INFO - 2021-06-07 18:56:51,575 - views - {"job_id":2,"task_id":2,"client_id":582112,"name":"Fit image
","time":"2021-06-07T18:55:22.176000Z","payload":{},"is_active":true,"username":"admin"}
[Mon Jun 07 18:56:51.575465 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:35000] INFO - 2021-06-07 18:56:51,575 - views - {"client_id":582112,"name":"Send user activity","time":"2021
-06-07T18:56:51.481000Z","payload":{"working_time":14505},"is_active":true,"username":"admin"}
2021-06-07 18:57:30,995 DEBG 'runserver' stderr output:
[Mon Jun 07 18:57:30.995435 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:35178] [2021-06-07 18:57:30,995] INFO cvat.server.task_2: get repository request
2021-06-07 18:57:30,995 DEBG 'runserver' stderr output:
[Mon Jun 07 18:57:30.995579 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:35178] INFO - 2021-06-07 18:57:30,995 - views - get repository request
2021-06-07 18:58:41,613 DEBG 'rqworker_low' stderr output:
DEBUG - 2021-06-07 18:58:41,612 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
2021-06-07 18:58:41,812 DEBG 'rqworker_default_1' stderr output:
DEBUG - 2021-06-07 18:58:41,812 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
2021-06-07 18:59:59,108 DEBG 'runserver' stderr output:
[Mon Jun 07 18:59:59.108022 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:35766] INFO - 2021-06-07 18:59:59,107 - views - {"client_id":582112,"name":"Send user activity","time":"2021
-06-07T18:59:59.009000Z","payload":{"working_time":50882},"is_active":true,"username":"admin"}
2021-06-07 19:01:13,421 DEBG 'runserver' stderr output:
[Mon Jun 07 19:01:13.421313 2021] [wsgi:error] [pid 322:tid 140652568004352] [client 172.28.0.7:35852] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/a
pi, referer: https://my-server-ip/tasks/2/jobs/2
2021-06-07 19:01:13,590 DEBG 'runserver' stderr output:
[Mon Jun 07 19:01:13.590114 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:35852] [2021-06-07 19:01:13,590] ERROR django.request: Gateway Timeout: /api/v1/lambda/functions/openvino-de
xtr
2021-06-07 19:01:13,590 DEBG 'runserver' stderr output:
[Mon Jun 07 19:01:13.590248 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:35852] ERROR - 2021-06-07 19:01:13,590 - log - Gateway Timeout: /api/v1/lambda/functions/openvino-dextr
2021-06-07 19:01:33,901 DEBG 'runserver' stderr output:
[Mon Jun 07 19:01:33.901047 2021] [core:info] [pid 322:tid 140652568803072] [client 172.28.0.7:36202] AH00128: File does not exist: /home/django/static/rest_framework/css/bootstrap.min.css.map
2021-06-07 19:01:54,672 DEBG 'rqworker_default_0' stderr output:
DEBUG - 2021-06-07 19:01:54,671 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

nuctl get function:

  NAMESPACE |      NAME      | PROJECT | STATE | NODE PORT | REPLICAS  
  nuclio    | openvino-dextr | cvat    | ready |     49153 | 1/1

nuctl version:

Client version:
"Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"root@gc-ase-ml-data-annotate1:/data/cvat#

systemctl show --property=Environment docker:

Environment=HTTP_PROXY=http://my-proxy-ip:8080 HTTPS_PROXY=http://my-proxy-ip:8080 NO_PROXY=localhost,127.0.0.1

cat ~/.docker/config.json:

{
 "proxies":
 {
   "default":
   {
     "httpProxy": "http://my-proxy-ip:8080",
     "httpsProxy": "http://my-proxy-ip:8080",
     "noProxy": "*.mydomain.com,localhost,127.0.0.0/8"
   }
 }
}

docker-compose.override.yml:

version: '3.3'
services:
  cvat_proxy:
    environment:
      CVAT_HOST: 'my-host-ip'
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - /etc/ssl/certs:/etc/ssl/certs
  cvat:
    environment:
      ALLOWED_HOSTS: '*'

Screenshots:

API Page:

API Output:

Logs for this request:

Next steps

I have joined gitter and shared details - waiting for response.

Issue Analytics

State:
Created 2 years ago
Comments:19 (6 by maintainers)

Top GitHub Comments

1reaction

nmanoviccommented, Sep 24, 2021

@nikste , @rathishkumar ,

Do you have proxy on your machine? If so you need:

Add information about your proxy into ~/.docker/config.json file: https://openvinotoolkit.github.io/cvat/docs/administration/basics/installation/#deploying-cvat-behind-a-proxy
Also it is important to add your docker gateway into noProxy (e.g. 172.17.0.1). Nuclio dashboard and serverless functions are talking over the docker gateway and if it is protected by a proxy, they cannot talk.
Down/Up nuclio container (Nuclio dashboard)

0reactions

nvqui97commented, Oct 14, 2022

Please help me to solve this. My server doesn’t have proxy. I also see the result in model container but on interface isn’t