Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Error when importing 10000 images

See original GitHub issue

My actions before raising this issue

Read/searched the docs
Searched past issues

I have 10000 images, 42MB each, mounted into a CVAT share.
Creating a task with 1 such image succeeds
Creating the task with 10000 of them fails - well, partly - as the imported images can be seen and annotated after re-login into CVAT,
but error logs show, some internal structure (i.e. a Task directory containing a client.log) has not been created.

Expected Behaviour

The Task should be created without error.

Current Behaviour

The Task is only partly created.
It can be seen after logout-login,
but whenever we annotate and save something,
an error panel pops up, telling the the log cannot be written

Possible Solution

Maybe, someone have a look at it, as it seems to be a timeout issue to me…

Steps to Reproduce (for bugs)

logged into fresh CVAT as superuser: django
created Project: Project1
with Labels: L1
created Task: Task1
Connected file share /media/sda -> A.png (50 byte size)
Advanced configuration
Not copying data into CVAT
Not using zip chunks
Use cache
Image quality: 100%
Format: CVAT for images 1.1
ok! Task created

created Task2
Connected file share /media/sda -> 0_16kx1k.png (42MB size)
ok! Task created

Try to create Task3
Connected file share /media/sda -> 0_16kx1k.png … 10000_16kx1k.png (10000 images with 42MB)
Could not create the task

Context

We try to annotate large RGB images (16k x 1k), which usually works well…

Your Environment

2 CPU slots with 6 cores each (24 virtual cores)
128GB RAM

Git hash commit (git log -1): commit 5e431714863d4be44dc05ecd1b76c9c056658103 (HEAD -> develop, origin/develop, origin/HEAD)
Author: Dmitry Kruchinin dmitryx.kruchinin@intel.com
Server v1.7
Core v3.17.0
Canvas v2.8.0
UI v1.25.0
Docker version: Docker 20.10.12, Docker-compose 1.29.2
Are you using Docker Swarm or Kubernetes? No
Operating System and version: Ubuntu 20.04
Code example or link to GitHub repo or gist to reproduce problem:
Other diagnostic information / logs:

Logs from `cvat` container

cvat_redis | 1:M 27 Jan 09:22:12.132 * 100 changes in 300 seconds. Saving… cvat_redis | 1:M 27 Jan 09:22:12.133 * Background saving started by pid 14 cvat_redis | 14:C 27 Jan 09:22:12.170 * DB saved on disk cvat_redis | 14:C 27 Jan 09:22:12.170 * RDB: 0 MB of memory used by copy-on-write cvat_redis | 1:M 27 Jan 09:22:12.233 * Background saving terminated with success cvat | 2022-01-27 09:23:21,547 DEBG ‘rqworker_default_0’ stderr output: cvat | INFO - 2022-01-27 09:23:21,547 - worker - default: cvat.apps.engine.task._create_thread(2, {‘chunk_size’: None, ‘size’: 0, ‘image_quality’: 100, ‘start_frame’: 0, 'st…) (/api/v1/tasks/2) cvat | cvat | 2022-01-27 09:23:21,547 DEBG ‘rqworker_default_0’ stderr output: cvat | DEBUG - 2022-01-27 09:23:21,547 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds. cvat | cvat | 2022-01-27 09:23:21,567 DEBG ‘rqworker_default_0’ stderr output: cvat | DEBUG - 2022-01-27 09:23:21,566 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 90 seconds. cvat | cvat | 2022-01-27 09:23:21,578 DEBG ‘rqworker_default_0’ stderr output: cvat | [2022-01-27 09:23:21,578] INFO cvat.server: create task #2 cvat | cvat | 2022-01-27 09:23:21,578 DEBG ‘rqworker_default_0’ stderr output: cvat | INFO - 2022-01-27 09:23:21,578 - task - create task #2 cvat | cvat | 2022-01-27 09:23:22,058 DEBG ‘rqworker_default_0’ stderr output: cvat | [2022-01-27 09:23:22,058] INFO cvat.server: Found frames 1 for Data #2 cvat | cvat | 2022-01-27 09:23:22,058 DEBG ‘rqworker_default_0’ stderr output: cvat | INFO - 2022-01-27 09:23:22,058 - task - Found frames 1 for Data #2 cvat | cvat | 2022-01-27 09:23:22,059 DEBG ‘rqworker_default_0’ stderr output: cvat | [2022-01-27 09:23:22,059] INFO cvat.server: New segment for task #2: start_frame = 0, stop_frame = 0 cvat | cvat | 2022-01-27 09:23:22,059 DEBG ‘rqworker_default_0’ stderr output: cvat | INFO - 2022-01-27 09:23:22,059 - task - New segment for task #2: start_frame = 0, stop_frame = 0 cvat | cvat | 2022-01-27 09:23:22,083 DEBG ‘rqworker_default_0’ stderr output: cvat | DEBUG - 2022-01-27 09:23:22,083 - worker - Handling successful execution of job /api/v1/tasks/2 cvat | cvat | 2022-01-27 09:23:22,086 DEBG ‘rqworker_default_0’ stderr output: cvat | INFO - 2022-01-27 09:23:22,086 - worker - default: Job OK (/api/v1/tasks/2) cvat | cvat | 2022-01-27 09:23:22,086 DEBG ‘rqworker_default_0’ stderr output: cvat | INFO - 2022-01-27 09:23:22,086 - worker - Result is kept for 500 seconds cvat | cvat | 2022-01-27 09:23:22,103 DEBG ‘rqworker_default_0’ stderr output: cvat | DEBUG - 2022-01-27 09:23:22,102 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds. cvat | cvat | 2022-01-27 09:23:22,103 DEBG ‘rqworker_default_0’ stderr output: cvat | DEBUG - 2022-01-27 09:23:22,103 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds. cvat | cvat | 2022-01-27 09:23:22,104 DEBG ‘rqworker_default_0’ stderr output: cvat | DEBUG - 2022-01-27 09:23:22,103 - worker - *** Listening on default… cvat | cvat | 2022-01-27 09:23:22,104 DEBG ‘rqworker_default_0’ stderr output: cvat | DEBUG - 2022-01-27 09:23:22,104 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds. cvat | cvat_redis | 1:M 27 Jan 09:27:21.116 * 100 changes in 300 seconds. Saving… cvat_redis | 1:M 27 Jan 09:27:21.117 * Background saving started by pid 15 cvat_redis | 15:C 27 Jan 09:27:21.145 * DB saved on disk cvat_redis | 15:C 27 Jan 09:27:21.145 * RDB: 0 MB of memory used by copy-on-write cvat_redis | 1:M 27 Jan 09:27:21.217 * Background saving terminated with success cvat | 2022-01-27 09:27:44,886 DEBG ‘rqworker_low’ stderr output: cvat | DEBUG - 2022-01-27 09:27:44,885 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds. cvat | cvat | 2022-01-27 09:28:57,181 DEBG ‘rqworker_default_1’ stderr output: cvat | DEBUG - 2022-01-27 09:28:57,181 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds. cvat | cvat | 2022-01-27 09:29:05,943 DEBG ‘runserver’ stderr output: cvat | [Thu Jan 27 09:29:05.943023 2022] [wsgi:error] [pid 743:tid 140638102210304] [client 172.20.0.5:55488] Timeout when reading response headers from daemon process ‘localhost:8080’: /tmp/cvat-server/htdocs/api, referer: https://<my.ip.address>/tasks/create?projectId=1 cvat | cvat | 2022-01-27 09:29:31,830 DEBG ‘rqworker_default_0’ stderr output: cvat | INFO - 2022-01-27 09:29:31,830 - worker - default: cvat.apps.engine.task._create_thread(3, {‘chunk_size’: None, ‘size’: 0, ‘image_quality’: 100, ‘start_frame’: 0, 'st…) (/api/v1/tasks/3) cvat | cvat | 2022-01-27 09:29:31,831 DEBG ‘rqworker_default_0’ stderr output: cvat | DEBUG - 2022-01-27 09:29:31,831 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds. cvat | cvat | 2022-01-27 09:29:31,850 DEBG ‘rqworker_default_0’ stderr output: cvat | DEBUG - 2022-01-27 09:29:31,848 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 90 seconds. cvat | cvat | 2022-01-27 09:29:31,868 DEBG ‘rqworker_default_0’ stderr output: cvat | [2022-01-27 09:29:31,867] INFO cvat.server: create task #3 cvat | cvat | 2022-01-27 09:29:31,868 DEBG ‘rqworker_default_0’ stderr output: cvat | INFO - 2022-01-27 09:29:31,867 - task - create task #3 cvat | cvat | 2022-01-27 09:29:38,156 DEBG ‘rqworker_default_0’ stderr output: cvat | [2022-01-27 09:29:38,155] INFO cvat.server: Found frames 10000 for Data #3 cvat | cvat | 2022-01-27 09:29:38,156 DEBG ‘rqworker_default_0’ stderr output: cvat | INFO - 2022-01-27 09:29:38,155 - task - Found frames 10000 for Data #3 cvat | cvat | 2022-01-27 09:29:38,156 DEBG ‘rqworker_default_0’ stderr output: cvat | [2022-01-27 09:29:38,156] INFO cvat.server: New segment for task #3: start_frame = 0, stop_frame = 9999 cvat | cvat | 2022-01-27 09:29:38,157 DEBG ‘rqworker_default_0’ stderr output: cvat | INFO - 2022-01-27 09:29:38,156 - task - New segment for task #3: start_frame = 0, stop_frame = 9999 cvat | cvat | 2022-01-27 09:29:38,717 DEBG ‘rqworker_default_0’ stderr output: cvat | DEBUG - 2022-01-27 09:29:38,716 - worker - Handling successful execution of job /api/v1/tasks/3 cvat |