Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Nuclio related 500 server error

See original GitHub issue

My actions before raising this issue

[ yes ] Read/searched the docs
[ yes ] Searched past issues

Current Behaviour

I have so far deployed Dextr and GPU tf faster rcnn on cvat. i used them right after deployment and they went well. but after a reboot of system they wont work anymore. saying : Interaction error occured Error: Request failed with status code 500. "500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations".

I have been deleting functions in nuctl and re deploying them, which gives me the same 500 error. Tho my nuctl version is 1.5.10, which is not the same with the official one.

and i also encountered this issue https://github.com/nuclio/nuclio/issues/2027#issue-776533528

I believe a thorough reinstall of both docker and cvat might solve the problem before my computer restarts after that…

Your Environment

cvat version 1.2.0
Docker version 20.10.1
Operating System and version : ubuntu 1804
nuctl version : 1.5.10
Code example or link to GitHub repo or gist to reproduce problem:
Other diagnostic information / logs:

Logs from `cvat` container

Next steps

You may join our Gitter channel for community support. docker logs cvat :


2021-01-04 07:37:31,381 DEBG 'runserver' stderr output:
[Mon Jan 04 07:37:31.380811 2021] [wsgi:error] [pid 712:tid 140249678059264] [remote 172.28.0.4:57076] INFO:cvat.client:{"client_id":845691,"name":"Send user activity","time":"2021-01-04T07:37:32.110000Z","payload":{"working_time":6417},"is_active":true,"username":""}

2021-01-04 07:37:41,729 DEBG 'rqworker_default_1' stderr output:
DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-01-04 07:37:41,730 DEBG 'rqworker_low' stderr output:
DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
INFO:rq.worker:Cleaning registries for queue: low

2021-01-04 07:37:41,971 DEBG 'rqworker_default_0' stderr output:
DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-01-04 07:38:09,026 DEBG 'runserver' stderr output:
[Mon Jan 04 07:38:09.025994 2021] [wsgi:error] [pid 712:tid 140249661273856] [remote 172.28.0.4:57110] [2021-01-04 07:38:09,025] ERROR django.request: Internal Server Error: /api/v1/lambda/functions/openvino-dextr

2021-01-04 07:38:09,026 DEBG 'runserver' stderr output:
[Mon Jan 04 07:38:09.026431 2021] [wsgi:error] [pid 712:tid 140249661273856] [remote 172.28.0.4:57110] ERROR:django.request:Internal Server Error: /api/v1/lambda/functions/openvino-dextr

2021-01-04 07:39:40,750 DEBG 'runserver' stderr output:
[Mon Jan 04 07:39:40.750004 2021] [wsgi:error] [pid 712:tid 140249686451968] [remote 172.28.0.4:57270] [2021-01-04 07:39:40,749] ERROR django.request: Internal Server Error: /api/v1/lambda/functions/openvino-dextr

2021-01-04 07:39:40,751 DEBG 'runserver' stderr output:
[Mon Jan 04 07:39:40.750389 2021] [wsgi:error] [pid 712:tid 140249686451968] [remote 172.28.0.4:57270] ERROR:django.request:Internal Server Error: /api/v1/lambda/functions/openvino-dextr

selected from ‘docker logs nuclio’

21.01.04 07:43:13.420 ashboard.platform.invoker (I) Executing function {"method": "POST", "url": "http://172.17.0.1:49155", "headers": {"Accept":["*/*"],"Accept-Encoding":["gzip, deflate"],"Connection":["close"],"Content-Length":["1405352"],"Content-Type":["application/json"],"User-Agent":["python-requests/2.24.0"],"X-Nuclio-Function-Name":["openvino-dextr"],"X-Nuclio-Function-Namespace":["nuclio"],"X-Nuclio-Log-Level":[""],"X-Nuclio-Path":["/"],"X-Nuclio-Project-Name":["cvat"],"X-Nuclio-Target":["openvino-dextr"]}}

21.01.04 07:46:53.672 dashboard.platform.docker (D) Container is healthy {"containerID": "3bb4b0d99fca17ab3564c536aba272b4469601d7d9626f3b34346a87865314ae"}
21.01.04 07:46:56.519 .api/function_invocations (W) Failed to invoke function {"err": "Failed to send HTTP request", "errVerbose": "\nError - Post \"http://172.17.0.1:49155\": dial tcp 172.17.0.1:49155: i/o timeout\n    /nuclio/pkg/platform/abstract/invoker.go:119\n\nCall stack:\nFailed to send HTTP request\n    /nuclio/pkg/platform/abstract/invoker.go:119\nFailed to send HTTP request", "errCauses": [{"error": "Post \"http://172.17.0.1:49155\": dial tcp 172.17.0.1:49155: i/o timeout"}]}
21.01.04 07:46:56.519          dashboard.server (D) Handled request {"requestMethod": "POST", "requestPath": "/api/function_invocations", "requestHeaders": {"Accept":["*/*"],"Accept-Encoding":["gzip, deflate"],"Connection":["close"],"Content-Length":["1405384"],"Content-Type":["application/json"],"User-Agent":["python-requests/2.24.0"],"X-Nuclio-Function-Name":["openvino-dextr"],"X-Nuclio-Function-Namespace":["nuclio"],"X-Nuclio-Log-Level":[""],"X-Nuclio-Path":["/"],"X-Nuclio-Project-Name":["cvat"],"X-Nuclio-Target":["openvino-dextr"]}, "requestBody": "{\"image\": \"iV"
, \"points\": [[500.02734375, 315.7109375], [479.892578125, 347.9267578125], [570.900390625, 408.330078125], [516.134765625, 426.8544921875], [418.6826171875, 375.3095703125], [654.6611328125, 322.9599609375], [544.3232421875, 374.50390625]]}", "responseStatus": 500, "responseBody": "{\"error\": \"Failed to invoke function\"}", "responseTime": "30.147497479s"}

‘docker logs nuclio-nuclio-openvino-dextr’

21.01.04 07:36:23.918 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
21.01.04 07:36:23.918            processor.http (D) Creating worker pool {"num": 2}
21.01.04 07:36:23.919 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-bvpcatqr7t3i9donl7ug.sock"}
21.01.04 07:36:23.919 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-bvpcatqr7t3i9donl7v0.sock"}
21.01.04 07:36:23.919 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
21.01.04 07:36:23.919 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
21.01.04 07:36:23.919 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
21.01.04 07:36:23.919 sor.http.w1.python.logger (D) Using Python executable {"path": "/opt/nuclio/common/python3"}
21.01.04 07:36:23.919 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
21.01.04 07:36:23.919 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio"}
21.01.04 07:36:23.919 sor.http.w0.python.logger (D) Using Python executable {"path": "/opt/nuclio/common/python3"}
21.01.04 07:36:23.919 sor.http.w1.python.logger (D) Running wrapper {"command": "/opt/nuclio/common/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-bvpcatqr7t3i9donl7ug.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
21.01.04 07:36:23.919 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio"}
21.01.04 07:36:23.919 sor.http.w0.python.logger (D) Running wrapper {"command": "/opt/nuclio/common/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-bvpcatqr7t3i9donl7v0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
[setupvars.sh] OpenVINO environment initialized
[setupvars.sh] OpenVINO environment initialized
21.01.04 07:36:24.257 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 24}
21.01.04 07:36:24.257 sor.http.w0.python.logger (D) Waiting for start
l{"datetime": "2021-01-04 07:36:24,257", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
21.01.04 07:36:24.275 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 23}
21.01.04 07:36:24.275 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2021-01-04 07:36:24,275", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2021-01-04 07:36:26,666", "level": "info", "message": "Init context...100%", "with": {"worker_id": "0"}}
l{"datetime": "2021-01-04 07:36:26,666", "level": "info", "message": "Replacing logger output", "with": {"handler_name": "default", "worker_id": "0"}}
21.01.04 07:36:26.666 sor.http.w0.python.logger (D) Started
l{"datetime": "2021-01-04 07:36:26,673", "level": "info", "message": "Init context...100%", "with": {"worker_id": "1"}}
l{"datetime": "2021-01-04 07:36:26,673", "level": "info", "message": "Replacing logger output", "with": {"handler_name": "default", "worker_id": "1"}}
21.01.04 07:36:26.674 sor.http.w1.python.logger (D) Started
21.01.04 07:36:26.674                 processor (I) Starting event timeout watcher {"timeout": "30s"}
21.01.04 07:36:26.674 .webadmin.server.triggers (D) Registered custom route {"pattern": "/{id}/stats", "method": "GET"}
21.01.04 07:36:26.674 processor.webadmin.server (D) Registered resource {"name": "triggers"}
21.01.04 07:36:26.674                 processor (W) No metric sinks configured, metrics will not be published
21.01.04 07:36:26.674                 processor (D) Starting triggers {"triggers": [{"ID":"myHttpTrigger","Logger":{},"WorkerAllocator":{},"Class":"sync","Kind":"http","Name":"myHttpTrigger","Statistics":{"EventsHandledSuccessTotal":0,"EventsHandledFailureTotal":0,"WorkerAllocatorStatistics":{"WorkerAllocationCount":0,"WorkerAllocationSuccessImmediateTotal":0,"WorkerAllocationSuccessAfterWaitTotal":0,"WorkerAllocationTimeoutTotal":0,"WorkerAllocationWaitDurationMilliSecondsSum":0,"WorkerAllocationWorkersAvailablePercentage":0}},"Namespace":"nuclio","FunctionName":"openvino-dextr"}]}
21.01.04 07:36:26.674            processor.http (I) Starting {"listenAddress": ":8080", "readBufferSize": 16384, "maxRequestBodySize": 33554432, "cors": null}
21.01.04 07:36:26.674 processor.webadmin.server (I) Listening {"listenAddress": ":8081"}
21.01.04 07:36:26.674                 processor (D) Processor started

‘docker logs cvat_proxy’

10.162.187.190 - - [04/Jan/2021:07:37:32 +0000] "GET /api/v1/jobs/2/reviews HTTP/1.1" 200 2 "http://:38383/tasks/2/jobs/2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"
10.162.187.190 - - [04/Jan/2021:07:38:09 +0000] "POST /api/v1/lambda/functions/openvino-dextr HTTP/1.1" 500 94 "http://38383/tasks/2/jobs/2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"
10.162.187.190 - - [04/Jan/2021:07:39:40 +0000] "POST /api/v1/lambda/functions/openvino-dextr HTTP/1.1" 500 94 "http:///tasks/2/jobs/2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"
10.162.187.190 - - [04/Jan/2021:07:43:43 +0000] "POST /api/v1/lambda/functions/openvino-dextr HTTP/1.1" 500 94 "h/tasks/2/jobs/2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"
10.162.187.190 - - [04/Jan/2021:07:46:56 +0000] "POST /api/v1/lambda/functions/openvino-dextr HTTP/1.1" 500 94 "http://jobs/2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"

and in nuclio dashboard, when i test openvino-dextr, it says as well Status: 500 Internal Server Error with { “error”: “Failed to invoke function” } so i tried invoking openvino-dextr in terminal.

cvat@:~/cvat/serverless$ nuctl invoke openvino-dextr
21.01.04 16:25:39.773    nuctl.platform.invoker (I) Executing function {"method": "GET", "url": "http://:49155", "headers": {"Content-Type":["text/plain"],"X-Nuclio-Log-Level":["info"],"X-Nuclio-Target":["openvino-dextr"]}}
21.01.04 16:25:39.779    nuctl.platform.invoker (I) Got response {"status": "500 Internal Server Error"}
21.01.04 16:25:39.779                     nuctl (I) >>> Start of function logs
21.01.04 16:25:39.779            openvino-dextr (I) call handler {"time": 1609748739778.5774, "worker_id": "0"}
21.01.04 16:25:39.779            openvino-dextr (E) Exception caught in handler {"exc": "'NoneType' object is not subscriptable", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 114, in serve_requests\n    self._handle_event(event)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 262, in _handle_event\n    entrypoint_output = self._entrypoint(self._context, event)\n  File \"/opt/nuclio/main.py\", line 18, in handler\n    points = data[\"points\"]\nTypeError: 'NoneType' object is not subscriptable\n", "worker_id": "0", "time": 1609748739779.3408}
21.01.04 16:25:39.779                     nuctl (I) <<< End of function logs

> Response headers:
Server = nuclio
Date = Mon, 04 Jan 2021 08:25:39 GMT
Content-Type = text/plain
Content-Length = 465

> Response body:
Exception caught in handler - "'NoneType' object is not subscriptable": Traceback (most recent call last):
  File "/opt/nuclio/_nuclio_wrapper.py", line 114, in serve_requests
    self._handle_event(event)
  File "/opt/nuclio/_nuclio_wrapper.py", line 262, in _handle_event
    entrypoint_output = self._entrypoint(self._context, event)
  File "/opt/nuclio/main.py", line 18, in handler
    points = data["points"]
TypeError: 'NoneType' object is not subscriptable

Issue Analytics

State:
Created 3 years ago
Comments:11 (1 by maintainers)

Top GitHub Comments

4reactions

yaochenglouiscommented, Jan 7, 2021

Final word: for those who use ufw, first find the port that your nuclio function listens to, and ufw allow port. that worked for me.

0reactions

yaochenglouiscommented, Jan 7, 2021

after disabling ufw, things went better. but i am having a new bug…

Top Results From Across the Web

Nuclio related 500 server error · Issue #2641 · opencv/cvat

I have been deleting functions in nuctl and re deploying them, which gives me the same 500 error.

opencv-cvat/public - Gitter

HTTPError: 500 Server Error: Internal server error for url: http://nuclio:8070/api/function_invocations for openvino models in CVAT (for example YOLO v3).

NUCLEO-F746ZG - STMicroelectronics

NUCLEO -F746ZG - STM32 Nucleo-144 development board with STM32F746ZG MCU, supports Arduino, ST Zio and morpho connectivity, NUCLEO-F746ZG, ...

functionconfig - Go Packages

Constants; func DecodeSecretKey(secretKey string) (string, error) ... according to the "x-nuclio-stream-no-ack" header ExplicitAckModeEnable ExplicitAckMode ...

kafka-cluster: Kafka Trigger - Nuclio

For stateless functions, this is not a problem. ... kafka-channel-buffer-size ) — The number of events to buffer in internal and external channels....