Error 507 when predicting

Hello,

I have a rather large model that I need to use for prediction. When I make the request, I receive an error 507 and a message stating that the worker has died on the server side

2020-06-10 11:30:41,286 [DEBUG] W-9000-model_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
	at java.base/java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:513)
	at java.base/java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:675)
	at org.pytorch.serve.wlm.Model.pollBatch(Model.java:155)
	at org.pytorch.serve.wlm.BatchAggregator.getRequest(BatchAggregator.java:33)
	at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:123)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

I suspect this is somehow related with the JVM memory. I am therefore using a config.properties file with the following entry

vmargs=-Xmx128g (of course the model needs much less than 128gb)

I am not using the GPU for this prediction as it is just a test. I am also running within a docker container.

How can I debug this? is there a way to get better error messages and a stack trace (for example finding out if pytorch has issues allocating the model or similar problems?)

[faustomilletari]

Thank you for your help. I will try on a bigger machine in the cloud with a GPU. I suspect that since the Docker container cannot swap, and since there are multiple workers which might holt their own copy of stuff, the memory consumption is so high that the system runs legitimately out of memory.

I will update you within 1 hour.

[harshbafna]

@faustomilletari : How many workers are you using for your model and can you share the output of top commands for different scenarios like after starting the workers, running the inference with smaller input size and larger input size.

Please also share the different log files from the log folder, which is by default generate on the path from where you start TorchServe