We couldn't kill mongod process, and tried a lot.

Here are the details - please ask me if there’s more information that’s required, thanks:

With libraries:

embed.mongo: 2.1.1
embed.process: 2.0.5

On Ubuntu server:

uname -a
Linux etb3 4.15.0-1035-azure #36~16.04.1-Ubuntu SMP Fri Nov 30 15:25:49 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

with ulimits:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 64083
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 64083
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Build log:

[mongod output] 2018-12-07T11:25:36.639+0000 I NETWORK  [conn2] end connection 127.0.0.1:58080 (0 connections now open)
[mongod output] 2018-12-07 11:25:36,650 INFO    i.c.networkmap.storage.EmbeddedMongo - manual shutdown in progress
2018-12-07T11:25:36.651+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:58104 #3 (1 connection now open)
[mongod output] 2018-12-07 11:25:36,655 ERROR   d.f.embed.mongo.runtime.Mongod - Got some response, should be an error message
2018-12-07T11:25:36.655+0000 I NETWORK  [conn3] end connection 127.0.0.1:58104 (0 connections now open)
[mongod output] 2018-12-07 11:25:36,755 WARN    d.f.embed.mongo.AbstractMongoProcess - could not stop mongod with db command, try next

2018-12-07T11:25:36.758+0000 I CONTROL  [signalProcessingThread] got signal 2 (Interrupt), will terminate after current cmd ends
[mongod output] 2018-12-07T11:25:36.758+0000 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
[mongod output] 2018-12-07T11:25:36.758+0000 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-44059.sock
[mongod output] 2018-12-07T11:25:36.758+0000 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
[mongod output] 2018-12-07T11:25:36.760+0000 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
[mongod output] Exception in thread "vert.x-eventloop-thread-8" java.lang.IllegalStateException: Couldn't kill mongod process!

----------------------------------------------------
Something bad happend. We couldn't kill mongod process, and tried a lot.
If you want this problem solved you can help us if you open a new issue.

Follow this link:
https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo/issues

Thank you:)
----------------------------------------------------

[shollander]

I am having the exact same issue as @ptempes.

We configure embedmongo to use a replicaset (single server) in order to test transactional methods with mongodb 4.0. We are doing it similar to this https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo/issues/257#issuecomment-407770356

Almost every time we encounter this error message on shutdown.

---

Something bad happend. We couldn’t kill mongod process, and tried a lot. If you want this problem solved you can help us if you open a new issue. …

The extractmongod process does go away almost immediately after the error message appears, so perhaps this is a timing issue. I can get the error message to go away by increasing the stopTimeoutInMillis() setting on MongoConfigBuilder. Increasing it by one second (six seconds instead of the default of five seconds) is enough for us.

I am wondering though, why does it take so long to shutdown when using the single node replica set? Perhaps there is something else that needs to be done in order to shutdown the replicaset cleanly?

Unfortunately, the ProcessControl class closes the output streams before issuing the shutdown command (ProcessControl.closeIoAndDestroy()) so I can’t even debug this to see where it is hanging.

[shollander]

Doing a little research into the mongodb wire format, I figured out how to add the timeoutSecs: 0 option. Changing SHUTDOWN_COMMAND to the following fixes the issue for me:

 	static final byte[] SHUTDOWN_COMMAND = { 0x58, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
 			(byte) 0xD4, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x61, 0x64, 0x6D, 0x69, 0x6E, 0x2E, 0x24, 0x63, 0x6D, 
 			0x64, 0x00, 0x00, 0x00, 0x00, 0x00, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF, 0x2C, 0x00, 0x00, 
 			0x00, 0x10, 0x73, 0x68, 0x75, 0x74, 0x64, 0x6F, 0x77, 0x6E, 0x00, 0x01, 0x00, 0x00, 0x00, 0x08, 0x66, 0x6F, 
 			0x72, 0x63, 0x65, 0x00, 0x01, 0x10, 0x74, 0x69, 0x6D, 0x65, 0x6F, 0x75, 0x74, 0x53, 0x65, 0x63, 0x73, 0x00, 
                        0x00, 0x00, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00 };