m3db docker image crashes on rancher desktop, runs fine on other docker environments

Rancher Desktop Version

0.7.1

Rancher Desktop K8s Version

1.21.0

What operating system are you using?

macOS

Operating System / Build Version

MacOS Monterey

What CPU architecture are you using?

x64

Linux only: what package format did you use to install Rancher Desktop?

No response

Windows User Only

No response

Actual Behavior

official m3db docker image crashes during execution with a SIGILL on rancher desktop, but works fine on docker desktop, minikube/vmware fusion and other docker environments on Mac.

See full output here: OUTPUT.txt

Steps to Reproduce

Run the docker image:

docker run -d -p 7201:7201 -p 7203:7203 --name m3db -v $(pwd)/m3db_data:/var/lib/m3db quay.io/m3db/m3dbnode:v1.4.2

The first two steps only need to be done once to configure the service.

Invoke REST API to create a namespace in m3db (this is async and takes 1-2 mins to complete):

curl --location --request POST 'http://localhost:7201/api/v1/database/create' \
--header 'Content-Type: application/json' \
--data-raw '{
    "namespaceName": "default_unaggregated",
    "retentionTime": "24h",
    "type": "local"
}

Wait approx 1-2 mins for namespace to be created before next step(ready namespace) will succeed:

curl --location --request POST 'http://localhost:7201/api/v1/services/m3db/namespace/ready' \
--header 'Content-Type: application/json' \
--data-raw '{
    "name": "default_unaggregated"
}'

Crash occurs on next step: insert a metric into the namespace:

curl --location --request POST 'http://localhost:7201/api/v1/json/write' \
--header 'Content-Type: application/json' \
--data-raw '{
    "tags": {
        "__name__": "third_avenue",
        "city": "new_york",
        "checkout": "1"
    },
    "timestamp": "1642741531",
    "value": 3347.26
}'

Result

SIGILL: illegal instruction PC=0xd01bf0 m=5 sigcode=2

goroutine 14333 [running]: github.com/m3db/m3/src/x/sync.getCore(0xc00fc88d80, 0x3, 0x3, 0x1, 0x230eba0, 0xc0a4637500, 0x22bc140, 0xc0a4637500, 0x22bc120, 0xc0a4637500, …) /go/src/github.com/m3db/m3/src/x/sync/cpu_linux_amd64.s:9 fp=0xc00609a218 sp=0xc00609a210 pc=0xd01bf0 github.com/m3db/m3/src/x/sync.CPUCore(…) /go/src/github.com/m3db/m3/src/x/sync/index_cpu.go:72

Expected Behavior

Docker image should run correctly

Additional Information

No response

[jandubois]

This looks like a frequent issue with m3db in virtual machines, e.g. https://github.com/m3db/m3/issues/3105, https://github.com/m3db/m3/issues/3659, https://github.com/m3db/m3/issues/3827.

The answer is always: Make sure you run on a machine that supports the rdtscp instruction!

We are running qemu with -cpu host, but it looks like qemu is disabling rdtscp even on some CPUs that would support it. Not quite sure where to go from here; I’ve filed a Lima issue yesterday to make the qemu cpu setting configurable: https://github.com/lima-vm/lima/issues/592. But I’m not sure if this will be sufficient.

Sounds like @evertonlperes can reproduce the problem, but just for reference, @nick-stephen and @evertonlperes, what is the exact model of your CPU, e.g.

$ sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz

[moio]

I think we will move Rancher Desktop to the latest qemu release from homebrew for the next release, so it should all work out, wouldn’t it?

Yes, I do not expect any problem from my perspective. Thanks!