Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Segfault on HEAD request
See original GitHub issueDescribe the bug
Sending any HEAD request causes a segfault.
To Reproduce Steps to reproduce the behavior:
- Use any Ambassador version more recent than 0.34.1
docker run -it -p 8080:80 --name=ambassador --rm quay.io/datawire/ambassador:0.35.3 --demo
- Send a HEAD request
curl -I localhost:8080/qotm/
- Segfault occurs
Expected behavior It should make a HEAD request, and not crash.
Versions (please complete the following information):
- Ambassador: 0.34.2 - 0.35.3
- Kubernetes environment: Both GKE and local with quay.io docker image
- Version: irrelevant
Additional context
The following stack trace is from the quay.io demo image.
[2018-07-20 13:18:23.099][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Segmentation fault, suspect faulting address 0x0
[2018-07-20 13:18:23.100][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<24> obj</usr/local/bin/envoy> (If unsymbolized, use tools/stack_decode.py):
[2018-07-20 13:18:23.100][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #0 0x65e003 (unknown)
[2018-07-20 13:18:23.100][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #1 0xb06ae6 (unknown)
[2018-07-20 13:18:23.101][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #2 0xb06b7b (unknown)
[2018-07-20 13:18:23.101][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #3 0xb06007 (unknown)
[2018-07-20 13:18:23.101][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #4 0xb780f4 (unknown)
[2018-07-20 13:18:23.101][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #5 0xb7987c (unknown)
[2018-07-20 13:18:23.101][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #6 0xb16ec1 (unknown)
[2018-07-20 13:18:23.102][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #7 0xb147f5 (unknown)
[2018-07-20 13:18:23.102][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #8 0xb16ec1 (unknown)
[2018-07-20 13:18:23.102][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #9 0xc7eaac (unknown)
[2018-07-20 13:18:23.102][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #10 0xc7bd76 (unknown)
[2018-07-20 13:18:23.102][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #11 0xc7bd9a (unknown)
[2018-07-20 13:18:23.102][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #12 0xc87dbf (unknown)
[2018-07-20 13:18:23.103][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #13 0xc7c8c8 (unknown)
[2018-07-20 13:18:23.103][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #14 0xc7c79c (unknown)
[2018-07-20 13:18:23.103][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #15 0xb872d0 (unknown)
[2018-07-20 13:18:23.103][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #16 0xb889ed (unknown)
[2018-07-20 13:18:23.103][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #17 0x91faca (unknown)
[2018-07-20 13:18:23.103][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #18 0x91fb61 (unknown)
[2018-07-20 13:18:23.104][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #19 0x917051 (unknown)
[2018-07-20 13:18:23.104][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #20 0x917f18 (unknown)
[2018-07-20 13:18:23.104][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #21 0x917db0 (unknown)
[2018-07-20 13:18:23.104][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #22 0x915786 (unknown)
[2018-07-20 13:18:23.104][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #23 0x919595 (unknown)
[2018-07-20 13:18:23.104][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #24 0x90caf7 (unknown)
[2018-07-20 13:18:23.105][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #25 0x90bb53 (unknown)
[2018-07-20 13:18:23.105][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #26 0x90bb83 (unknown)
[2018-07-20 13:18:23.105][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #27 0xe940b1 (unknown)
[2018-07-20 13:18:23.105][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #28 0xe9480e (unknown)
[2018-07-20 13:18:23.105][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #29 0x90679d (unknown)
[2018-07-20 13:18:23.106][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #30 0x8f70f8 (unknown)
[2018-07-20 13:18:23.106][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #31 0x8f6c1d (unknown)
[2018-07-20 13:18:23.106][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #32 0x8f77ee (unknown)
[2018-07-20 13:18:23.106][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #33 0x4e4f7b (unknown)
[2018-07-20 13:18:23.106][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #34 0xeed2ed (unknown)
[2018-07-20 13:18:23.107][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #35 0xeed312 (unknown)
[2018-07-20 13:18:23.107][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<24> obj</usr/glibc-compat/lib/libpthread.so.0>
[2018-07-20 13:18:23.107][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #36 0x7f920e932586 (unknown)
[2018-07-20 13:18:23.107][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<24> obj</usr/glibc-compat/lib/libc.so.6>
[2018-07-20 13:18:23.107][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<24> #37 0x7f920e2d297e (unknown)
[2018-07-20 13:18:23.107][24][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 24
got SIGCHLD
PID=16 was killed with signal=11
Due to abnormal exit, force killing all child processes and exiting
exiting due to lack of child processes
AMBASSADOR: envoy exited with status 1
Here's the envoy.json we were trying to run with:
{
"listeners": [
{
"address": "tcp://0.0.0.0:80",
"filters": [
{
"type": "read",
"name": "http_connection_manager",
"config": {"codec_type": "auto",
"stat_prefix": "ingress_http",
"access_log": [
{
"format": "ACCESS [%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n",
"path": "/dev/fd/1"
}
],
"route_config": {
"virtual_hosts": [
{
"name": "backend",
"domains": ["*"],"routes": [
{
"timeout_ms": 3000,"prefix": "/ambassador/v0/check_ready","prefix_rewrite": "/ambassador/v0/check_ready","weighted_clusters": {
"clusters": [
{ "name": "cluster_127_0_0_1_8877", "weight": 100.0 }
]
}
}
,
{
"timeout_ms": 3000,"prefix": "/ambassador/v0/check_alive","prefix_rewrite": "/ambassador/v0/check_alive","weighted_clusters": {
"clusters": [
{ "name": "cluster_127_0_0_1_8877", "weight": 100.0 }
]
}
}
,
{
"timeout_ms": 3000,"prefix": "/ambassador/v0/","prefix_rewrite": "/ambassador/v0/","weighted_clusters": {
"clusters": [
{ "name": "cluster_127_0_0_1_8877", "weight": 100.0 }
]
}
}
,
{
"timeout_ms": 3000,"prefix": "/ambassador/","prefix_rewrite": "/ambassador/","weighted_clusters": {
"clusters": [
{ "name": "cluster_127_0_0_1_8877", "weight": 100.0 }
]
}
}
,
{
"timeout_ms": 3000,"prefix": "/httpbin/","prefix_rewrite": "/","host_rewrite": "httpbin.org","weighted_clusters": {
"clusters": [
{ "name": "cluster_httpbin_org_80", "weight": 100.0 }
]
}
}
,
{
"timeout_ms": 3000,"prefix": "/cqrs/","headers": [{"name": ":method", "regex": false, "value": "PUT"}],"prefix_rewrite": "/qotm/quote/","weighted_clusters": {
"clusters": [
{ "name": "cluster_demo_getambassador_io", "weight": 100.0 }
]
}
}
,
{
"timeout_ms": 3000,"prefix": "/cqrs/","headers": [{"name": ":method", "regex": false, "value": "GET"}],"prefix_rewrite": "/qotm/quote/","weighted_clusters": {
"clusters": [
{ "name": "cluster_demo_getambassador_io", "weight": 100.0 }
]
}
}
,
{
"timeout_ms": 3000,"prefix": "/qotm/","prefix_rewrite": "/qotm/","weighted_clusters": {
"clusters": [
{ "name": "cluster_demo_getambassador_io", "weight": 100.0 }
]
}
}
]
}
]
},
"filters": [
{"type": "decoder",
"name": "extauth",
"config": {"allowed_headers": ["x-extauth-required", "x-authenticated-as", "x-qotm-session"], "cluster": "cluster_ext_auth", "path_prefix": "/auth/v0", "timeout_ms": 5000}
},{
"name": "cors",
"config": {}
},{"type": "decoder",
"name": "router",
"config": {}
}
]
}
}
]
}
],
"admin": {
"address": "tcp://127.0.0.1:8001",
"access_log_path": "/tmp/admin_access_log"
},
"cluster_manager": {
"clusters": [
{
"name": "cluster_127_0_0_1_8877",
"connect_timeout_ms": 3000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://127.0.0.1:8877"
}
]},
{
"name": "cluster_demo_getambassador_io",
"connect_timeout_ms": 3000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://demo.getambassador.io:80"
}
]},
{
"name": "cluster_ext_auth",
"connect_timeout_ms": 3000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://demo.getambassador.io:80"
}
]},
{
"name": "cluster_httpbin_org_80",
"connect_timeout_ms": 3000,
"type": "strict_dns",
"lb_type": "round_robin",
"hosts": [
{
"url": "tcp://httpbin.org:80"
}
]}
]
},
"statsd_udp_ip_address": "127.0.0.1:8125",
"stats_flush_interval_ms": 1000
}AMBASSADOR: shutting down (1)
[2018-07-20 13:18:23 +0000] [13] [INFO] Handling signal: hup
[2018-07-20 13:18:23 +0000] [13] [INFO] Hang up: Master
Issue Analytics
- State:
- Created 5 years ago
- Comments:5 (4 by maintainers)
Top Results From Across the Web
Segfault on HEAD request · Issue #646 - GitHub
Describe the bug Sending any HEAD request causes a segfault. To Reproduce Steps to reproduce the behavior: Use any Ambassador version more ...
Read more >PHP :: Bug #58836 :: segfault on HEAD requests
Description: ------------ When using memcache-2.2.5, and Wordpress Batcache extension, any HEAD request causes PHP to crash and apache to segfault.
Read more >C - segmentation fault when accessing a pointer in header file
What I need is to do the dynamic allocation in func2.h , so I don't want any calculation involving veins[][] in main.c before...
Read more >Segmentation fault on 3.0.22 and head when doing a request ...
It appears to be when FreeRadius attempts a status_check after getting no response from a server when it's attempting to proxy a request, ......
Read more >Segfault errors on a search head - Splunk Community
Hello, We encounter this type of message in the Splunk Serch Head, which causes the restart of the splunk service and the delays...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Possibly related to https://github.com/datawire/ambassador/issues/586
Fixed in #659, released in https://github.com/datawire/ambassador/releases/tag/0.36.0