Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Linkerd directs traffic to pods of old service version after dTAB change

See original GitHub issue

Issue Type:

  • Bug report
  • Feature request

What happened: Linkerd deployed as daemonset and services deployed use grpc. We had 100% traffic going to search-1-0-3 service. Changed the dtabs so that 100% goes to search-1-0-4 service. Linkerd pods were still sending traffic to search-1-0-3 even after 1.5 hours. Deleted linkerd pods and the new ones that were spawned started sending traffic to search-1-0-4.

What you expected to happen: Expected Linkerd to send new traffic to search-1-0-4 without having to restart pods.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?: Interpreter: io.l5d.namerd metrics.json of bad pod:

{
   "jvm/start_time": 1.52452268E12,
   "jvm/application_time_millis": 7.6481466E8,
   "jvm/classes/total_loaded": 8901,
   "jvm/classes/current_loaded": 8867,
   "jvm/classes/total_unloaded": 34,
   "jvm/postGC/Par_Survivor_Space/max": 3.5782656E7,
   "jvm/postGC/Par_Survivor_Space/used": 947448,
   "jvm/postGC/CMS_Old_Gen/max": 7.1584973E8,
   "jvm/postGC/CMS_Old_Gen/used": 5.9990424E7,
   "jvm/postGC/Par_Eden_Space/max": 2.86326784E8,
   "jvm/postGC/Par_Eden_Space/used": 0,
   "jvm/postGC/used": 6.0937872E7,
   "jvm/nonheap/committed": 7.1032832E7,
   "jvm/nonheap/max": -1,
   "jvm/nonheap/used": 6.5350248E7,
   "jvm/tenuring_threshold": 2,
   "jvm/thread/daemon_count": 114,
   "jvm/thread/count": 115,
   "jvm/thread/peak_count": 127,
   "jvm/mem/postGC/Par_Survivor_Space/max": 3.5782656E7,
   "jvm/mem/postGC/Par_Survivor_Space/used": 947448,
   "jvm/mem/postGC/CMS_Old_Gen/max": 7.1584973E8,
   "jvm/mem/postGC/CMS_Old_Gen/used": 5.9990424E7,
   "jvm/mem/postGC/Par_Eden_Space/max": 2.86326784E8,
   "jvm/mem/postGC/Par_Eden_Space/used": 0,
   "jvm/mem/postGC/used": 6.0937872E7,
   "jvm/mem/metaspace/max_capacity": 1.1051991E9,
   "jvm/mem/buffer/direct/max": 0,
   "jvm/mem/buffer/direct/count": 1,
   "jvm/mem/buffer/direct/used": 1,
   "jvm/mem/buffer/mapped/max": 0,
   "jvm/mem/buffer/mapped/count": 0,
   "jvm/mem/buffer/mapped/used": 0,
   "jvm/mem/allocations/eden/bytes": 1.48179747E11,
   "jvm/mem/current/used": 1.35877824E8,
   "jvm/mem/current/CMS_Old_Gen/max": 7.1584973E8,
   "jvm/mem/current/CMS_Old_Gen/used": 6.2531088E7,
   "jvm/mem/current/Metaspace/max": -1,
   "jvm/mem/current/Metaspace/used": 5.1663952E7,
   "jvm/mem/current/Par_Eden_Space/max": 2.86326784E8,
   "jvm/mem/current/Par_Eden_Space/used": 7049032,
   "jvm/mem/current/Par_Survivor_Space/max": 3.5782656E7,
   "jvm/mem/current/Par_Survivor_Space/used": 947448,
   "jvm/mem/current/Compressed_Class_Space/max": 1.07374182E9,
   "jvm/mem/current/Compressed_Class_Space/used": 8234392,
   "jvm/mem/current/Code_Cache/max": 5.0331648E7,
   "jvm/mem/current/Code_Cache/used": 5451904,
   "jvm/num_cpus": 48,
   "jvm/gc/msec": 8623898,
   "jvm/gc/eden/pause_msec.count": 6,
   "jvm/gc/eden/pause_msec.max": 1478,
   "jvm/gc/eden/pause_msec.min": 11,
   "jvm/gc/eden/pause_msec.p50": 195,
   "jvm/gc/eden/pause_msec.p90": 295,
   "jvm/gc/eden/pause_msec.p95": 1478,
   "jvm/gc/eden/pause_msec.p99": 1478,
   "jvm/gc/eden/pause_msec.p9990": 1478,
   "jvm/gc/eden/pause_msec.p9999": 1478,
   "jvm/gc/eden/pause_msec.sum": 2374,
   "jvm/gc/eden/pause_msec.avg": 395.66666,
   "jvm/gc/ParNew/msec": 8438370,
   "jvm/gc/ParNew/cycles": 16895,
   "jvm/gc/ConcurrentMarkSweep/msec": 185528,
   "jvm/gc/ConcurrentMarkSweep/cycles": 287,
   "jvm/gc/cycles": 17182,
   "jvm/fd_limit": 1048576,
   "jvm/compilation/time_msec": 144755,
   "jvm/uptime": 7.7337024E8,
   "jvm/safepoint/sync_time_millis": 4073,
   "jvm/safepoint/total_time_millis": 8552999,
   "jvm/safepoint/count": 48023,
   "jvm/heap/committed": 1.10014464E8,
   "jvm/heap/max": 1.03795917E9,
   "jvm/heap/used": 7.054596E7,
   "jvm/fd_count": 839,
   "rt/grpc-incoming/server/0.0.0.0/4140/sent_bytes": 4664030133,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_received_bytes.count": 3,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_received_bytes.max": 20242,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_received_bytes.min": 20042,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_received_bytes.p50": 20042,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_received_bytes.p90": 20242,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_received_bytes.p95": 20242,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_received_bytes.p99": 20242,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_received_bytes.p9990": 20242,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_received_bytes.p9999": 20242,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_received_bytes.sum": 60228,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_received_bytes.avg": 20076,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_duration.count": 3,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_duration.max": 3335179,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_duration.min": 1987960,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_duration.p50": 2706265,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_duration.p90": 3335179,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_duration.p95": 3335179,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_duration.p99": 3335179,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_duration.p9990": 3335179,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_duration.p9999": 3335179,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_duration.sum": 8015068,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_duration.avg": 2671689.2,
   "rt/grpc-incoming/server/0.0.0.0/4140/connects": 14427,
   "rt/grpc-incoming/server/0.0.0.0/4140/success": 1026994,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/stream_failures": 8,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/data_bytes.count": 604,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/data_bytes.max": 10087,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/data_bytes.min": 3729,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/data_bytes.p50": 9132,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/data_bytes.p90": 9694,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/data_bytes.p95": 9790,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/data_bytes.p99": 9987,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/data_bytes.p9990": 10087,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/data_bytes.p9999": 10087,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/data_bytes.sum": 5332937,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/data_bytes.avg": 8829.366,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/failures": 8,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/failures/com.twitter.finagle.buoyant.h2.Reset$NoError$": 8,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/stream_success": 1027004,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/stream_duration_ms.count": 604,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/stream_duration_ms.max": 1617,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/stream_duration_ms.min": 14,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/stream_duration_ms.p50": 123,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/stream_duration_ms.p90": 402,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/stream_duration_ms.p95": 453,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/stream_duration_ms.p99": 790,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/stream_duration_ms.p9990": 1554,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/stream_duration_ms.p9999": 1617,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/stream_duration_ms.sum": 113634,
   "rt/grpc-incoming/server/0.0.0.0/4140/request/stream/stream_duration_ms.avg": 188.13576,
   "rt/grpc-incoming/server/0.0.0.0/4140/request_latency_ms.count": 604,
   "rt/grpc-incoming/server/0.0.0.0/4140/request_latency_ms.max": 1617,
   "rt/grpc-incoming/server/0.0.0.0/4140/request_latency_ms.min": 14,
   "rt/grpc-incoming/server/0.0.0.0/4140/request_latency_ms.p50": 123,
   "rt/grpc-incoming/server/0.0.0.0/4140/request_latency_ms.p90": 402,
   "rt/grpc-incoming/server/0.0.0.0/4140/request_latency_ms.p95": 453,
   "rt/grpc-incoming/server/0.0.0.0/4140/request_latency_ms.p99": 790,
   "rt/grpc-incoming/server/0.0.0.0/4140/request_latency_ms.p9990": 1554,
   "rt/grpc-incoming/server/0.0.0.0/4140/request_latency_ms.p9999": 1617,
   "rt/grpc-incoming/server/0.0.0.0/4140/request_latency_ms.sum": 113635,
   "rt/grpc-incoming/server/0.0.0.0/4140/request_latency_ms.avg": 188.13742,
   "rt/grpc-incoming/server/0.0.0.0/4140/received_bytes": 9573129612,
   "rt/grpc-incoming/server/0.0.0.0/4140/read_timeout": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/write_timeout": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_sent_bytes.count": 3,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_sent_bytes.max": 18144,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_sent_bytes.min": 11366,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_sent_bytes.p50": 11480,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_sent_bytes.p90": 18144,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_sent_bytes.p95": 18144,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_sent_bytes.p99": 18144,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_sent_bytes.p9990": 18144,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_sent_bytes.p9999": 18144,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_sent_bytes.sum": 41064,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_sent_bytes.avg": 13688,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_requests.count": 3,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_requests.max": 7,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_requests.min": 5,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_requests.p50": 5,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_requests.p90": 7,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_requests.p95": 7,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_requests.p99": 7,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_requests.p9990": 7,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_requests.p9999": 7,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_requests.sum": 17,
   "rt/grpc-incoming/server/0.0.0.0/4140/connection_requests.avg": 5.6666665,
   "rt/grpc-incoming/server/0.0.0.0/4140/thread_usage/requests/mean": 6.516129,
   "rt/grpc-incoming/server/0.0.0.0/4140/thread_usage/requests/relative_stddev": 0.5357067,
   "rt/grpc-incoming/server/0.0.0.0/4140/thread_usage/requests/stddev": 3.4907339,
   "rt/grpc-incoming/server/0.0.0.0/4140/transit_latency_ms.count": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/socket_unwritable_ms": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/stream_failures": 9,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/data_bytes.count": 604,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/data_bytes.max": 7559,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/data_bytes.min": 5,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/data_bytes.p50": 5664,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/data_bytes.p90": 6013,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/data_bytes.p95": 6195,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/data_bytes.p99": 6576,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/data_bytes.p9990": 6911,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/data_bytes.p9999": 7559,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/data_bytes.sum": 2702738,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/data_bytes.avg": 4474.732,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/failures": 9,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/failures/com.twitter.finagle.buoyant.h2.Reset$Cancel$": 8,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/failures/com.twitter.finagle.buoyant.h2.StreamError$Remote": 1,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/stream_success": 1026994,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/stream_duration_ms.count": 605,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/stream_duration_ms.max": 272,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/stream_duration_ms.min": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/stream_duration_ms.p50": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/stream_duration_ms.p90": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/stream_duration_ms.p95": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/stream_duration_ms.p99": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/stream_duration_ms.p9990": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/stream_duration_ms.p9999": 272,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/stream_duration_ms.sum": 272,
   "rt/grpc-incoming/server/0.0.0.0/4140/response/stream/stream_duration_ms.avg": 0.44958678,
   "rt/grpc-incoming/server/0.0.0.0/4140/closes": 13901,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/stream_failures": 8,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/total_latency_ms.count": 605,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/total_latency_ms.max": 1617,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/total_latency_ms.min": 14,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/total_latency_ms.p50": 124,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/total_latency_ms.p90": 406,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/total_latency_ms.p95": 457,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/total_latency_ms.p99": 790,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/total_latency_ms.p9990": 1554,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/total_latency_ms.p9999": 1617,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/total_latency_ms.sum": 114359,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/total_latency_ms.avg": 189.02315,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/data_bytes.count": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/data/frames": 1027001,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/data/bytes.count": 605,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/data/bytes.max": 7559,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/data/bytes.min": 5,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/data/bytes.p50": 5664,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/data/bytes.p90": 6073,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/data/bytes.p95": 6195,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/data/bytes.p99": 6576,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/data/bytes.p9990": 6911,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/data/bytes.p9999": 7559,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/data/bytes.sum": 2708795,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/data/bytes.avg": 4477.347,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/reset": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/local/trailers": 1026994,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/data/frames": 1027096,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/data/bytes.count": 606,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/data/bytes.max": 10087,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/data/bytes.min": 3729,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/data/bytes.p50": 9132,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/data/bytes.p90": 9694,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/data/bytes.p95": 9790,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/data/bytes.p99": 9987,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/data/bytes.p9990": 10087,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/data/bytes.p9999": 10087,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/data/bytes.sum": 5355911,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/data/bytes.avg": 8838.137,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/reset": 76,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/remote/trailers": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/failures": 8,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/failures/com.twitter.finagle.buoyant.h2.Reset$NoError$": 4,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/failures/com.twitter.finagle.buoyant.h2.Reset$Cancel$": 4,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/stream_success": 1026994,
   "rt/grpc-incoming/server/0.0.0.0/4140/stream/open_streams": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/failures": 96,
   "rt/grpc-incoming/server/0.0.0.0/4140/failures/com.twitter.finagle.buoyant.h2.Reset$Cancel$": 95,
   "rt/grpc-incoming/server/0.0.0.0/4140/failures/com.twitter.finagle.buoyant.h2.StreamError$Remote": 1,
   "rt/grpc-incoming/server/0.0.0.0/4140/socket_writable_ms": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/dtab/size.count": 0,
   "rt/grpc-incoming/server/0.0.0.0/4140/requests": 1027096,
   "rt/grpc-incoming/server/0.0.0.0/4140/handletime_us.count": 606,
   "rt/grpc-incoming/server/0.0.0.0/4140/handletime_us.max": 336,
   "rt/grpc-incoming/server/0.0.0.0/4140/handletime_us.min": 33,
   "rt/grpc-incoming/server/0.0.0.0/4140/handletime_us.p50": 48,
   "rt/grpc-incoming/server/0.0.0.0/4140/handletime_us.p90": 89,
   "rt/grpc-incoming/server/0.0.0.0/4140/handletime_us.p95": 100,
   "rt/grpc-incoming/server/0.0.0.0/4140/handletime_us.p99": 117,
   "rt/grpc-incoming/server/0.0.0.0/4140/handletime_us.p9990": 143,
   "rt/grpc-incoming/server/0.0.0.0/4140/handletime_us.p9999": 336,
   "rt/grpc-incoming/server/0.0.0.0/4140/handletime_us.sum": 34728,
   "rt/grpc-incoming/server/0.0.0.0/4140/handletime_us.avg": 57.30693,
   "rt/grpc-incoming/server/0.0.0.0/4140/connections": 505,
   "rt/grpc-incoming/service/svc/search.Search/failures": 2,
   "rt/grpc-incoming/service/svc/search.Search/failures/com.twitter.finagle.buoyant.h2.Reset$Cancel$": 2,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/connect_latency_ms.count": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/failed_connect_latency_ms.count": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/sent_bytes": 9802185207,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service_creation/service_acquisition_latency_ms.count": 606,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service_creation/service_acquisition_latency_ms.max": 1478,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service_creation/service_acquisition_latency_ms.min": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service_creation/service_acquisition_latency_ms.p50": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service_creation/service_acquisition_latency_ms.p90": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service_creation/service_acquisition_latency_ms.p95": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service_creation/service_acquisition_latency_ms.p99": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service_creation/service_acquisition_latency_ms.p9990": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service_creation/service_acquisition_latency_ms.p9999": 1478,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service_creation/service_acquisition_latency_ms.sum": 1477,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service_creation/service_acquisition_latency_ms.avg": 2.4372938,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/connection_received_bytes.count": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/connection_duration.count": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/failure_accrual/removals": 1,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/failure_accrual/probes": 1,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/failure_accrual/removed_for_ms": 5000,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/failure_accrual/revivals": 1,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/connects": 3,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/success": 1026981,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/stream_failures": 8,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/data_bytes.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/data_bytes.max": 10087,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/data_bytes.min": 3729,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/data_bytes.p50": 9132,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/data_bytes.p90": 9694,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/data_bytes.p95": 9790,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/data_bytes.p99": 9987,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/data_bytes.p9990": 10087,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/data_bytes.p9999": 10087,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/data_bytes.sum": 5342604,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/data_bytes.avg": 8830.75,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/failures": 8,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/failures/com.twitter.finagle.buoyant.h2.Reset$NoError$": 8,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/stream_success": 1026988,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/stream_duration_ms.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/stream_duration_ms.max": 1554,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/stream_duration_ms.min": 13,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/stream_duration_ms.p50": 123,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/stream_duration_ms.p90": 402,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/stream_duration_ms.p95": 448,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/stream_duration_ms.p99": 744,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/stream_duration_ms.p9990": 1523,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/stream_duration_ms.p9999": 1554,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/stream_duration_ms.sum": 112275,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request/stream/stream_duration_ms.avg": 185.5785,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request_latency_ms.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request_latency_ms.max": 1554,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request_latency_ms.min": 13,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request_latency_ms.p50": 123,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request_latency_ms.p90": 402,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request_latency_ms.p95": 448,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request_latency_ms.p99": 744,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request_latency_ms.p9990": 1523,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request_latency_ms.p9999": 1554,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request_latency_ms.sum": 111525,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/request_latency_ms.avg": 184.33884,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/requeues_per_request.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/requeues_per_request.max": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/requeues_per_request.min": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/requeues_per_request.p50": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/requeues_per_request.p90": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/requeues_per_request.p95": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/requeues_per_request.p99": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/requeues_per_request.p9990": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/requeues_per_request.p9999": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/requeues_per_request.sum": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/requeues_per_request.avg": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/request_limit": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/budget_exhausted": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/cannot_retry": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/not_open": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/budget": 124,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/retries/requeues": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/received_bytes": 4645912066,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/read_timeout": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/write_timeout": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/connection_sent_bytes.count": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/connection_requests.count": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/success": 1026981,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/stream_failures": 8,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/data_bytes.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/data_bytes.max": 10087,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/data_bytes.min": 3729,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/data_bytes.p50": 9132,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/data_bytes.p90": 9694,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/data_bytes.p95": 9790,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/data_bytes.p99": 9987,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/data_bytes.p9990": 10087,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/data_bytes.p9999": 10087,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/data_bytes.sum": 5342604,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/data_bytes.avg": 8830.75,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/failures": 8,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/failures/com.twitter.finagle.buoyant.h2.Reset$NoError$": 8,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/stream_success": 1026988,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/stream_duration_ms.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/stream_duration_ms.max": 1554,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/stream_duration_ms.min": 13,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/stream_duration_ms.p50": 123,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/stream_duration_ms.p90": 402,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/stream_duration_ms.p95": 448,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/stream_duration_ms.p99": 744,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/stream_duration_ms.p9990": 1523,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/stream_duration_ms.p9999": 1554,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/stream_duration_ms.sum": 112275,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request/stream/stream_duration_ms.avg": 185.5785,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request_latency_ms.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request_latency_ms.max": 1554,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request_latency_ms.min": 13,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request_latency_ms.p50": 123,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request_latency_ms.p90": 402,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request_latency_ms.p95": 448,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request_latency_ms.p99": 744,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request_latency_ms.p9990": 1523,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request_latency_ms.p9999": 1554,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request_latency_ms.sum": 111528,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/request_latency_ms.avg": 184.3438,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/stream_failures": 18,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/data_bytes.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/data_bytes.max": 7559,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/data_bytes.min": 5,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/data_bytes.p50": 5664,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/data_bytes.p90": 6073,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/data_bytes.p95": 6195,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/data_bytes.p99": 6576,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/data_bytes.p9990": 6911,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/data_bytes.p9999": 7559,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/data_bytes.sum": 2708795,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/data_bytes.avg": 4477.347,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/failures": 18,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/failures/com.twitter.finagle.buoyant.h2.Reset$Cancel$": 16,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/failures/com.twitter.finagle.buoyant.h2.StreamError$Remote": 2,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/stream_success": 1026978,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/stream_duration_ms.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/stream_duration_ms.max": 295,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/stream_duration_ms.min": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/stream_duration_ms.p50": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/stream_duration_ms.p90": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/stream_duration_ms.p95": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/stream_duration_ms.p99": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/stream_duration_ms.p9990": 272,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/stream_duration_ms.p9999": 295,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/stream_duration_ms.sum": 899,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/response/stream/stream_duration_ms.avg": 1.4859505,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/stream_failures": 11,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/total_latency_ms.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/total_latency_ms.max": 1554,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/total_latency_ms.min": 14,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/total_latency_ms.p50": 123,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/total_latency_ms.p90": 402,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/total_latency_ms.p95": 453,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/total_latency_ms.p99": 744,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/total_latency_ms.p9990": 1523,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/total_latency_ms.p9999": 1554,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/total_latency_ms.sum": 112682,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/total_latency_ms.avg": 186.25124,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/data_bytes.count": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/failures": 11,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/failures/com.twitter.finagle.buoyant.h2.Reset$NoError$": 4,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/failures/com.twitter.finagle.buoyant.h2.Reset$Cancel$": 6,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/failures/com.twitter.finagle.buoyant.h2.StreamError$Remote": 1,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/stream/stream_success": 1026978,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/failures": 102,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/failures/com.twitter.finagle.buoyant.h2.Reset$Cancel$": 101,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/failures/com.twitter.finagle.buoyant.h2.StreamError$Remote": 1,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/service/svc/search.Search/requests": 1027080,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/socket_unwritable_ms": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/stream_failures": 18,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/data_bytes.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/data_bytes.max": 7559,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/data_bytes.min": 5,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/data_bytes.p50": 5664,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/data_bytes.p90": 6073,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/data_bytes.p95": 6195,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/data_bytes.p99": 6576,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/data_bytes.p9990": 6911,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/data_bytes.p9999": 7559,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/data_bytes.sum": 2708795,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/data_bytes.avg": 4477.347,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/failures": 18,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/failures/com.twitter.finagle.buoyant.h2.Reset$Cancel$": 16,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/failures/com.twitter.finagle.buoyant.h2.StreamError$Remote": 2,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/stream_success": 1026978,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/stream_duration_ms.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/stream_duration_ms.max": 295,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/stream_duration_ms.min": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/stream_duration_ms.p50": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/stream_duration_ms.p90": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/stream_duration_ms.p95": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/stream_duration_ms.p99": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/stream_duration_ms.p9990": 272,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/stream_duration_ms.p9999": 295,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/stream_duration_ms.sum": 899,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/response/stream/stream_duration_ms.avg": 1.4859505,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/closes": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/nack_admission_control/dropped_requests": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/stream_failures": 11,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/total_latency_ms.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/total_latency_ms.max": 1554,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/total_latency_ms.min": 14,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/total_latency_ms.p50": 123,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/total_latency_ms.p90": 402,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/total_latency_ms.p95": 453,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/total_latency_ms.p99": 744,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/total_latency_ms.p9990": 1523,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/total_latency_ms.p9999": 1554,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/total_latency_ms.sum": 112680,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/total_latency_ms.avg": 186.24794,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/data_bytes.count": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/data/frames": 1027072,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/data/bytes.count": 606,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/data/bytes.max": 10087,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/data/bytes.min": 3729,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/data/bytes.p50": 9132,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/data/bytes.p90": 9694,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/data/bytes.p95": 9790,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/data/bytes.p99": 9987,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/data/bytes.p9990": 10087,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/data/bytes.p9999": 10087,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/data/bytes.sum": 5355911,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/data/bytes.avg": 8838.137,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/reset": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/local/trailers": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/data/frames": 1026989,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/data/bytes.count": 605,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/data/bytes.max": 7559,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/data/bytes.min": 5,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/data/bytes.p50": 5664,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/data/bytes.p90": 6073,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/data/bytes.p95": 6195,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/data/bytes.p99": 6576,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/data/bytes.p9990": 6911,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/data/bytes.p9999": 7559,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/data/bytes.sum": 2708795,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/data/bytes.avg": 4477.347,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/reset": 29,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/remote/trailers": 1026981,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/failures": 11,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/failures/com.twitter.finagle.buoyant.h2.Reset$NoError$": 4,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/failures/com.twitter.finagle.buoyant.h2.Reset$Cancel$": 6,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/failures/com.twitter.finagle.buoyant.h2.StreamError$Remote": 1,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/stream_success": 1026978,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/stream/open_streams": 31,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/failures": 102,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/failures/com.twitter.finagle.buoyant.h2.Reset$Cancel$": 101,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/failures/com.twitter.finagle.buoyant.h2.StreamError$Remote": 1,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/available": 3,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/singletonpool/connects/fail": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/singletonpool/connects/dead": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/socket_writable_ms": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/cancelled_connects": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/dtab/size.count": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/requests": 1027080,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/loadbalancer/size": 3,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/loadbalancer/rebuilds": 1,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/loadbalancer/closed": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/loadbalancer/load": 4,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/loadbalancer/meanweight": 1,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/loadbalancer/adds": 3,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/loadbalancer/updates": 1,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/loadbalancer/algorithm/p2c_least_loaded": 1,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/loadbalancer/available": 3,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/loadbalancer/max_effort_exhausted": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/loadbalancer/busy": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/loadbalancer/removes": 0,
   "rt/grpc-incoming/client/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3/connections": 3,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/connect_latency_ms.count": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/failed_connect_latency_ms.count": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/sent_bytes": 50710,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/service_creation/service_acquisition_latency_ms.count": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/connection_received_bytes.count": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/connection_duration.count": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/connects": 2,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/success": 6,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/request_latency_ms.count": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/current_lease_ms": 1.8446744E13,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/drained": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/marked_busy": 23,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/ping_latency_us.count": 12,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/ping_latency_us.max": 864,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/ping_latency_us.min": 541,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/ping_latency_us.p50": 660,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/ping_latency_us.p90": 830,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/ping_latency_us.p95": 830,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/ping_latency_us.p99": 864,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/ping_latency_us.p9990": 864,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/ping_latency_us.p9999": 864,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/ping_latency_us.sum": 8162,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/ping_latency_us.avg": 680.1667,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/ping": 5896,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/revivals": 22,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/failures": 1,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/failures/com.twitter.util.TimeoutException": 1,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/failuredetector/close": 1,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/leased": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/mux/draining": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/retries/requeues_per_request.count": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/retries/request_limit": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/retries/budget_exhausted": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/retries/cannot_retry": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/retries/not_open": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/retries/budget": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/retries/requeues": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/received_bytes": 49279,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/read_timeout": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/write_timeout": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/nametreecache/expires": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/nametreecache/evicts": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/nametreecache/misses": 1,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/nametreecache/oneshots": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/dtabcache/expires": 2,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/dtabcache/evicts": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/dtabcache/misses": 3,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/dtabcache/oneshots": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/namecache/expires": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/namecache/evicts": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/namecache/misses": 1,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/namecache/oneshots": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/bind_latency_us.count": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/bind/failures": 2,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/bind/failures/com.twitter.finagle.Failure": 2,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/bind/success": 6,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/bind/failure": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/bind/requests": 6,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/connection_sent_bytes.count": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/connection_requests.count": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/addrcache.size": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/delegate/success": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/delegate/failure": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/socket_unwritable_ms": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/closes": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/nack_admission_control/dropped_requests": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/failures": 4,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/failures/interrupted": 4,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/failures/interrupted/com.twitter.finagle.Failure": 4,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/addr/failures": 2,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/addr/failures/com.twitter.finagle.Failure": 2,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/addr/success": 6,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/addr/failure": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/addr/requests": 6,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/available": 1,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/singletonpool/connects/fail": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/singletonpool/connects/dead": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/bindcache.size": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/request_payload_bytes.count": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/socket_writable_ms": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/cancelled_connects": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/response_payload_bytes.count": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/dtab/size.count": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/dtab/success": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/dtab/failure": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/requests": 10,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/loadbalancer/size": 1,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/loadbalancer/rebuilds": 139509,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/loadbalancer/closed": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/loadbalancer/load": 2,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/loadbalancer/meanweight": 1,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/loadbalancer/adds": 1,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/loadbalancer/updates": 139509,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/loadbalancer/algorithm/p2c_least_loaded": 1,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/loadbalancer/available": 1,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/loadbalancer/max_effort_exhausted": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/loadbalancer/busy": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/loadbalancer/removes": 0,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/pending": 2,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/protocol/thriftmux": 1,
   "rt/grpc-incoming/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/connections": 2,
   "rt/grpc-incoming/bindcache/path/expires": 2,
   "rt/grpc-incoming/bindcache/path/evicts": 0,
   "rt/grpc-incoming/bindcache/path/misses": 3,
   "rt/grpc-incoming/bindcache/path/oneshots": 0,
   "rt/grpc-incoming/bindcache/bound/expires": 2,
   "rt/grpc-incoming/bindcache/bound/evicts": 0,
   "rt/grpc-incoming/bindcache/bound/misses": 3,
   "rt/grpc-incoming/bindcache/bound/oneshots": 0,
   "rt/grpc-incoming/bindcache/tree/expires": 2,
   "rt/grpc-incoming/bindcache/tree/evicts": 0,
   "rt/grpc-incoming/bindcache/tree/misses": 3,
   "rt/grpc-incoming/bindcache/tree/oneshots": 0,
   "rt/grpc-incoming/bindcache/client/expires": 2,
   "rt/grpc-incoming/bindcache/client/evicts": 0,
   "rt/grpc-incoming/bindcache/client/misses": 3,
   "rt/grpc-incoming/bindcache/client/oneshots": 0,
   "rt/external/server/0.0.0.0/4142/sent_bytes": 0,
   "rt/external/server/0.0.0.0/4142/connection_received_bytes.count": 0,
   "rt/external/server/0.0.0.0/4142/connection_duration.count": 0,
   "rt/external/server/0.0.0.0/4142/connects": 0,
   "rt/external/server/0.0.0.0/4142/success": 0,
   "rt/external/server/0.0.0.0/4142/request_latency_ms.count": 0,
   "rt/external/server/0.0.0.0/4142/received_bytes": 0,
   "rt/external/server/0.0.0.0/4142/read_timeout": 0,
   "rt/external/server/0.0.0.0/4142/write_timeout": 0,
   "rt/external/server/0.0.0.0/4142/connection_sent_bytes.count": 0,
   "rt/external/server/0.0.0.0/4142/connection_requests.count": 0,
   "rt/external/server/0.0.0.0/4142/nacks": 0,
   "rt/external/server/0.0.0.0/4142/thread_usage/requests/mean": 0,
   "rt/external/server/0.0.0.0/4142/thread_usage/requests/relative_stddev": 0,
   "rt/external/server/0.0.0.0/4142/thread_usage/requests/stddev": 0,
   "rt/external/server/0.0.0.0/4142/transit_latency_ms.count": 0,
   "rt/external/server/0.0.0.0/4142/socket_unwritable_ms": 0,
   "rt/external/server/0.0.0.0/4142/closes": 0,
   "rt/external/server/0.0.0.0/4142/status/1XX": 0,
   "rt/external/server/0.0.0.0/4142/status/4XX": 0,
   "rt/external/server/0.0.0.0/4142/status/2XX": 0,
   "rt/external/server/0.0.0.0/4142/status/error": 0,
   "rt/external/server/0.0.0.0/4142/status/3XX": 0,
   "rt/external/server/0.0.0.0/4142/status/5XX": 0,
   "rt/external/server/0.0.0.0/4142/time/1XX.count": 0,
   "rt/external/server/0.0.0.0/4142/time/4XX.count": 0,
   "rt/external/server/0.0.0.0/4142/time/2XX.count": 0,
   "rt/external/server/0.0.0.0/4142/time/error.count": 0,
   "rt/external/server/0.0.0.0/4142/time/3XX.count": 0,
   "rt/external/server/0.0.0.0/4142/time/5XX.count": 0,
   "rt/external/server/0.0.0.0/4142/nonretryable_nacks": 0,
   "rt/external/server/0.0.0.0/4142/socket_writable_ms": 0,
   "rt/external/server/0.0.0.0/4142/dtab/size.count": 0,
   "rt/external/server/0.0.0.0/4142/requests": 0,
   "rt/external/server/0.0.0.0/4142/pending": 0,
   "rt/external/server/0.0.0.0/4142/handletime_us.count": 0,
   "rt/external/server/0.0.0.0/4142/connections": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/nametreecache/expires": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/nametreecache/evicts": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/nametreecache/misses": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/nametreecache/oneshots": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/dtabcache/expires": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/dtabcache/evicts": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/dtabcache/misses": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/dtabcache/oneshots": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/namecache/expires": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/namecache/evicts": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/namecache/misses": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/namer/namecache/oneshots": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/bind/success": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/bind/failure": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/addrcache.size": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/delegate/success": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/delegate/failure": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/addr/success": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/addr/failure": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/bindcache.size": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/request_payload_bytes.count": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/response_payload_bytes.count": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/dtab/success": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/dtab/failure": 0,
   "rt/external/interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig/protocol/thriftmux": 1,
   "rt/external/bindcache/path/expires": 0,
   "rt/external/bindcache/path/evicts": 0,
   "rt/external/bindcache/path/misses": 0,
   "rt/external/bindcache/path/oneshots": 0,
   "rt/external/bindcache/bound/expires": 0,
   "rt/external/bindcache/bound/evicts": 0,
   "rt/external/bindcache/bound/misses": 0,
   "rt/external/bindcache/bound/oneshots": 0,
   "rt/external/bindcache/tree/expires": 0,
   "rt/external/bindcache/tree/evicts": 0,
   "rt/external/bindcache/tree/misses": 0,
   "rt/external/bindcache/tree/oneshots": 0,
   "rt/external/bindcache/client/expires": 0,
   "rt/external/bindcache/client/evicts": 0,
   "rt/external/bindcache/client/misses": 0,
   "rt/external/bindcache/client/oneshots": 0
}

client_state.json of bad pod:

{
   "/$/inet/namerd.service-mesh-namespace.svc.cluster.local/4100": {
      "state": "bound",
      "addresses": [
         "namerd.service-mesh-namespace.svc.cluster.local:4100"
      ]
   },
   "/#/io.l5d.k8s/some-namespace/grpc/search-v1-0-3": {
      "state": "bound",
      "addresses": [
         "<some-ip>:8080",
         "<some-other-ip>:8080",
         "<another-ip>:8080"
      ]
   }
}

Environment:

  • linkerd/namerd version: 1.3.7
  • Platform, version, and config files (Kubernetes, DC/OS, etc): Kubernetes

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:14 (14 by maintainers)

github_iconTop GitHub Comments

1reaction
adleongcommented, Jun 14, 2018

Cool. Keep us in the loop!

1reaction
adleongcommented, May 15, 2018

Thanks for the update! We’re currently working on a feature that will give us a lot more insight into these kinds of issues and help us debug better (https://github.com/linkerd/linkerd/pull/1943). In the meantime, I’d still be very interested to hear if you see these same types of issues with the io.l5d.mesh interpreter.

Read more comments on GitHub >

github_iconTop Results From Across the Web

A Service Mesh for Kubernetes, Part IV: Continuous ... - Linkerd
In this article, we'll show you an example of how to use Linkerd's routing rules, called dtabs, to automatically alter traffic flow through ......
Read more >
Request continuously routed to terminating pod #6238 - GitHub
Workaround: If the client reopens the connection after getting a 502 , retries hit the ready pods. The target pods container four containers:....
Read more >
Blue/Green Examples for Istio & Linkerd on Kubernetes
This rule directs 100% of traffic to the blue version of the service, as seen by the “labels” directive under “route”. If I...
Read more >
How Kubernetes Services Work – BMC Software | Blogs
Pods in a Kubernetes deployment are regularly created and destroyed, causing their IP addresses to change constantly. It will create ...
Read more >
View live data with Container insights - Azure Monitor
View Kubernetes logs, events, and pod metrics in real time ... feature for direct access to your Azure Kubernetes Service (AKS) container ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

io.buoyant:admin-names_2.12:jar:1.4.1 not available on Maven central

Next page

Enable plugins to emit stat into L5D stat trees for Announcer