opentelemetry-instrumentation-grpc client instrumentation fails if channel has interceptors
See original GitHub issueDescribe your environment
Verified on python 3.8.6 with opentelemetry 0.17b0
Steps to reproduce
tl;dr: GrpcInstrumentorClient().instrument()
+ a channel intercepted with grpc.intercept_channel
Based on the gRPC helloworld example, sub out greeter_client.py
for:
# Copyright 2015 gRPC authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""The Python implementation of the GRPC helloworld.Greeter client."""
from __future__ import print_function
import logging
import grpc
from opentelemetry import trace
from opentelemetry.instrumentation.grpc import GrpcInstrumentorClient
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
ConsoleSpanExporter,
SimpleExportSpanProcessor,
)
import helloworld_pb2
import helloworld_pb2_grpc
trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
SimpleExportSpanProcessor(ConsoleSpanExporter())
)
GrpcInstrumentorClient().instrument()
class Interceptor(
grpc.UnaryUnaryClientInterceptor,
grpc.UnaryStreamClientInterceptor,
grpc.StreamUnaryClientInterceptor,
grpc.StreamStreamClientInterceptor,
):
def intercept_unary_unary(self, continuation, client_call_details, request):
return self._intercept_call(continuation, client_call_details, request)
def intercept_unary_stream(self, continuation, client_call_details, request):
return self._intercept_call(continuation, client_call_details, request)
def intercept_stream_unary(self, continuation, client_call_details, request_iterator):
return self._intercept_call(continuation, client_call_details, request_iterator)
def intercept_stream_stream(self, continuation, client_call_details, request_iterator):
return self._intercept_call(continuation, client_call_details, request_iterator)
def _intercept_call(self, continuation, client_call_details, request_or_iterator):
return continuation(client_call_details, request_or_iterator)
def run():
# NOTE(gRPC Python Team): .close() is possible on a channel and should be
# used in circumstances in which the with statement does not fit the needs
# of the code.
interceptors = [Interceptor()]
channel = grpc.insecure_channel('localhost:50051')
channel = grpc.intercept_channel(channel, *interceptors)
stub = helloworld_pb2_grpc.GreeterStub(channel)
response = stub.SayHello(helloworld_pb2.HelloRequest(name='you'))
print("Greeter client received: " + response.message)
if __name__ == '__main__':
logging.basicConfig()
run()
What is the expected behavior?
The client should continue to behave as a standard, working gRPC client, now with instrumentation
What is the actual behavior?
Requests made through the client now error, e.g.:
❯ python greeter_client.py
Traceback (most recent call last):
File "greeter_client.py", line 76, in <module>
run()
File "greeter_client.py", line 70, in run
response = stub.SayHello(helloworld_pb2.HelloRequest(name='you'))
File "/home/isobel/workspace/tmp/otlp/grpc/lib/python3.8/site-packages/grpc/_interceptor.py", line 216, in __call__
response, ignored_call = self._with_call(request,
File "/home/isobel/workspace/tmp/otlp/grpc/lib/python3.8/site-packages/grpc/_interceptor.py", line 257, in _with_call
return call.result(), call
File "/home/isobel/workspace/tmp/otlp/grpc/lib/python3.8/site-packages/grpc/_interceptor.py", line 126, in result
raise self._exception
File "/home/isobel/workspace/tmp/otlp/grpc/lib/python3.8/site-packages/grpc/_interceptor.py", line 241, in continuation
response, call = self._thunk(new_method).with_call(
TypeError: with_call() got an unexpected keyword argument 'wait_for_ready'
Note: I’ve so far only verified this causes problems with UnaryUnary requests. I assume that some or all of the other types fail similarly.
Additional context
Happy to fix this myself 😃
Issue Analytics
- State:
- Created 3 years ago
- Comments:6 (4 by maintainers)
Top Results From Across the Web
How can I close Channel when using Interceptors
Normally when I was using stubs I was creating ChannelImpl (by NettyChannelBuilder) and then when program ends I used channelImpl.shutdown().
Read more >Source code for opentelemetry.instrumentation.grpc
GreeterStub(channel) response = stub.SayHello(helloworld_pb2.HelloRequest(name="YOU")) print("Greeter client received: " + response.message) if __name__ ...
Read more >Thoughts on HTTP instrumentation with OpenTelemetry
First, let's think about HTTP client API: it has send call to execute a request; it likely has handlers/interceptors for auth, retries, logging, ......
Read more >IBM WebSphere MQ AMS MCA interception example
In this MCA interception example, the interceptors are provided at the server end of the channel, and an older client runtime is used...
Read more >Chapter 3. Messages and channels - Spring Integration in Action
Introducing messages and channels; How different types of channels work; Customizing channel functionality with dispatchers and interceptors;
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Ugh, I knew this one was going to come up based on the hidden magic of how the “wrapper” thing was initially designed, it’s why I redesigned the server part to be able to handle an existing interceptor chain.
It should be a pretty easy fix, just to mimic how it’s done in the server. If you want to work on it @iredelmeier, say so - otherwise I could probably get to it myself 😃
I ran the code with the following packages installed
and got the following output:
I believe this issue has been fixed. Please re-open if that’s not the case @iredelmeier