Protobuf version incompatibility

Description [NOTE]: # Pip install grpc+google-cloud-core+newrelic end up with a newrelic version which is not compatible with the protobuf installed

Expected Behavior [NOTE]: # Library should not crash when importing it

Troubleshooting or NR Diag results [NOTE]: #

~ ❯ python -c 'import newrelic.agent'                                                                                                                            🐍 testenv 3.9.9   2.7.4
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/xxx/.pyenv/versions/testenv/lib/python3.9/site-packages/newrelic/agent.py", line 15, in <module>
    from newrelic.config import (
  File "/Users/xxx/.pyenv/versions/testenv/lib/python3.9/site-packages/newrelic/config.py", line 26, in <module>
    import newrelic.api.application
  File "/Users/xxx/.pyenv/versions/testenv/lib/python3.9/site-packages/newrelic/api/application.py", line 19, in <module>
    import newrelic.core.agent
  File "/Users/xxx/.pyenv/versions/testenv/lib/python3.9/site-packages/newrelic/core/agent.py", line 33, in <module>
    import newrelic.core.application
  File "/Users/xxx/.pyenv/versions/testenv/lib/python3.9/site-packages/newrelic/core/application.py", line 32, in <module>
    from newrelic.core.config import global_settings
  File "/Users/xxx/.pyenv/versions/testenv/lib/python3.9/site-packages/newrelic/core/config.py", line 44, in <module>
    from newrelic.core.infinite_tracing_pb2 import (  # pylint: disable=W0611,C0412  # noqa: F401
  File "/Users/xxx/.pyenv/versions/testenv/lib/python3.9/site-packages/newrelic/core/infinite_tracing_pb2.py", line 45, in <module>
    _descriptor.FieldDescriptor(
  File "/Users/xxx/.pyenv/versions/testenv/lib/python3.9/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

Steps to Reproduce [NOTE]:

# New python virtualenv with nothing installed
pyenv virtualenv 3.9.9 testenv
 pyenv activate testenv

# Install dependencies
pip install grpc google-api-corenewrelic

# Run python which just imports newrelic.agent
python -c 'import newrelic.agent'

Your Environment [TIP]: # Python 3.9.9 (shouldn’t be relevant) and latest versions of grpc/google-cloud-core/newrelic. Happens on my Mac and inside docker too.

Additional context [TIP]: # I have seen that in newrelic’s setup.py the grpc and protobuf version is only in the dependencies when using newrelic[infinite-tracing], but in this case we’re not using it. Since we installed grpc/protobuf for other reasons, the lib seems to think infinite-tracing is enabled and the protobuf version required is the correct one, which it isn’t and then crashes

[drewbrew]

Google yanked protobuf 4.21, FWIW.

[TimPansino]

Seeing as Google has yanked protobuf 4.21.0, it seems the issue is currently not a problem with the latest non-yanked version of protobuf.

It looks like a breaking change with the introduction of protobuf 4, anyone seeing this issue can safely pin protobuf<4 in their requirements to avoid this issue. We’ll get this version pinned in our next release while we wait for a stable release in the 4 series.