FastAPI / Starlette log shipping to APM Server

Describe the bug: The FastAPI / Starlette client doesn’t seem to be sending data from the python logger to the APM server. I see that elasticapm.contrib.flask.ElasticAPM support logging=True as an argument, but don’t see the same for elasticapm.contrib.starlette.ElasticAPM. So I’m not sure if this is a bug or feature request.

To Reproduce

1. Run the FastAPI app specified in main.py (see Additional context below).
2. Send requests to the API by running.

for i in $(seq 100); do http :8000/foo foo="${i}@email.domain" bar="${i}"; done

1. Check the elasticsearch index pattern apm-* if logged data has been added.
2. Result: Only the message from apm_client.capture_message('hello, world!') has been uploaded, but the message from logger.warning('This is a warning') was not uploaded.

Environment (please complete the following information)

• OS: OSX
• Python version: 3.8
• Framework and version: fastapi==0.54.1
• APM Server version: 7.7.0
• Agent version: elastic-apm==5.6.0

Additional context This is the code in main.py.

from fastapi import Body, FastAPI
from pydantic import BaseModel
from elasticapm.contrib.starlette import make_apm_client, ElasticAPM
from elasticapm.handlers.logging import LoggingFilter, Formatter
import elasticapm

# Parse loggs to be correlated with APM 
# https://www.elastic.co/guide/en/apm/agent/python/master/log-correlation.html
formatter = Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler = logging.StreamHandler()
handler.setFormatter(formatter)
handler.addFilter(LoggingFilter())
handler.setLevel(logging.DEBUG)
logger = logging.getLogger("elasticapm")
logger.addHandler(handler)

# Create the fastapi app
app = FastAPI()

# Creating the fastapi apm middleware
# https://www.elastic.co/guide/en/apm/agent/python/master/starlette-support.html
# https://www.elastic.co/guide/en/apm/agent/python/master/configuration.html
settings = {
    'SERVER_URL':os.environ.get('APM_SERVER_URL'),
    'SERVICE_NAME':os.environ.get('APM_SEVICE_NAME'),
    'SECRET_TOKEN':os.environ.get('APM_SECRET_TOKEN'),
    'CAPTURE_BODY':'all',
    'CAPTURE_HEADERS': True,
    'DEBUG': True,
    'COLLECT_LOCAL_VARIABLES':'all',
    'ELASTIC_APM_AUTO_LOG_STACKS':True
    } 
apm_client = make_apm_client(settings)
app.add_middleware(ElasticAPM, client=apm_client)

# Below is a fast API example app
# https://fastapi.tiangolo.com/
class Foo(BaseModel):
    foo: str
    bar: str = None

@app.post("/foo", response_model=Foo)
def foo(request: Foo = Body(...)):
    result = {
        'foo': request.foo,
        'bar': request.bar,
    }
    # Log a message with the root logger
    logger.warning('This is a warning')

    # Log a generic message with capture_message:
    apm_client.capture_message('hello, world!')

    return result

[basepi]

@zero0nee We’d love to have you work on this!

However, I think the direction I’d really like to take this is to add log-shipping as a top-level feature of the agent, rather than continuing to add it to each individual instrumentation/integration. To that end, I need to find some time to crystallize the design of this in my mind a little so I can provide you some good guidance on where to start on adding this as a feature. Stay tuned, I’ll revisit this either tomorrow or next week.

[DanielFerreiraJorge]

Hi!

I too am really interested in FastAPI being on par with flask in terms of features of the apm agent. FastAPI is a really important framework to be left behind. It is not really usable right now…