How to use authentication for django-prometheus when deployed to production?

Every path in our urls go through a LoginMiddleware that requires login authentication and redirects the user to /admin/login if not authenticated. The exceptions are the API enpoints which use a token authentication.

This project is deployed in Kubernetes and when the requests are made to /metrics they are redirected to the login page. Since we don’t want to make the /metrics without authentication how can I use an authentication method for django-prometheus in production?

[everspader]

Found a final solution. I added a custom handle for the /metrics in my middleware’s process_request() method like this:

def process_request(self, request):
    path = request.path_info.lstrip('/')

    # Handle '/metrics' endpoint to use a separate authentication backend
    # rather than JWT token or admin login
    if 'metrics' in path:
        if 'Authorization' in request.headers:
            auth = request.headers['Authorization']
            if auth.startswith('Basic '):
                auth = auth.split('Basic ')[1]
                auth = base64.b64decode(auth).decode('utf-8')
                username, password = auth.split(':')
                user = authenticate(request=request, username=username, password=password)

                if user:
                    return self.get_response(request)

        return redirect(settings.LOGIN_URL)

This will force the request to /metrics to authenticate so the first method attempted will be my custom authentication backend, that needs to be placed as first on the list in settings.py. The final backends.py looks like this:

from django.conf import settings
from django.contrib.auth.backends import BaseBackend
from django.contrib.auth.hashers import check_password
from django.contrib.auth.models import User


class MetricsAuthBackend(BaseBackend):
    """
    Custom authentication backend for the /metrics endpoint only.

    This backend will allow prometheus to authenticate with a
    username and password in the headers rather than with a
    JWT token.
    """

    def authenticate(self, request, username=None, password=None):
        """
        Authenticate a user based on the username and password.
        """

        if '/metrics' in request.path:
            if username == settings.PROMETHEUS_USERNAME:
                try:
                    user = User.objects.get(username=username)

                    if user:
                        pwd_valid = check_password(password, user.password)
                        if pwd_valid:
                            return user
                        raise Exception('Invalid password')

                except User.DoesNotExist:
                    return None

        return None

    def get_user(self, user_id):
        try:
            return User.objects.get(pk=user_id)
        except User.DoesNotExist:
            return None

[cristianrat]

I also came across this issue, looked at various options and in the end my solution is this: In the ingress, instead of allowing /metrics, I forward it to a “blackhole” nginx service with nothing in it so /metrics is a 404 via ingress But prometheus hits the pod directly (and not via ingress), so can parse the metrics correctly I don’t want the world to know I have /metrics, and also wanted to avoid all the extra setup of passwords and so on My ingress looks like so

...
...
      - path: /metrics 
        pathType: Prefix
        backend:
          service:
            name: nginx-blackhole-service
            port:
              number: 80         
...
...

Where the blackhole service is a basic service with nginx and nothing else (ie: /metrics would be a 404, since it has nothing)