Pytest hangs at "collecting..." when using GitLab CI/CD service

Problem description

Yesterday, I ran into a problem when running a GitLab CI/CD pipeline using pytest to perform the integration tests step.

I can run my tests locally and I confirmed that I can reach my service on GitLab CI through Curl. However, when I try to reach my service through an http request inside my test stub, pytest gets stuck at “collecting…”, without any log to indicate what the error could be.

I have also tried to change the http request host to our production server or even giving it a wrong host and that works fine.

I do not believe the error is with GitLab, because I can reach the service just fine with Curl and I don’t think it is a DNS problem, because pytest is not throwing any error, it is simply stuck.

Either way, I believe there should at least be more logs on the pytest side to help me debug this problem, because right now I have no idea what can be causing this issue.

Example:

This is my .gitlab-ci.yml file:

test:
  image: python:latest
  stage: test
  tags:
    - docker
  variables:
    FF_NETWORK_PER_BUILD: 1 # Allows communication between services

    # Postgresql
    POSTGRES_USER: postgres
    POSTGRES_PASSWORD: xxx
    POSTGRES_DB: account_manager

  services:
    - name: postgres:latest
      alias: postgresql
      command: [ "postgres", "-c", "wal_level=logical" ]

    # SERVICE THAT GETS PYTEST STUCK
    - name: gitlab-docker-registry/account-manager-service:latest
      alias: account-manager-test

  before_script:
    - apt-get update

    # Test connection to account-manager-test service
    - |
      curl -H "X-Correlation-ID: 5c70300e-9f45-4447-bd09-72f4ba3c8209" http://account-manager-test:8080/api/account/user?user-ids=1234567892
    # Works just fine

    - pip install pytest

  script:
    - pytest device_manager_service -s -vvv --full-trace --color=yes --code-highlight=yes

My account-manager-test container has a flask-sqlalchemy API and I communicate with it using http with the python requests module. Below is the part of my test stub where I use the gitlab service in question:

import requests
import json

....
# The host is the service alias
new_user = requests.post(url="http://account-manager-test:8080/api/account"+ "/register", data=register_data, headers=headers)

new_user_id = json.loads(new_user.content)['user_id']

In my understanding, there should be no problems using the gitlab ci service here. However, this is the output I get on my shared runner:

I left it running for 1 hour and got no output from pytest.

And just to make sure, I made a normal python script that made an http request using the service alias (without pytest involved) and that worked! So I’m convinced the problem is indeed on the side of pytest.

Container environment:

pytest==6.2.*
pytest-cov==2.11.*
pytest-randomly==3.7.*
Flask-Testing==0.8.*
jsonschema==3.2.*
prance==0.20.*
importlib-metadata==4.11.*
connexion==2.7.*
python-dateutil==2.8.*
setuptools==56.2.*
Flask==1.1.*
waitress==2.0.*
Flask-SQLAlchemy==2.5.*
psycopg2==2.8.*
prometheus-flask-exporter==0.18.*
SQLAlchemy==1.4.*
opencensus==0.7.*
opencensus-ext-ocagent==0.7.*
opencensus-ext-flask==0.7.*
opencensus-ext-sqlalchemy==0.1.*
opencensus-ext-requests==0.7.*
opencensus-ext-logging==0.1.*
python-json-logger==2.0.*
kafka-python==2.0.*
marshmallow==3.13.*
urllib3==1.26.*
PyJWT==2.3.*
markupsafe==2.0.1

I know this is not an easily reproducible issue (you need a gitlab runner to reproduce), but I think that pytest should at least print some output to help the programmer understand where the error might be. If anyone has any sort of input, or has had a similar experience to mine, I would be very grateful for your help.

[Vasco27]

Well, I had -s in every test, so I’m guessing the python on gitlab might be set to do block buffering instead of line buffering, but I’m not really sure. The fact is that I tried locally and it is printing ok during the collection phase, so it does seem like the problem is with the gitlab runner buffering.

Anyway, thank you for your help debugging. I know where the error is now and I can debug it properly. (and btw, your videos are great, you just gained a subscriber 😃 )

[RonnyPfannschmidt]

closing as resolved as ci interaction hiding ci related errors