Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Testing with Complement has gotten so slow - 2 minutes to see test results
See original GitHub issueI used to be able to make a change in Synapse and see Complement results after 20 seconds. Now it takes 1 minute 15 seconds just to build the Docker images at the start. And over 2 minutes to see the test results for running a single Complement test. This makes local dev such a pain :feelsgood:
Docker build is too slow
When I’m just doing feature work, I don’t care about workers, I don’t care about Postgres (whatever is new and making it slow).
1 minute 15 seconds of Docker building after changing Synapse:
Complement run ./scripts-dev/complement.sh
$ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestJumpToDateEndpoint/parallel/federation/can_get_pagination_token_after_getting_remote_event_from_timestamp_to_event_endpoint
[+] Building 22.0s (25/25) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 5.62kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 35B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1 0.3s
=> CACHED docker-image://docker.io/docker/dockerfile:1@sha256:443aab4ca21183e069e7d8b2dc68006594f40bddf1b15bbd83f5137bd93e80e2 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> [internal] load .dockerignore 0.0s
=> [internal] load metadata for docker.io/library/python:3.9-slim 0.3s
=> [internal] load build context 0.3s
=> => transferring context: 7.35MB 0.3s
=> [requirements 1/6] FROM docker.io/library/python:3.9-slim@sha256:c01a2db78654c1923da84aa41b829f6162011e3a75db255c24ea16fa2ad563a0 0.0s
=> CACHED [builder 2/7] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cache,target=/var/lib/apt,sharing=locked apt-get update -qq && apt-get install -yqq build-essential 0.0s
=> CACHED [requirements 2/6] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cache,target=/var/lib/apt,sharing=locked apt-get update -qq && apt-get install -yqq git && rm -rf / 0.0s
=> CACHED [requirements 3/6] RUN --mount=type=cache,target=/root/.cache/pip pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5" 0.0s
=> CACHED [requirements 4/6] WORKDIR /synapse 0.0s
=> CACHED [requirements 5/6] COPY pyproject.toml poetry.lock /synapse/ 0.0s
=> CACHED [requirements 6/6] RUN /root/.local/bin/poetry export --extras all -o /synapse/requirements.txt ${TEST_ONLY_SKIP_DEP_HASH_VERIFICATION:+--without-hashes} 0.0s
=> CACHED [builder 3/7] COPY --from=requirements /synapse/requirements.txt /synapse/ 0.0s
=> CACHED [builder 4/7] RUN --mount=type=cache,target=/root/.cache/pip pip install --prefix="/install" --no-deps --no-warn-script-location -r /synapse/requirements.txt 0.0s
=> [builder 5/7] COPY synapse /synapse/synapse/ 0.5s
=> [builder 6/7] COPY pyproject.toml README.rst /synapse/ 0.0s
=> [builder 7/7] RUN pip install --prefix="/install" --no-deps --no-warn-script-location /synapse 8.8s
=> CACHED [stage-2 2/5] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cache,target=/var/lib/apt,sharing=locked apt-get update -qq && apt-get install -yqq curl gosu l 0.0s
=> [stage-2 3/5] COPY --from=builder /install /usr/local 2.6s
=> [stage-2 4/5] COPY ./docker/start.py /start.py 0.1s
=> [stage-2 5/5] COPY ./docker/conf /conf 0.1s
=> exporting to image 2.5s
=> => exporting layers 2.5s
=> => writing image sha256:cf22ee98c4023602eaa86fe9b09d8043f4f01138f2aec7c745387444d0d45249 0.0s
=> => naming to docker.io/matrixdotorg/synapse 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Building 15.3s (12/12) FINISHED
=> [internal] load build definition from Dockerfile-workers 0.0s
=> => transferring dockerfile: 1.45kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 35B 0.0s
=> [internal] load metadata for docker.io/matrixdotorg/synapse:latest 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 30.16kB 0.0s
=> [stage-0 1/7] FROM docker.io/matrixdotorg/synapse:latest 0.6s
=> [stage-0 2/7] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cache,target=/var/lib/apt,sharing=locked apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y 8.5s
=> [stage-0 3/7] RUN --mount=type=cache,target=/root/.cache/pip pip install supervisor~=4.2 4.9s
=> [stage-0 4/7] RUN rm /etc/nginx/sites-enabled/default 0.4s
=> [stage-0 5/7] COPY ./docker/conf-workers/* /conf/ 0.0s
=> [stage-0 6/7] COPY ./docker/prefix-log /usr/local/bin/ 0.1s
=> [stage-0 7/7] COPY ./docker/configure_workers_and_start.py /configure_workers_and_start.py 0.0s
=> exporting to image 0.4s
=> => exporting layers 0.4s
=> => writing image sha256:0b0ae26a584d7ae58927a2daba3b508d61245e1a41d4ac9ba5b92371a96d5445 0.0s
=> => naming to docker.io/matrixdotorg/synapse-workers 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Building 42.5s (13/13) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/matrixdotorg/synapse-workers:latest 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 177B 0.0s
=> [1/8] FROM docker.io/matrixdotorg/synapse-workers:latest 0.2s
=> [2/8] RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -yqq postgresql-13 34.7s
=> [3/8] RUN pg_ctlcluster 13 main start && su postgres -c "echo "ALTER USER postgres PASSWORD 'somesecret'; CREATE DATABASE synapse ENCODING 'UTF8' LC_COLLATE='C' LC_CTYPE='C' template=template0;" | p 3.8s
=> [4/8] RUN mv /conf/shared.yaml.j2 /conf/shared-orig.yaml.j2 0.4s
=> [5/8] COPY conf/workers-shared-extra.yaml.j2 /conf/shared.yaml.j2 0.1s
=> [6/8] WORKDIR /data 0.1s
=> [7/8] COPY conf/postgres.supervisord.conf /etc/supervisor/conf.d/postgres.conf 0.1s
=> [8/8] COPY conf/start_for_complement.sh / 0.1s
=> exporting to image 2.9s
=> => exporting layers 2.9s
=> => writing image sha256:df7d7a1d4443960b49dbb310bdbd4af323c33c7900d26bf291c3fad894683453 0.0s
=> => naming to docker.io/library/complement-synapse 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Images built; running complement
...
Relevant code: https://github.com/matrix-org/synapse/blob/dcc7873700da4a818e84c44c6190525d39a854cb/scripts-dev/complement.sh#L89-L107
With cache, the Synapse build used to take ~10s seconds, https://github.com/matrix-org/synapse/pull/9610, which has now doubled to ~20 seconds. And the rest was trivial.
Complement being slow
@kegsay pointed out these couple issues which would also slow down single test runs (stopping and shutting down containers):
The deploys at the start seem noticeable now at 20 seconds:
=== RUN TestJumpToDateEndpoint
2022/07/07 08:44:59 Sharing [SERVER_NAME=hs1 SYNAPSE_COMPLEMENT_DATABASE=sqlite SYNAPSE_COMPLEMENT_USE_WORKERS=] host environment variables with container
2022/07/07 08:45:07 Sharing [SERVER_NAME=hs2 SYNAPSE_COMPLEMENT_DATABASE=sqlite SYNAPSE_COMPLEMENT_USE_WORKERS=] host environment variables with container
2022/07/07 08:45:17 Sharing [SERVER_NAME=hs1 SYNAPSE_COMPLEMENT_DATABASE=sqlite SYNAPSE_COMPLEMENT_USE_WORKERS=] host environment variables with container
2022/07/07 08:45:17 Sharing [SERVER_NAME=hs2 SYNAPSE_COMPLEMENT_DATABASE=sqlite SYNAPSE_COMPLEMENT_USE_WORKERS=] host environment variables with container
msc3030_test.go:24: Deploy times: 17.9124869s blueprints, 5.628477139s containers
Test output no longer streams as it runs
This is just a related extra.
The tests used to stream out what it was doing as it went instead of only barfing it all up at the end. This used to work until we got multiple packages in Complement (tests and csapi ), https://github.com/matrix-org/complement/issues/215. This doesn’t help with making Complement go faster but it does help with the feeling of it doing something vs just waiting around arbitrarily.
The way I workaround it now is to remove the ... from the end of
so it only runs against the single tests package where the feature work I care about is in.
Issue Analytics
- State:
- Created a year ago
- Comments:11 (7 by maintainers)
Top Results From Across the Web
Why It Takes So Long To Get Most COVID-19 Test Results
The FDA in recent days has approved tests from two companies that promise results in 45 minutes or less, but those likely will...
Read more >Turnaround time for COVID-19 test results
Review typical turnaround times to find out when you can expect to get your results for COVID-19 active infection, antigen, or antibody testing....
Read more >What Takes So Long? A Behind-The-Scenes Look At ...
A common complaint about the testing process is the long turnaround time for results.
Read more >Why Your COVID Test Results Take That Long
The longest analysis is for the PCR test, which varies from lab to lab but often requires about 1.5 to 2 hours, he...
Read more >8 tips for getting the best results from at-home COVID-19 ...
To ensure you're using an accurate antigen test, check the FDA's list of antigen tests that have received emergency use authorization. 2. Follow ......
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
This will be the rust build step, I guess. Maybe we can reorder the dockerfile to cache the rust build output? Otherwise I don’t have any great ideas here.
Try sticking
-p 1on thecomplement.shcommandline.