Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Responses 0.19.0 seems to break some s3 mocks
See original GitHub issueThe 0.19.0 Responses seems to break some s3 mocks.
We noticed a lot of failing test cases in our CI after today’s 0.19.0 upgrade. All of them are S3-mock related: https://github.com/apache/airflow/runs/5447285682?check_suite_focus=true#step:8:18784
Environment
Apache Airflow development environment.
Which SDK and version?
- responses 0.19.0
Steps to Reproduce
- This is the fastest way to bring the development environment of Airflow to demonstrate the error:
docker run --env "AIRFLOW__CORE__EXECUTOR=SequentialExecutor" \
--env "AIRFLOW__CORE__SQL_ALCHEMY_CONN=sqlite:////root/airflow/airflow.db" \
-it \
ghcr.io/apache/airflow/main/ci/python3.7:8bb092fb6bbafbdca5a08d5f30329044fdd9794c
- Once the above command succeds, you should see this:
Airflow home: /root/airflow
Airflow sources: /opt/airflow
Airflow core SQL connection: sqlite:////root/airflow/airflow.db
Using already installed airflow version
No need for www assets recompilation.
===============================================================================================
Checking integrations and backends
===============================================================================================
-----------------------------------------------------------------------------------------------
Disabled integrations: kerberos mongo redis cassandra openldap trino pinot rabbitmq
Enable them via --integration <INTEGRATION_NAME> flags (you can use 'all' for all)
Your dags for webserver and scheduler are read from /root/airflow/dags directory
You can add /files/airflow-breeze-config directory and place variables.env
In it to make breeze source the variables automatically for you
You can add /files/airflow-breeze-config directory and place .tmux.conf
in it to make breeze use your local .tmux.conf for tmux
You can add /files/airflow-breeze-config directory and place init.sh
In it to make breeze source an initialization script automatically for you
root@6a1ac9c2648f:/opt/airflow#
- Run example test (in the container):
pytest tests/providers/amazon/aws/hooks/test_s3.py::TestAwsS3Hook::test_delete_bucket_if_not_bucket_exist
- It fails with error inside moto library (but apparently related to
responseslibrary:
/usr/local/lib/python3.7/site-packages/moto/core/models.py:118:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.7/site-packages/moto/core/models.py:96: in start
self.enable_patching(reset)
/usr/local/lib/python3.7/site-packages/moto/core/models.py:362: in enable_patching
callback=convert_flask_to_responses_response(value),
/usr/local/lib/python3.7/site-packages/responses/__init__.py:603: in add
self._registry.add(method)
/usr/local/lib/python3.7/site-packages/responses/registries.py:53: in add
....
- Downgrade
responsesto 0.18.0
root@6a1ac9c2648f:/opt/airflow# pip install responses==0.18.0
install responses==0.18.0
Collecting responses==0.18.0
Downloading responses-0.18.0-py3-none-any.whl (38 kB)
Requirement already satisfied: urllib3>=1.25.10 in /usr/local/lib/python3.7/site-packages (from responses==0.18.0) (1.26.8)
Requirement already satisfied: requests<3.0,>=2.0 in /usr/local/lib/python3.7/site-packages (from responses==0.18.0) (2.27.1)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.7/site-packages (from requests<3.0,>=2.0->responses==0.18.0) (2.0.12)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests<3.0,>=2.0->responses==0.18.0) (2020.12.5)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests<3.0,>=2.0->responses==0.18.0) (3.3)
Installing collected packages: responses
Attempting uninstall: responses
Found existing installation: responses 0.19.0
Uninstalling responses-0.19.0:
Successfully uninstalled responses-0.19.0
Successfully installed responses-0.18.0
- Rerun the example test. This time it will succeed:
root@6a1ac9c2648f:/opt/airflow# pytest tests/providers/amazon/aws/hooks/test_s3.py::TestAwsS3Hook::test_delete_bucket_if_not_bucket_exist
============================================================================================================================================== test session starts ==============================================================================================================================================
platform linux -- Python 3.7.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/airflow, configfile: pytest.ini
plugins: anyio-3.5.0, instafail-0.4.2, timeouts-1.2.1, requests-mock-1.9.3, httpx-0.20.0, forked-1.4.0, xdist-2.5.0, asyncio-0.18.2, flaky-3.7.0, rerunfailures-9.1.1, cov-3.0.0
asyncio: mode=strict
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
collected 1 item
tests/providers/amazon/aws/hooks/test_s3.py::TestAwsS3Hook::test_delete_bucket_if_not_bucket_exist PASSED [100%]
=============================================================================================================================================== warnings summary ================================================================================================================================================
airflow/configuration.py:376
/opt/airflow/airflow/configuration.py:376: FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }
}/{%% if ti.map_index >= 0 %%}map_index={{ ti.map_index }}/{%% endif %%}attempt={{ try_number }}.log' in the running config, but please update your config before Apache Airflow 3.0.
FutureWarning,
-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================================================================================= 1 passed, 1 warning in 0.60s ==========================================================================================================================================
root@6a1ac9c2648f:/opt/airflow#
Expected Result
I expect that the 0.19.0 release will not break moto’s s3 mocking 😃.
Actual Result
It breaks it
Issue Analytics
- State:
- Created 2 years ago
- Reactions:6
- Comments:16 (4 by maintainers)
Top Results From Across the Web
Moto seems to stop mocking after upgrading boto3
It looks like (worryingly) the mock has stopped working altogether and is trying to actually access s3, here is an example test function ......
Read more >moto Changelog - pyup.io
This is a major release, and as such contains some breaking changes. * Removed: ... Moto now supports mocking S3-equivalent libraries such as...
Read more >Package List — Spack 0.20.0.dev0 documentation
It features a few simple effects, all of the editing features you should need, ... Test Dependencies: py-nose, py-mock; Description: This package provides...
Read more >What's New - Oracle Linux Yum Server
... mock-3.5-1.el8 - Builds packages inside chroots (Update) ... Framework for collecting user feedback for apps via telemetry and surveys (New).
Read more >List of supported software - EasyBuild documentation
The alleleCount package primarily exists to prevent code duplication between some other projects, specifically AscatNGS and Battenberg. As of v4 the perl ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Yes, it is related, @beliaev-maksim. There’s two underlying issues:
CallbackResponse, and one of the values within that class doesn’t like being copiedGuarding our own add-method was both the easiest and the most foolproof fix.
respones 0.19.0 also break moto (versions 2.3.1 to 3.0.6)