Feature Request: support conditionally-skipping tests

Hello and thank you for developing and maintaining this library! The ability to safely write unit tests that perform sqlalchemy operations within Flask applications is a breath of fresh air 😃

This feature request is for a use case in which database connectivity isn’t guaranteed to be available at test runtime, depending on the environment.

For situations like that it’d be convenient to decorate test cases as conditional upon database connectivity, and for those tests then to get skipped (instead of failed) when connectivity’s not available.

I’ve made some attempts to handle this situation for one codebase, and have some findings and suggestions based on that, but have a sense that the approach may be non-ideal and would benefit from discussion and feedback.

One challenge is that the plugin’s _transaction fixture attempts to create a database connection and transaction without handling errors at the moment (ref). That does make sense when connectivity is expected to be available, and provides a concise and fail-fast behaviour that’s useful in most cases.

Another challenge is determining how test authors would opt-in to connectivity-conditional tests; unittest provides a skipUnless function that could provide a readable and comprehensible expression of ‘skip this test unless a transaction object is available’. In practice, it seemed tricky to write a skip-condition function that would run before injection of test fixtures and that could also be migrated into the plugin in future.

Alternatively pytest supports custom markers that could similarly be used to decorate tests with preconditions.

This feature request is accompanied by a pull requests attempting to solve the problem via one proposed approach – adding a custom pytest marker with autouse enabled as part of the project-under-test codebase – accompanied by some naive error handling in the pytest-flask-sqlalchemy codebase. It would be possible to migrate the custom marker into the plugin codebase at a later date if this seems like a reasonable approach.

[jeancochrane]

Thanks for this writeup and the initial work! Can you give an example of a case where database connectivity wouldn’t be guaranteed at test time? I’m having trouble getting an idea of when this feature would be useful. My instinct is that a unit test should either have database access or not, and indeterminacy would indicate a test that hasn’t fully isolated the functionality it seeks to test.

The custom marker implemented in https://github.com/openculinary/backend/pull/20 seems like the right API if this is indeed a feature we want to support in core. I’m not quite sold on the exception handling in https://github.com/jeancochrane/pytest-flask-sqlalchemy/pull/31, however, because I think for the majority of use cases it’s important to surface errors in the connection setup rather than suppressing them. Once we decide on the usefulness of this feature I’d be happy to think through some alternatives.

[jayaddison]

As noted in #42, we can consider this functionality out-of-scope for pytest-flask-sqlalchemy.