Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
test_dbrestore tests apparently fail when run locally
See original GitHub issueObserved behavior
test_dbrestore tests apparently fail when run locally:
(k012x)d@mac ~/P/l/r/k012x (coach-data) [1]> pytest kolibri/core/deviceadmin/tests/test_dbrestore.py
================================================================== test session starts ===================================================================
platform darwin -- Python 3.6.5, pytest-3.7.1, py-1.7.0, pluggy-0.8.0
Django settings: kolibri.deployment.default.settings.test (from ini file)
rootdir: /Users/d/Projects/learning_equality/repos/k012x, inifile: pytest.ini
plugins: pythonpath-0.7.2, django-3.3.3, cov-2.5.1
collected 11 items
kolibri/core/deviceadmin/tests/test_dbrestore.py F...F..FFF. [100%]
======================================================================== FAILURES ========================================================================
______________________________________________________________________ test_latest _______________________________________________________________________
def test_latest():
with pytest.raises(RuntimeError):
> call_command("dbrestore", latest=True)
kolibri/core/deviceadmin/tests/test_dbrestore.py:63:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv/lib/python3.6/site-packages/django/core/management/__init__.py:131: in call_command
return command.execute(*args, **defaults)
.venv/lib/python3.6/site-packages/django/core/management/base.py:330: in execute
output = self.handle(*args, **options)
kolibri/core/deviceadmin/management/commands/dbrestore.py:76: in handle
use_backup = search_latest(search_root, fallback_version)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
search_root = '/Users/d/kolibri/012x/backups', fallback_version = '0.12'
def search_latest(search_root, fallback_version):
logger.info("Searching latest backup in {}...".format(search_root))
newest = None # Should be a path/filename.sqlite3
newest_dtm = None
# All file names have to be according to the fall back version.
prefix = "db-v{}".format(fallback_version)
backups = os.listdir(search_root)
backups = filter(lambda f: f.endswith(".dump"), backups)
backups = filter(lambda f: f.startswith(prefix), backups)
# Everything is sorted alphanumerically, and since dates in the
# filenames behave accordingly, we can now traverse the list
# without having to access meta data, just use the file name.
backups = list(backups)
backups.sort()
for backup in backups:
try:
dtm = get_dtm_from_backup_name(backup)
except ValueError:
continue
# Always pick the newest version
> if is_full_version(backup) or dtm > newest_dtm:
E TypeError: '>' not supported between instances of 'str' and 'NoneType'
kolibri/core/deviceadmin/utils.py:176: TypeError
------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------
INFO Beginning database restore
INFO Searching latest backup in /Users/d/kolibri/012x/backups...
_________________________________________________________________ test_inactive_kolibri __________________________________________________________________
def test_inactive_kolibri():
"""
Tests that we cannot restore while kolibri is active
"""
with patch(
"kolibri.utils.server.get_status",
side_effect=mock_status_not_running
) as gs:
# Since there's no backups available during a test, this should fail!
with pytest.raises(RuntimeError):
> call_command("dbrestore", latest=True)
kolibri/core/deviceadmin/tests/test_dbrestore.py:103:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv/lib/python3.6/site-packages/django/core/management/__init__.py:131: in call_command
return command.execute(*args, **defaults)
.venv/lib/python3.6/site-packages/django/core/management/base.py:330: in execute
output = self.handle(*args, **options)
kolibri/core/deviceadmin/management/commands/dbrestore.py:76: in handle
use_backup = search_latest(search_root, fallback_version)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
search_root = '/Users/d/kolibri/012x/backups', fallback_version = '0.12'
def search_latest(search_root, fallback_version):
logger.info("Searching latest backup in {}...".format(search_root))
newest = None # Should be a path/filename.sqlite3
newest_dtm = None
# All file names have to be according to the fall back version.
prefix = "db-v{}".format(fallback_version)
backups = os.listdir(search_root)
backups = filter(lambda f: f.endswith(".dump"), backups)
backups = filter(lambda f: f.startswith(prefix), backups)
# Everything is sorted alphanumerically, and since dates in the
# filenames behave accordingly, we can now traverse the list
# without having to access meta data, just use the file name.
backups = list(backups)
backups.sort()
for backup in backups:
try:
dtm = get_dtm_from_backup_name(backup)
except ValueError:
continue
# Always pick the newest version
> if is_full_version(backup) or dtm > newest_dtm:
E TypeError: '>' not supported between instances of 'str' and 'NoneType'
kolibri/core/deviceadmin/utils.py:176: TypeError
------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------
INFO Beginning database restore
INFO Searching latest backup in /Users/d/kolibri/012x/backups...
________________________________________________________________ test_restore_from_latest ________________________________________________________________
@pytest.mark.django_db
@pytest.mark.filterwarnings('ignore:Overriding setting DATABASES')
def test_restore_from_latest():
"""
Tests that we cannot restore while kolibri is active
"""
if not is_sqlite_settings():
return
with patch(
"kolibri.utils.server.get_status",
side_effect=mock_status_not_running
):
# Create something special in the database!
from kolibri.core.auth.models import Facility
Facility.objects.create(name="test latest", kind=FACILITY)
# Create a backup file from the current test database
call_command("dbbackup")
# Also add in a file with an old time stamp to ensure its ignored
sql = "syntax error;"
fbroken = "db-v{}_2015-08-02_00-00-00.dump".format(kolibri.__version__)
open(os.path.join(default_backup_folder(), fbroken), "w").write(sql)
# Add an unparsable file name
fbroken = "db-v{}_.dump".format(kolibri.__version__)
open(os.path.join(default_backup_folder(), fbroken), "w").write(sql)
# Restore it into a new test database setting
with override_settings(DATABASES=MOCK_DATABASES):
from django import db
# Destroy current connections and create new ones:
db.connections.close_all()
db.connections = db.ConnectionHandler()
> call_command("dbrestore", latest=True)
kolibri/core/deviceadmin/tests/test_dbrestore.py:152:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv/lib/python3.6/site-packages/django/core/management/__init__.py:131: in call_command
return command.execute(*args, **defaults)
.venv/lib/python3.6/site-packages/django/core/management/base.py:330: in execute
output = self.handle(*args, **options)
kolibri/core/deviceadmin/management/commands/dbrestore.py:76: in handle
use_backup = search_latest(search_root, fallback_version)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
search_root = '/Users/d/kolibri/012x/backups', fallback_version = '0.12'
def search_latest(search_root, fallback_version):
logger.info("Searching latest backup in {}...".format(search_root))
newest = None # Should be a path/filename.sqlite3
newest_dtm = None
# All file names have to be according to the fall back version.
prefix = "db-v{}".format(fallback_version)
backups = os.listdir(search_root)
backups = filter(lambda f: f.endswith(".dump"), backups)
backups = filter(lambda f: f.startswith(prefix), backups)
# Everything is sorted alphanumerically, and since dates in the
# filenames behave accordingly, we can now traverse the list
# without having to access meta data, just use the file name.
backups = list(backups)
backups.sort()
for backup in backups:
try:
dtm = get_dtm_from_backup_name(backup)
except ValueError:
continue
# Always pick the newest version
> if is_full_version(backup) or dtm > newest_dtm:
E TypeError: '>' not supported between instances of 'str' and 'NoneType'
kolibri/core/deviceadmin/utils.py:176: TypeError
------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------
Installed 2 object(s) from 1 fixture(s)
Backed up database to: /Users/d/kolibri/012x/backups/db-v0.12.0a2.dev0+git.222.g81cb6240_2019-01-15_17-29-20.dump
------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------
INFO Beginning database restore
INFO Searching latest backup in /Users/d/kolibri/012x/backups...
____________________________________________________________ test_restore_from_file_to_memory ____________________________________________________________
@pytest.mark.django_db
@pytest.mark.filterwarnings('ignore:Overriding setting DATABASES')
def test_restore_from_file_to_memory():
"""
Restores from a file dump to a database stored in memory and reads contents
from the new database.
"""
if not is_sqlite_settings():
return
with patch(
"kolibri.utils.server.get_status",
side_effect=mock_status_not_running
):
# Create something special in the database!
from kolibri.core.auth.models import Facility
Facility.objects.create(name="test file", kind=FACILITY)
# Create a backup file from the current test database
dest_folder = tempfile.mkdtemp()
> backup = dbbackup(kolibri.__version__, dest_folder=dest_folder)
kolibri/core/deviceadmin/tests/test_dbrestore.py:175:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
kolibri/core/deviceadmin/utils.py:106: in dbbackup
for line in db.connections['default'].connection.iterdump():
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
connection = <sqlite3.Connection object at 0x107bc6810>
def _iterdump(connection):
"""
Returns an iterator to the dump of the database in an SQL text format.
Used to produce an SQL dump of the database. Useful to save an in-memory
database for later restoration. This function should not be called
directly but instead called from the Connection method, iterdump().
"""
cu = connection.cursor()
yield('BEGIN TRANSACTION;')
# sqlite_master table contains the SQL CREATE statements for the database.
q = """
SELECT "name", "type", "sql"
FROM "sqlite_master"
WHERE "sql" NOT NULL AND
"type" == 'table'
ORDER BY "name"
"""
schema_res = cu.execute(q)
for table_name, type, sql in schema_res.fetchall():
if table_name == 'sqlite_sequence':
yield('DELETE FROM "sqlite_sequence";')
elif table_name == 'sqlite_stat1':
yield('ANALYZE "sqlite_master";')
elif table_name.startswith('sqlite_'):
continue
# NOTE: Virtual table support not implemented
#elif sql.startswith('CREATE VIRTUAL TABLE'):
# qtable = table_name.replace("'", "''")
# yield("INSERT INTO sqlite_master(type,name,tbl_name,rootpage,sql)"\
# "VALUES('table','{0}','{0}',0,'{1}');".format(
# qtable,
# sql.replace("''")))
else:
yield('{0};'.format(sql))
# Build the insert statement for each row of the current table
table_name_ident = table_name.replace('"', '""')
res = cu.execute('PRAGMA table_info("{0}")'.format(table_name_ident))
column_names = [str(table_info[1]) for table_info in res.fetchall()]
q = """SELECT 'INSERT INTO "{0}" VALUES({1})' FROM "{0}";""".format(
table_name_ident,
",".join("""'||quote("{0}")||'""".format(col.replace('"', '""')) for col in column_names))
> query_res = cu.execute(q)
E sqlite3.OperationalError: database table is locked: kolibriauth_collection
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sqlite3/dump.py:55: OperationalError
------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------
Installed 2 object(s) from 1 fixture(s)
_____________________________________________________________ test_restore_from_file_to_file _____________________________________________________________
@pytest.mark.django_db
@pytest.mark.filterwarnings('ignore:Overriding setting DATABASES')
def test_restore_from_file_to_file():
"""
Restores from a file dump to a database stored in a file and reads contents
from the new database.
"""
if not is_sqlite_settings():
return
with patch(
"kolibri.utils.server.get_status",
side_effect=mock_status_not_running
):
# Create something special in the database!
from kolibri.core.auth.models import Facility
Facility.objects.create(name="test file", kind=FACILITY)
# Create a backup file from the current test database
dest_folder = tempfile.mkdtemp()
# Purposefully destroy the connection pointer, which is the default
# state of an unopened connection
from django import db
db.connections['default'].connection = None
> backup = dbbackup(kolibri.__version__, dest_folder=dest_folder)
kolibri/core/deviceadmin/tests/test_dbrestore.py:210:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
kolibri/core/deviceadmin/utils.py:106: in dbbackup
for line in db.connections['default'].connection.iterdump():
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
connection = <sqlite3.Connection object at 0x105d2f1f0>
def _iterdump(connection):
"""
Returns an iterator to the dump of the database in an SQL text format.
Used to produce an SQL dump of the database. Useful to save an in-memory
database for later restoration. This function should not be called
directly but instead called from the Connection method, iterdump().
"""
cu = connection.cursor()
yield('BEGIN TRANSACTION;')
# sqlite_master table contains the SQL CREATE statements for the database.
q = """
SELECT "name", "type", "sql"
FROM "sqlite_master"
WHERE "sql" NOT NULL AND
"type" == 'table'
ORDER BY "name"
"""
schema_res = cu.execute(q)
for table_name, type, sql in schema_res.fetchall():
if table_name == 'sqlite_sequence':
yield('DELETE FROM "sqlite_sequence";')
elif table_name == 'sqlite_stat1':
yield('ANALYZE "sqlite_master";')
elif table_name.startswith('sqlite_'):
continue
# NOTE: Virtual table support not implemented
#elif sql.startswith('CREATE VIRTUAL TABLE'):
# qtable = table_name.replace("'", "''")
# yield("INSERT INTO sqlite_master(type,name,tbl_name,rootpage,sql)"\
# "VALUES('table','{0}','{0}',0,'{1}');".format(
# qtable,
# sql.replace("''")))
else:
yield('{0};'.format(sql))
# Build the insert statement for each row of the current table
table_name_ident = table_name.replace('"', '""')
res = cu.execute('PRAGMA table_info("{0}")'.format(table_name_ident))
column_names = [str(table_info[1]) for table_info in res.fetchall()]
q = """SELECT 'INSERT INTO "{0}" VALUES({1})' FROM "{0}";""".format(
table_name_ident,
",".join("""'||quote("{0}")||'""".format(col.replace('"', '""')) for col in column_names))
> query_res = cu.execute(q)
E sqlite3.OperationalError: database table is locked: kolibriauth_collection
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sqlite3/dump.py:55: OperationalError
------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------
Installed 2 object(s) from 1 fixture(s)
========================================================== 5 failed, 6 passed in 10.26 seconds ===========================================================
Expected behavior
tests pass
User-facing consequences
confusing false failures
Steps to reproduce
check out latest develop, run
pytest kolibri/core/deviceadmin/tests/test_dbrestore.py
Context
- macos
- python 3
Issue Analytics
- State:
- Created 5 years ago
- Comments:6 (6 by maintainers)
Top Results From Across the Web
Why tests pass locally but fail in Jenkins | Object Partners
Symptom: Tests pass locally when run in a certain order, but fail in Jenkins when run in a different order (or tests pass/fail...
Read more >Unit test fails in Azure DevOps but passes locally
When tests are run using Test Explorer "Run all" it works locally, but when tests are run as part of the build process...
Read more >Locally-run tests pass, but Jenkins tests fail; why, and how can ...
After Googling around a little for an answer, I found that apparently no one has come up with an accepted answer, either on...
Read more >Why Do My Tests Pass Locally but Fail on CircleCI?
If your tests only pass when run in a certain order, they may fail on CircleCI. "Out of Memory" errors. If a process...
Read more >devtools::test(...) passes but tests fail in covr ... - GitHub
Running my tests of recetox-xMSannotator using devtools::test() works completely fine, but calling covr::package_coverage() fails in finding local test data.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
searches head and scratches issues rigorously
edit: confirmed that the thing I previously posted in this comment was unrelated