Prototyping `dallinger develop` command with PsyNet: 5pts

The dallinger develop command is very promising and looks like it will save us a lot of time in the future.

When applying it to our own experiments we do have some problems though and they are quite subtle for us to debug. We were hoping that @jessesnyder could help here by trying to run the code himself.

As a starting point, it would be great to trial one of the PsyNet demos, for example static. The PsyNet repository can be found here.

Ordinarily I would run this demo by navigating to the demo directory (demos/static) and run dallinger debug. Here I instead follow the Dallinger develop instructions as specified here.

The first issue I get is an import error, that doesn’t occur with dallinger debug.

(psynet) peter@dhcp-10-248-39-251 develop % dallinger develop launch

❯❯ Error accessing http://127.0.0.1:5000/launch (500):
{"status": "error", "message": "Failed to load experiment in /launch: No module named 'psynet.experiment'"}

This error comes from the top of experiment.py:

# pylint: disable=unused-import,abstract-method

##########################################################################################
# Imports
##########################################################################################

import logging

from flask import Markup

import psynet.experiment

It turns out one can sidestep this issue by copying the static directory outside the PsyNet source code tree and rerunning the developer commands from there. So, this issue is not so crucial, but it would be nice to solve.

When I do this, I can get a few pages through, until ‘Do you want to enable custom stimulus and stimulus version filters?’. I then run into a more serious error that I have been unable to debug yet.

INFO:werkzeug:127.0.0.1 - - [17/Sep/2021 15:00:14] "POST /response HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/peter/git/psynet/psynet/experiment.py", line 946, in route_response
    res = exp.process_response(
  File "/Users/peter/git/psynet/psynet/experiment.py", line 630, in process_response
    self.timeline.advance_page(self, participant)
  File "/Users/peter/git/psynet/psynet/timeline.py", line 1329, in advance_page
    new_elt.consume(experiment, participant)
  File "/Users/peter/git/psynet/psynet/timeline.py", line 230, in consume
    target_elt = self.get_target(experiment, participant)
  File "/Users/peter/git/psynet/psynet/timeline.py", line 264, in get_target
    val = call_function(
  File "/Users/peter/git/psynet/psynet/utils.py", line 107, in call_function
    return function(*arg_values)
  File "/Users/peter/git/psynet/psynet/trial/main.py", line 1533, in <lambda>
    lambda experiment, participant: self._get_current_trial(participant)
  File "/Users/peter/git/psynet/psynet/trial/main.py", line 1489, in _get_current_trial
    return self.trial_class.query.get(trial_id)
  File "<string>", line 2, in get
    
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/sqlalchemy/util/deprecations.py", line 390, in warned
    return fn(*args, **kwargs)
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 944, in get
    return self._get_impl(ident, loading.load_on_pk_identity)
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 948, in _get_impl
    return self.session._get_impl(
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2839, in _get_impl
    return db_load_fn(
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 568, in load_on_pk_identity
    return result.one()
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/sqlalchemy/engine/result.py", line 1374, in one
    return self._only_one_row(
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/sqlalchemy/engine/result.py", line 558, in _only_one_row
    row = onerow(hard_close=True)
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/sqlalchemy/engine/result.py", line 1238, in _fetchone_impl
    return self._real_result._fetchone_impl(hard_close=hard_close)
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/sqlalchemy/engine/result.py", line 1626, in _fetchone_impl
    row = next(self.iterator, _NO_ROW)
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 150, in chunks
    rows = [proc(row) for row in fetch]
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 150, in <listcomp>
    rows = [proc(row) for row in fetch]
  File "/Users/peter/opt/miniconda3/envs/psynet/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 1300, in polymorphic_instance
    raise sa_exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Row with identity key (<class 'melody.psychophysics.RatingPitchIntervalTrial'>, (2,), None) can't be loaded into an object; the polymorphic discriminator column 'info.type' refers to mapped class RatingPitchIntervalPractiseTrial->info, which is not a sub-mapper of the requested mapped class RatingPitchIntervalPractiseTrial->info

The responsible line of code is this:

return self.trial_class.query.get(trial_id)

The trial_class is defined in experiment.py. I can successfully retrieve this Trial object if I instead use a class defined in the psynet package:

return Trial.query.get(trial_id)

Looking at the classes of these objects:

Trial.query.get(trial_id).__class__
# <class 'dallinger_experiment.experiment.AnimalTrial'>

self.trial_class
# <class 'develop.experiment.AnimalTrial'>

Somehow AnimalTrial is getting registered in SQLAlchemy under dallinger_experiment.experiment.AnimalTrial instead of under develop.experiment.AnimalTrial, but I’m not sure how/why this is happening. I hope maybe @jessesnyder can figure it out!

PS I tried to reproduce the equivalent problem with the Dallinger MCMCP demo but had no luck.

[jessesnyder]

I feel like I really close, but am missing some bit of insight. @pmcharrison, I’m confident you were correct in your initial suspicion about the problem being rooted in the different fully-qualified class names:

Trial.query.get(trial_id).__class__
# <class 'dallinger_experiment.experiment.AnimalTrial'>

# vs

self.trial_class
# <class 'develop.experiment.AnimalTrial'>

When SQLAlchemy wants to instantiate a python object from a database row, it does a check of identity between two classes (and also if one class is a subclass of another) and we fail both checks.

From _get_current_trial() into SQLAlchemy object loading:

# classes superficially look the same:
sqlalchemy.orm.loading:sub_mapper: mapped class AnimalTrial->info, mapper: mapped class AnimalTrial->info

# but they're not, when their package/namespace is included in the comparison:
sqlalchemy.orm.loading:sub_mapper.class_: <class 'dallinger_experiment.experiment.AnimalTrial'>, mapper.class_: <class 'develop.experiment.AnimalTrial'>

# so False is returned, leading to the exception shortly afterwards
sqlalchemy.orm.loading:return False

The problems originate with creating AnimalTrialMaker instances in the experiment’s experiment.py module. In both dallinger debug and when running Flask directly (“develop mode”), more than one instance is created, since this happens whenever the module is imported. Also in both cases, sometimes the fully qualified class of AnimalTrial that’s passed to the AnimalTrialMaker constructor has a non-standard (meaning “anything other than dallinger_experiment.experiment.AnimalTrial”) class name. The difference is in what happens later.

For some reason, when running dallinger debug, the AnimalTrialMaker instance on which _get_current_trial() is called is always one with the correct AnimalTrial class assigned to self.trial_class, while when running “develop mode”, it’s aways an instance with the wrong AnimalTrial class.

dallinger debug

# wrong (dallinger cli process)
2021-11-18 12:24:57,223 - root - INFO - In TrialMaker.__init__, trial_class arg: <class 'TEMP_VERIFICATION_PACKAGE.experiment.AnimalTrial'>
2021-11-18 12:24:57,223 - root - INFO - self: <TEMP_VERIFICATION_PACKAGE.experiment.AnimalTrialMaker object at 0x1078aa490>

# right (dallinger cli process)
2021-11-18 12:24:57,229 - root - INFO - In TrialMaker.__init__, trial_class arg: <class 'dallinger_experiment.experiment.AnimalTrial'>
2021-11-18 12:24:57,229 - root - INFO - self: <dallinger_experiment.experiment.AnimalTrialMaker object at 0x11da825e0>

# wrong (web process)
12:25:00 PM web.1   |  INFO:root:In TrialMaker.__init__, trial_class arg: <class 'de8c0017-f5ff-c4ee-925c-0cf5c4b5ecbd.experiment.AnimalTrial'>
12:25:00 PM web.1   |  INFO:root:self: <de8c0017-f5ff-c4ee-925c-0cf5c4b5ecbd.experiment.AnimalTrialMaker object at 0x109bcf070>

# right (web process)
12:25:00 PM web.1   |  INFO:root:In TrialMaker.__init__, trial_class arg: <class 'dallinger_experiment.experiment.AnimalTrial'>
12:25:00 PM web.1   |  INFO:root:self: <dallinger_experiment.experiment.AnimalTrialMaker object at 0x11ff4bf40>

# wrong (worker process)
12:25:02 PM worker.1 |  2021-11-18 12:25:02,258 In TrialMaker.__init__, trial_class arg: <class 'de8c0017-f5ff-c4ee-925c-0cf5c4b5ecbd.experiment.AnimalTrial'>
12:25:02 PM worker.1 |  2021-11-18 12:25:02,259 self: <de8c0017-f5ff-c4ee-925c-0cf5c4b5ecbd.experiment.AnimalTrialMaker object at 0x1200cc610>

# then, the critical moment, when participant clicks that magic button, an AnimalTrialMaker instance with the right class is used
12:25:58 PM web.1   |  INFO:root:IN _get_current_trial, self.trial_class: <class 'dallinger_experiment.experiment.AnimalTrial'>
12:25:58 PM web.1   |  INFO:root:self: <dallinger_experiment.experiment.AnimalTrialMaker object at 0x11ff4bf40>

./run.sh (with not clock, for simplicity)

# wrong
INFO:root:In TrialMaker.__init__, trial_class arg: <class 'develop.experiment.AnimalTrial'>
INFO:root:self: <develop.experiment.AnimalTrialMaker object at 0x10a4c3820>

# wrong
INFO:root:In TrialMaker.__init__, trial_class arg: <class 'develop.experiment.AnimalTrial'>
INFO:root:self: <develop.experiment.AnimalTrialMaker object at 0x10d18a520>

# right
INFO:root:In TrialMaker.__init__, trial_class arg: <class 'dallinger_experiment.experiment.AnimalTrial'>
INFO:root:self: <dallinger_experiment.experiment.AnimalTrialMaker object at 0x1234ce310>

# but in _get_current_trial, the wrong AnimalTrialMaker is used
INFO:root:IN _get_current_trial, self.trial_class: <class 'develop.experiment.AnimalTrial'>
INFO:root:self: <develop.experiment.AnimalTrialMaker object at 0x10d18a520>

Taking a break and hoping lightning will strike.

[pmcharrison]

Hi @jessesnyder, finding this quite confusing, as when I run dallinger debug I’m finding that get_current_trial() is definitely called as soon as I respond to the first question, see video:

https://user-images.githubusercontent.com/22684998/142463156-b1fc6e95-347d-4fe9-99c3-fff405be967a.mp4