Prolific recruiter - more specification - 13pts

High Level Task Breakdown/Questions/Mullings

• Prolific API wrapper (prolific.py) and tests(!) Given that there’s no sandbox, I think some kind of isolation tests will be the best we can do. Probably wise not to get too ambitious here.
• Iron out details of recruiter
  • close_recruitment() - do we want to implement this?
  • Change signature of Recruiter.reward_bonus to take the worker ID (may as well pass all the things a recruiter might need)
  • Implement reward_bonus()
  • Implement recruit() based on new Prolific API (see below… it’s a PATCH call). We’ll need a way to determine the current number of participants, because the way this Prolific API works is to provide a new total, not a number to increment by
  • We’ll need self.is_in_progress (check if a Study/HIT is already running when calling open_recruitment()) Maybe we store the current study ID in redis the same way we now do for the MTurkRecruiter.
  • I don’t think we can re-use reward, because iirc in Prolific this is an hourly rate, not a lump sum
• Config system: which variables belong as Dallinger vars, and which get dumped in prolific_recruitment_config.json or whatever we want to call it
• Questions:
  • Do we want to use prolific_ as a config var prefix, or prolific:? I prefer the later, since it’s clearer it’s a prefix, but the mturk keys don’t follow this, so maybe that ship is sailed if we prefer to maximize consistency
  • How do we decide which configuration keys go in dallinger config variables? Maybe the answer is “as few as possible”? Only ones that are “generic” (title, description, base_pay, etc.)? Or maybe the required ones get top-level status and the optional ones go in the special file?
  • Ideas on how to determine total current participants (see above)? ~Persist it?~ Query it?
• Incorporate PR feedback
• Do we want/need Prolific API integration tests to run on CI? Answer: YES
  • Get Prolific API token into Github CI secrets
  • Update CI test config to run with the required pytest flags
• Get CI passing if possible - currently docker tests fail because of new configuration keys
• Documentation
  • Some general docs about Prolific recruitment lifecycle
  • Other?
• Clean up

---

This issue continues (and replaces) #3499. It contains more detailed specification of the implementation.

Preliminaries:

1. Prolific API can be found here: https://www.prolific.co/docs/api-docs/public/
2. We have a Dallinger account with an API token, slack me if you need it.
3. We have a computational audition account + API token that can be also used for additional testing (slack me if you need it).
4. We have a joining slack channel for support with the prolific integration team, please let me know if you want to join it.
5. Previously we thought that prolific is not supporting dynamic recruiting (namely changing the number of participants after the survey is started). In the API documentation this is explicitly mentioned as impossible. However, discussing the integration with the Prolific people they told us that they will share with us an undocumented feature that allow to update recruitment after the survey had started. I am giving the detailed answer near the “recruit” method.
6. The API of prolific is versioned, we should have a way to define the prolific API version via a parameter. Currently we only plan to actively support the current API version (v01).
7. Prolific does not have a sandbox environment - to do testing we will have to use active participants. Nori will reserve funds for that, and you can use these funds for testing. If more money is needed we can set this up.

Recruiting parameters:

1. All the main recruiting parameters should be specified in the config.txt under the [prolific] section. Recruiting parameters that are relevant to prolific should have a name that starts with “prolific_” (namely: prolific_parameter_name = value)
2. Some parameters like “description” seems to overlap so much that we think that can be directly used (namely use the same variable name for Mturk and Prolific experiments, as the description is really the same thing across recruiters).
3. Secondary recruiting parameters are going to be stored in a file specified in prolific_recruitment_config = filename. This allows adding optional parameters, without declaring them on dallinger, allowing also for API changes in prolific without changing much in dallinger. This is similar to how we treat an additional qualification file in the current dallinger. Use something like: recruitment_config=json.loads(self.config.get(“prolific_recruitment_config”))
4. The prolific API token should be specified in .dallinger_config.txt in the same way that the Mturk credentials are specified.
5. There should be a way for the recruiter class to access this security token as it is needed in order to make API calls.

Recruiter methods to implement:

Open_recruitment should call “Create a draft study” (https://www.prolific.co/docs/api-docs/public/#tag/Studies/paths/~1api~1v1~1studies~1/post) Here is a list of the parameters.

- Name- specified in the config by the user (config)

• Description- description of the study for the participants to read before starting the study; this is conceptually identical to Mturk’s “description” field and maybe reusing the same variable, as here the two variables really mean the same thing.
• External_study_url- dallinger provides this to Prolific this is the deployed app URL (for example https://dlgr-nori-app.herokuapp.com/)
• Prolific_id_option - should always be “url_parameters”, as we always receive participants with url parameters specifying their prolific ID.
• Completion_code- this is a “secret” code used to finalize the study, supposed to be not known to users before completing the HIT. We need to provide here a long randomized string.
• Completion_option - should be always “url” since we are always redirecting the user rather than giving him a code to type manually.
• Total_available_places - this is initial recruit size
• Estimated_completion_time - need to be specified in the config as a parameter (this parameter maybe can be called “prolific_est_duration” that is different then “duration” that in the M-turk case represents the maximal duration (typically longer than the actual duration).
• Reward-this is like ”base_payment”: in Prolific maybe it should be called prolific_base_payment, alternatively we can reuse the same variable.
• device_compatibility - parameters taken from prolific_recruitment_config
• Peripheral_requirements - parameters taken from prolific_recruitment_config - Eligibility_requirements - parameters taken from prolific_recruitment_config (typically this is empty for everyone).
• status - initially should be ‘UNPUBLISHED’ but then changed to ‘ACTIVE’ at the end of the recruitment. In the end of the process it should also revoke the Publish a draft study API call to finalize.

Normalize_entry_information- It is possible to configure what kind of URL prolific is creating but typically it will have the following form:

?PROLIFIC_PID={{%PROLIFIC_PID%}}&STUDY_ID={{%STUDY_ID%}}&SESSION_ID={{%SESSION_ID%}}
PROLIFIC_PID  is like Mturk’s “worker_id”
STUDY_ID is like “hit_id”
SESSION_ID is like “assignment_id”

The parameters can be translated to these values by the proper normalized entry information. Note that we discovered a related bug in #2652, however in this case this should work nevertheless.

Recruit This should increment the current count of participants. Prolific will provide us with help on how to do it. But this will likely start with the “Get a study” API call to know the current number of participants and based on the number of recruited participants we will increment and update the participant count. So imagine that the function is called with n=10, and we currently have 70 participants recruited as indicated by the call to “Get a study” then the resulting update will be of 80 participants.

Here is the note from prolific how this can be done: Doing a PATCH on the study and modifying the total_available_places is enough to do so.

url = f"{PROLIFIC_API_BASE}/api/v1/studies/{study_id}/"
requests.request(
    "PATCH", url, headers={
        "Authorization": f"Token {prolific_token}",
        "Content-Type": "application/json",
    }, data=json.dumps({"total_available_places": total_available_places + 3})
)

As we said before, this can only be done for normal requirements, not for representative samples or quota studies. The new number of places must be more than the current number of places taken. So increasing always works, but decreasing the number of places might return an error.

A final check we do is that the account must have enough available balance to pay for the extra places.

close_recruitment Change the status of the experiment to ‘PAUSED’ or maybe ‘COMPLETED’ (not sure here?)

Assign_experiment_qualifications This is not going to be implemented in the first stage, we may need another discussion on how to implement qualifications in cases that this is not supported (for example we can create our own key value system like qualification).

Compensate_worker - delay implementation of this function to a later stage (open an issue for it)?

exit_response Should be customized to HIT the completion route. Dallinger should apply this API call upon completion (Execute status transition): https://www.prolific.co/docs/api-docs/public/#tag/Submissions/paths/~1api~1v1~1submissions~1{id}~1transition~1/post

Note that prolific has a getter API function (Get submission details) that allows you to get the status of a submission (a submission is similar to a HIT in mturk). Implementing the execute status transition may start with calling the Get submission details API function to test the validity of the transition (though I am not sure this is needed).

Reward_bonus Use Prolific API call (bonuses) :https://www.prolific.co/docs/api-docs/public/#tag/Bonuses

Notify_duration_exceeded: no need to implement

Rejects_questionnaire_from:not sure if needed to be implemented

Submitted_event:not sure if needed to be implemented.

[jacobyn]

Regarding:

Implement recruit() based on new Prolific API (see below… it’s a PATCH call). We’ll need a way to determine the current number of participants, because the way this Prolific API works is to provide a new total, not a number to increment by

and

Ideas on how to determine total current participants (see above)? Persist it? Query it?

@jessesnyder My initial idea was this, so making another API call to “know” the current recruited people, and then make transparent and identical logic to the current dallinger recruit logic. Namely:

Recruit - should increment the current count of participants. This will likely start with the “Get a study” API call to know the current number of participants and based on the number of recruited participants we will increment and update the participant count. So imagine that the function is called with n=10, and we currently have 70 participants recruited as indicated by the call to “Get a study” then the resulting update will be of 80 participants.

[jessesnyder]

Thanks, @jonasanso - this seems to be working well!