Automatically blacklist workers who auto-accept HITs but don't start them

Some workers use auto-accepting scripts to quickly accept HITs as they are published. However, this typically adds the HIT to the workers queue and there can be a significant delay before they start the work. This is fine in some cases (and these workers do good work) but it causes problems when experiments rely on groups of participants completing the work synchronously as groups can take a long time to fill.

The current solution is to over-recruit by as much as 50%, thus there is excess recruitment space that the auto-accepting workers can fill, while other participants can take part. However, this is inefficient for two reasons: (1) it is hard to know how much to over-recruit by, and (2) over-recruitment typically leads to excess participant fees as participants who arrive after all networks are full are still paid for trying to take part.

I suggest the following solution: Dallinger automatically detects auto-accepting participants, and gives them a hidden qualification that optionally blocks them from future HITs put up by the experimenter.

FAQ: How does Dallinger detect auto-accepting participants? A background process can monitor for any participants who arrive >10 minutes after their assignment accepted notification arrived.

How does Dallinger assign the qualification? Dallinger can already do this.

How does the qualification block participants? It can be automatically added to the qualification_blacklist variable at runtime

How is the blocking made optional? The config file can have a boolean parameter that determines whether the qualification is added to the qualification_blacklist variable at runtime, e.g. block_autoaccepting_workers = true

What about false positives? We can give each participant “three lives”, and each time they don’t start working within 10 minutes the value of their qualification increases by 1. We can then block only those workers who have the qualification with a value of 3 or more.

@jacobyn @lottybrand @jessesnyder What do you think?

[thomasmorgan]

Thanks @pmcharrison. That works fine, except in synchronous group experiments where the auto-accepting workers hold up their entire group (the group cannot start until everyone has arrived) which causes everyone to expire, even those who have been patiently waiting the whole time.

[thomasmorgan]

Ah ok, I’ve been running a lot of grouped experiments which have quite high attrition rates, so the graceful catching is really useful. Still, it would be pretty easy to give everyone who finishes a bonus even if their data is garbled.