Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Automatically blacklist workers who auto-accept HITs but don't start them

See original GitHub issue

Some workers use auto-accepting scripts to quickly accept HITs as they are published. However, this typically adds the HIT to the workers queue and there can be a significant delay before they start the work. This is fine in some cases (and these workers do good work) but it causes problems when experiments rely on groups of participants completing the work synchronously as groups can take a long time to fill.

The current solution is to over-recruit by as much as 50%, thus there is excess recruitment space that the auto-accepting workers can fill, while other participants can take part. However, this is inefficient for two reasons: (1) it is hard to know how much to over-recruit by, and (2) over-recruitment typically leads to excess participant fees as participants who arrive after all networks are full are still paid for trying to take part.

I suggest the following solution: Dallinger automatically detects auto-accepting participants, and gives them a hidden qualification that optionally blocks them from future HITs put up by the experimenter.

FAQ: How does Dallinger detect auto-accepting participants? A background process can monitor for any participants who arrive >10 minutes after their assignment accepted notification arrived.

How does Dallinger assign the qualification? Dallinger can already do this.

How does the qualification block participants? It can be automatically added to the qualification_blacklist variable at runtime

How is the blocking made optional? The config file can have a boolean parameter that determines whether the qualification is added to the qualification_blacklist variable at runtime, e.g. block_autoaccepting_workers = true

What about false positives? We can give each participant “three lives”, and each time they don’t start working within 10 minutes the value of their qualification increases by 1. We can then block only those workers who have the qualification with a value of 3 or more.

@jacobyn @lottybrand @jessesnyder What do you think?

Issue Analytics

  • State:open
  • Created 3 years ago
  • Comments:10 (5 by maintainers)

github_iconTop GitHub Comments

thomasmorgancommented, Oct 5, 2020

Thanks @pmcharrison. That works fine, except in synchronous group experiments where the auto-accepting workers hold up their entire group (the group cannot start until everyone has arrived) which causes everyone to expire, even those who have been patiently waiting the whole time.

thomasmorgancommented, Oct 9, 2020

Ah ok, I’ve been running a lot of grouped experiments which have quite high attrition rates, so the graceful catching is really useful. Still, it would be pretty easy to give everyone who finishes a bonus even if their data is garbled.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Employment Laws on Blacklisting - Work -
Recruiters blacklist job seekers for skipping interviews, failing background checks, inflating their qualifications and lying on resumes.
Read more >
How to Use iPhone Auto-Answer | PCMag
1. Update Your Phone 2. Call Audio Routing 3. Auto-Answer Calls
Read more >
if I don't accept calendar invitations, will people assume I won't ...
Open the window and go to the tracking tab. You can see who has accepted or not. I prefer to do it this...
Read more >
Living as a Turker - Notes From Below
Some scripts are used to automatically catch HITs as they are posted on the site. Most Turkers use PANDAs (Preview And Auto Accept)...
Read more >
As a shopper, is there any way to auto-accept an order on ...
You don't want to auto-accept orders from InstaCart because not all orders ... I work 6 days a week and do not want...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found