AWS Elastic Beanstalk Worker Environment Support

AWS Elastic Beanstalk has a useful feature “Worker Environments”, to separate the scale out and management of infrastructure for long running/background work handling. This automatically takes care of a number of things like pulling items from the SQS queue and sending failed jobs to a Dead Letter Queue, as a consequence it doesn’t work directly with any existing Django task/worker system. Django-Q is already the easiest task system to get up and running with, so it would be really awesome if it had functionality to support taking full advantage of Elastic Beanstalk scaling out background work automatically, to further take away the pain of properly doing long running work in the background.

From my reading of the AWS Elastic Beanstalk Worker Environment Documentation and referencing the architecture section of the Django-Q documentation it looks like supporting this will require:

• Possibly creating a new broker type.
• Definitely creating a new daemon, like qcluster, but using a modified versions of both the monitor and pusher components in order to handle tasks being pushed in by HTTP web requests rather than pulled in from the broker. Fundamentally it would:
  • require the “pusher” to run some kind of web server, listening for incoming HTTP requests from the worker environment task daemon.
  • require the “pusher” to ‘hold on to’ the HTTP request while the task is on the internal task queue.
  • require the “pusher” to monitor the result queue and respond with a HTTP 200 OK when the task is done
  • require the monitor to not report back delivery receipts since this would be handled by the pusher/web server
  • require a version of, or alternative to the SQS broker that understands how to find the configuration for the environment if auto configured, and how to use the elastic beanstalk worker environment SQS as a task broker.

It seems like a lot, but from the time I’ve already spent reading the django-q code, I think most of the work will probably be making an alternative to qcluster that contains a web server and connecting that to the existing pieces the correct way, and much of the existing code will be unchanged. The primary changes being modifying Sentinel.spawn_pusher to support spawning a different pusher which would also be passed the self.results_queue, if using the Elastic Beanstalk broker… and then writing an alternative ‘pusher’ that can do what is needed regarding handling the HTTP requests.

Thoughts?

[vlntsolo]

@RachellCalhoun yes, please see the gist https://gist.github.com/vlntsolo/34261e6026ac0e303c40c6ece9961182

[timomeara]

@timomeara could you please give me a hint. Besides the reverted file names, is there anything that can cause the problems? I am still getting TimeoutError on deploying.

where is it timing out? look at the eb-activity log, you should see each command in your .config look for something like:

... Command 04_mkdir_for_log_and_pid] : Starting activity...
... Command 04_mkdir_for_log_and_pid] : Completed activity.
... Command 05_djangoq_configure] : Starting activity...
... Command 05_djangoq_configure] : Completed activity.
... Command 06_djangoq_run] : Starting activity...
... Command 06_djangoq_run] : Completed activity. Result:

my .config runs the django-q scripts last, after migrate, collect static and all that here’s that part:

04_mkdir_for_log_and_pid:
    command: "mkdir -p /var/log/djangoq/ /var/run/djangoq/"
05_djangoq_configure:
    command: "cp .ebextensions/django-q-cluster.sh /opt/elasticbeanstalk/hooks/appdeploy/post/ && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/django-q-cluster.sh"
    cwd: "/opt/python/ondeck/app"
06_djangoq_run:
    command: "/opt/elasticbeanstalk/hooks/appdeploy/post/django-q-cluster.sh"