Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Cloudformation TM Way Ahead

See original GitHub issue

👋 @hotosm/tech

Last week, we saw a pretty significant performance improvement in the tasking manager. By changing gunicorn to use threads instead of synchronous workers, the tasking manager is able to:

Take on more requests, and complete them asynchronously in a shorter period of time.
Use smaller and cheaper AWS instances
Use a smaller and cheaper RDS instance

This also means that we’ll have more detailed metrics, and ways to monitor our stacks.

Next steps to wrap up the CloudFormation work

We have a few key decisions to make as we wrap up this migration.

Please weigh in with your thoughts on the following, @xamanu, @smit1678, @dakotabenjamin and the rest of @hotosm/tech-staff, so we can wrap up the CloudFormation migration in a way that makes the most sense.

Upgrade to an ALB (Application load balancer): Currently, we are using an elastic load balancer. We can migrate to a more advanced “Application Load Balancer” to take advantage of better metrics and load balancer performance and customisation.
Take a call on our deployment strategy: Moving to CloudFormation from ElasticBeanstalk would require us to change our deployment methods. Currently, our deploys are tied together tightly with CircleCI. This means, a code merge with the develop branch creates an automated deploy to our staging stack, and a code merge with master creates an automated deploy to the production stack. After the switch, we have one of two options:
- Easy: We would have to manually deploy to a staging stack using cfn-config and after testing the changes, manually deploy to a production stack using cfn-config. This would mean that we no longer do code merges. We always
  - Use cfn-config to deploy to the staging stack from the development branch, or the branch used to create a PR. (before the merge)
  - If things look good on staging, we merge the branch with master, and deploy to production. Anything merged with master is always final, and in line with production. So, a merge with master is always accompanied with a deploy to production
- Hard If we want to retain the automated deployment model we currently follow, we would have to build this with AWS Code Deploy, using AWS Code Config. I would prefer us going ahead with the easy deployment strategy for our first iteration, but document the process really well, so that anybody on the team is able to deploy.
Spring cleaning Decide on maintaining EBS files and data. Should we dump, or keep?
Cleanup old RDS snapshots and databases We have a ton of old snapshots, and databases that are stale. We can dump several of these.
Spot instances We have a call with our AWS manager tomorrow, when we can make a decision about this. This will be a org-wide push, and will not be implemented for the first iteration of the tasking manager.