If your microservices are using ECS services and the deployments are going through CloudFormation, that sometimes can cause weird issues. For example, if there is an application error, missing some environment variables, health checks, etc. In that situation, CloudFormation default behavior is to rollback the template to the previous state, but sometimes the CF stack can be stuck, and after a long time can throw UPDATE_ROLLBACK_FAILED state.
There are a lot of tutorials that explain how to fix that by simply clicking on Continue update rollback next Resources to skip - section and selecting the resources that you want to skip, but when it comes to ECS deployment it may not work.
Prerequisites
- AWS account
Fixing the CF stack
Step 1. Open the ECS service, choose the cluster and find the service from the AWS ECS services tab.
Step 2. Once the service is opened click on the Deployments section.
Step 3. Click on the Update button in the right top corner.
Step 4. To get back the stack in a stable state, open the Revision drop-down list and choose some of the previous state numbers (don’t choose the latest). Also make sure that Force new deployment is enabled:
Step 5. You can leave the other configuration as it is, so that way it will trick the CF and force a new deployment to get back the CF stack to the previous stable state.
Conclusion
In this tutorial, we covered all the needed steps to fix ECS services that are deployed through the CloudFormation template which is stuck in UPDATE_ROLLBACK_FAILED state. Feel free to leave a comment below and if you find this tutorial useful, follow our official channel on Telegram.
