If your microservices are using ECS services and the deployments are going through CloudFormation, that sometimes can cause weird issues. For example, if there is an application error, missing some environment variables, health checks, etc… In that situation, CloudFormation default behavior is to
rollback the template to the previous state, but sometimes the CF stack can be stuck, and after a long time can throw
UPDATE_ROLLBACK_FAILED state. There are a lot of tutorials that explain how to fix that by simply clicking on
Continue update rollback next
Resources to skip - section and selecting the resources that you want to skip, but when it comes to ECS deployment it may not work.
- AWS Account
Fixing the CF stack
Step 1. Open the ECS service, choose the cluster and find the service from the AWS ECS services tab
Step 2. Once the service is opened click on the
Step 3. Click on the
Update button on the right top corner
Step 4. To get back the stack in a stable state, open the
Revision drop-down list and choose some of the previous state numbers (don’t choose the
latest) and also make sure that the
Force new deployment is enabled:
Step 5. You can leave the other configuration as it is, so that way it will trick the CF and force a new deployment to get back the CF stack to the previous stable state.
In this tutorial, we covered all the needed steps to fix ECS services that are deployed through the CloudFormation template which is stuck in
UPDATE_ROLLBACK_FAILED state. Feel free to leave a comment below and if you find this tutorial useful, follow our official channel on telegram.