How to drain an ECS instance

While troubleshooting the ECS cluster you may notice some EC2 instances might be unhealthy due to some processes being in a stuck state. Here I’m going to show how to drain an ECS instance by replicating it with a new one. Follow these steps when the ECS instance is unhealthy, unreachable or AWS had a notice that will retire it.

Prerequisites

• AWS account
• ECS cluster

Before draining the wobbly EC2 instance, we will create a new one for replication.

Add new instance to the ECS cluster

1. Open the Amazon EC2 console.
2. In the navigation pane, choose Auto Scaling Groups.
3. Select for editing the one you want to add an instance to it.
4. Increase the number of Desired Capacity and Min for one (e.g. change it to 7 if it was 6).
   
5. Wait for the creation of the new instance to be done.

Drain the unhealthy instance

1. In the EC2 Dashboard check the ID of the instance you want to drain.
2. Open the Amazon ECS console and choose the cluster.
3. Choose ECS Instances tab and select the checkbox for the container instance.
4. Choose Actions, Drain instances and finally, click Done.
   
5. Wait until the Running tasks for the instance to be 0.

Remove the unhealthy instance

1. Open the Amazon EC2 console.
2. Terminate the unhealthy instance that you just drained.
3. Wait for the instance state to be terminated.

Lower the number of instances in the Auto Scaling Group for one

1. Open the Amazon EC2 console.
2. In the navigation pane, choose Auto Scaling Groups.
3. Select for editing the one you want to lower the number of the instances.
4. Lower the number of Desired Capacity and Min for one (e.g. change it to 6 if it was 7).

Conclusion

It’s important to add a new instance before draining the unhealthy one, cause that way the ECS agent can spread the containers and avoid some potential bottlenecks. Feel free to leave a comment below and if you find this tutorial useful, follow our official channel on Telegram.