Provides infrastructure layer compute capabilities, including both bare metal and virtual servers with various optimizations includins compute, memory, IO, and disk. Also supports accelerations options such as GPUs, FPGAs, Inferentia and Trainium.
Provides image recognition capability for images (in batch or real-time) and video that provides a analysis of the content such as real-world objects, faces, celebrities, and path mapping.
Provides a publish/subscribe notification service with multiple subscription types including Amazon Simple Queue Service (SQS), Amazon Kinesis Data Firehose, AWS Lambda, generic HTTPS endpoints, SMS and email.
A serverless, fully-managed, message queue service that supports producing, store, and consuming messages and enables loose coupling between applications.
Provides private networking capability spanning multiple availability zones and supporting subnets, routing, network access control groups, security groups and gateways.
Provides tracing of service invocations in distributed applications for observability, allowing users to diagnose issues or optimize their service interactions.
All about Cloud, mostly about Amazon Web Services (AWS)
Termination Protection Gotcha!
2017-12-06 / 430 words / 3 minutes
Termination Protection is a feature designed to prevent accidental deletion of AWS resources. The AWS Management Console added support for Termination protection for Amazon Elastic Compute Cloud (EC2) instances has been around since 2011. The concept is simple. The resource has a flag. The AWS API checks the flag before deleting the resource. If the flag is set the deletion fails.
An Auto Scaling group contained two EC2 instances. While I’m usually a huge advocate of automation and immutable infrastructure, one of those instances requires a long tedious manual configuration process which relied on favors from other teams. It wasn’t possible to ask for their help to reconfigure the other instance in the same way, but without those modifications, the application wouldn’t run properly. Success or failure would depend on where the Application Load Balancer (ALB) routed the traffic. I also didn’t want to have an EC2 instance sitting around unused.
I remembered that I could use termination protection and then reduce the Auto Scaling group min, max and desired count from two to one. The documentation, found here, stated:
You can also use instance protection to prevent Auto Scaling from selecting specific instances for termination when scaling in.
The manually configured EC2 instance was identified then EC2 Termination Protection was set. I then reduced the Auto Scaling group counts and watched the AWS Console with horror as my manually configured EC2 instance was shutdown.
The Postmortem
I quickly tried a few things to see if I could interrupt the process, but nothing prevented the loss of that instance. I went back to the documentation and read it again more carefully. The documentation stated:
To change the instance protection setting for an instance using the console
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
On the navigation pane, choose Auto Scaling Groups.
Select the Auto Scaling group.
On the Instances tab, select the instance.
To enable instance protection, choose Actions, Instance Protection, Set Scale In Protection. When prompted, choose Set Scale In Protection.
The termination protection process for Auto Scaling groups was completely different from the termination protection process for EC2 instances! From the Auto Scaling groups part of the AWS Console, “Scale In Protection” replaces “termination protection“.
There’s a saying “A smart man learns from his own mistakes, but a wise man learns from someone else’s“. I certainly have learned from this, and I hope you have learned too!
All data and information provided on this site is for informational
purposes only. cloudninja.cloud makes no representations as to accuracy,
completeness, currentness, suitability, or validity of any information
on this site and will not be liable for any errors, omissions, or
delays in this information or any losses, injuries, or damages
arising from its display or use. All information is provided on an
as-is basis.
This is a personal weblog. The opinions expressed here represent my
own and not those of my employer. My opinions may change over time.
