Cloud Ninja
All about Cloud, mostly about Amazon Web Services (AWS)

Reusing Automation : AWS CloudFormation Config Sets

 2017-02-14 /  1254 words /  6 minutes

AWS CloudFormation is a orchestration tool which deploys infrastructure. It supports nested CloudFormation templates. Nested CloudFormation templates provide a way to reuse large components of infrastructure, such as VPCs, Subnets, Auto Scaling Groups and EC2 Instances. CloudFormation Config Sets provide an alternative reuse mechanism, within an EC2 instance.

User Data

Passing a user data script can customize EC2 instances at launch time. The cloud-init process reads and executes the script, which must be less than 16,384 bytes long.

AWS CloudFormation adds the cfn-init process. cfn-init greatly simplifies the customization process and is not subject to the 16,384 byte restriction of the userData script.

CloudFormation Config Sets

A simple cfn-init configuration contains a single configuration section, called “config”. If cfn-init is called without the –c parameter, “default” is invoked by default.

This simple template starts a CentOS AMI in the US-East-1 (N. VA) region. It installs the CloudFormation tools using a user data script, and then runs cfn-init without the –c option (line 43). The CloudFormation configuration then uses the “default” configSet (lines 46-48) and installs MariaDB from the CentOS repository:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

AWSTemplateFormatVersion: "2010-09-09"

Description:
  Flexible Server Configuration

Parameters:
  dbRootPassword:
    Type: String
    Description: Database root password.
  secGrpId:
    Type: String
    Description: Security Group Id.
  subNetId:
    Type: String
    Description: Subnet Id.

Resources:
  FlexServer:
    Type: "AWS::EC2::Instance"
    Properties:
      AvailabilityZone: us-east-1a
      IamInstanceProfile: Backup
      ImageId: ami-6d1c2007 # CentOS 7 - https://aws.amazon.com/marketplace/pp/B00O7WM7QW/&ref_=_mkt_ste_menu
      InstanceType: t2.micro
      KeyName: amazonian
      SecurityGroupIds: # for VPCs
        - !Sub ${secGrpId}
      SubnetId: !Sub ${subNetId}
      UserData:
        "Fn::Base64":
          !Sub |
            #!/bin/bash -xe
            yum -y update
            # Setup cfn-bootstrap
            cd /opt
            curl https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz > aws-cfn-bootstrap-latest.tar.gz
            tar zxvf aws-cfn-bootstrap-latest.tar.gz
            cd aws-cfn-bootstrap-1.4
            python setup.py build
            python setup.py install
            # Run cfn-init
            /bin/cfn-init -v --stack ${AWS::StackName} --resource FlexServer --region ${AWS::Region}
    Metadata:
      AWS::CloudFormation::Init:
        configSets:
          default:       # The "default" configSet is used when cfn-init is called without -c option
            - "mariadb"
        mariadb:         # This is the MariaDB that comes with CentOS Base Repo
          packages:
            yum:
              mariadb-server: []
          files:
            /tmp/mariadb_config_stdin.txt:
              content: !Sub |

                Y
                ${dbRootPassword}
                ${dbRootPassword}
                Y
                Y
                Y
                Y
              mode: "000600"
              owner: "root"
              group: "root"
          commands:
            0100_start_mariadb:
              command: "/bin/systemctl start  mariadb.service"
            0200_run_secure_script:
              command: "/bin/mysql_secure_installation < /tmp/mariadb_config_stdin.txt"
            0300_cleanup:
              command: "rm -vf /tmp/mariadb_config_stdin.txt"
          services:
            sysvinit:
              mariadb:
                enabled: true
                ensureRunning: true

Another example of a simple template also starts a CentOS AMI in the US-East-1 (N. VA) region. It also installs the CloudFormation tools using a user data script, and then also runs cfn-init without the –c option. The CloudFormation configuration then installs the Apache Web Server from the CentOS repository and then PHP:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

AWSTemplateFormatVersion: "2010-09-09"

Description:
  Flexible Server Configuration

Parameters:
  dbRootPassword:
    Type: String
    Description: Database root password.
  secGrpId:
    Type: String
    Description: Security Group Id.
  subNetId:
    Type: String
    Description: Subnet Id.

Resources:
  FlexServer:
    Type: "AWS::EC2::Instance"
    Properties:
      AvailabilityZone: us-east-1a
      IamInstanceProfile: Backup
      ImageId: ami-6d1c2007 # CentOS 7 - https://aws.amazon.com/marketplace/pp/B00O7WM7QW/&ref_=_mkt_ste_menu
      InstanceType: t2.micro
      KeyName: amazonian
      SecurityGroupIds: # for VPCs
        - !Sub ${secGrpId}
      SubnetId: !Sub ${subNetId}
      UserData:
        "Fn::Base64":
          !Sub |
            #!/bin/bash -xe
            yum -y update
            # Setup cfn-bootstrap
            cd /opt
            curl https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz > aws-cfn-bootstrap-latest.tar.gz
            tar zxvf aws-cfn-bootstrap-latest.tar.gz
            cd aws-cfn-bootstrap-1.4
            python setup.py build
            python setup.py install
            # Run cfn-init
            /bin/cfn-init -v --stack ${AWS::StackName} --resource FlexServer --region ${AWS::Region}
    Metadata:
      AWS::CloudFormation::Init:
        configSets:
          default:       # The "default" configSet is used when cfn-init is called without -c option
            - "Apache"
            - "PHP"

        # Install Apache Web Server (httpd)
        Apache:
          packages:
            yum:
              httpd: []
          services:
            sysvinit:
              httpd:
                enabled: true
                ensureRunning: true

        PHP:
          packages:
            yum:
              php: []
              php-mysql: []

Rather than having two different CloudFormation scripts, with “default” sections, we can include “db”, “httpd” and “all” Config Sets, and use the “-c” parameter to select between them. The following CloudFormation template adds a parameter, mode, which switches between the ConfigSets:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105

AWSTemplateFormatVersion: "2010-09-09"

Description:
  Flexible Server Configuration

Parameters:
  dbRootPassword:
    Type: String
    Description: Database root password.
  secGrpId:
    Type: String
    Description: Security Group Id.
  subNetId:
    Type: String
    Description: Subnet Id.
  mode:
    Type: String
    Description: Mode of Install (all, db, httpd)

Resources:
  FlexServer:
    Type: "AWS::EC2::Instance"
    Properties:
      AvailabilityZone: us-east-1a
      IamInstanceProfile: Backup
      ImageId: ami-6d1c2007 # CentOS 7 - https://aws.amazon.com/marketplace/pp/B00O7WM7QW/&ref_=_mkt_ste_menu
      InstanceType: t2.micro
      KeyName: amazonian
      SecurityGroupIds: # for VPCs
        - !Sub ${secGrpId}
      SubnetId: !Sub ${subNetId}
      UserData:
        "Fn::Base64":
          !Sub |
            #!/bin/bash -xe
            yum -y update
            # Setup cfn-bootstrap
            cd /opt
            curl https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz > aws-cfn-bootstrap-latest.tar.gz
            tar zxvf aws-cfn-bootstrap-latest.tar.gz
            cd aws-cfn-bootstrap-1.4
            python setup.py build
            python setup.py install
            # Run cfn-init
            /bin/cfn-init -v -c ${mode} --stack ${AWS::StackName} --resource FlexServer --region ${AWS::Region}
    Metadata:
      AWS::CloudFormation::Init:
        configSets:
          httpd:       # The "default" configSet is used when cfn-init is called without -c option
            - "Apache"
            - "PHP"
          db:
            - "mariadb"
          all:
            - "mariadb"
            - "Apache"
            - "PHP"

        # Install Apache Web Server (httpd)
        Apache:
          packages:
            yum:
              httpd: []
          services:
            sysvinit:
              httpd:
                enabled: true
                ensureRunning: true

        PHP:
          packages:
            yum:
              php: []
              php-mysql: []

        mariadb:         # This is the MariaDB that comes with CentOS Base Repo
          packages:
            yum:
              mariadb-server: []
          files:
            /tmp/mariadb_config_stdin.txt:
              content: !Sub |

                Y
                ${dbRootPassword}
                ${dbRootPassword}
                Y
                Y
                Y
                Y
              mode: "000600"
              owner: "root"
              group: "root"
          commands:
            0100_start_mariadb:
              command: "/bin/systemctl start  mariadb.service"
            0200_run_secure_script:
              command: "/bin/mysql_secure_installation < /tmp/mariadb_config_stdin.txt"
            0300_cleanup:
              command: "rm -vf /tmp/mariadb_config_stdin.txt"
          services:
            sysvinit:
              mariadb:
                enabled: true
                ensureRunning: true

When the AWS CLI CloudFormation call includes –parameters ParameterKey=mode,ParameterValue=httpd,UsePreviousValue=true, that forces cfn-init to use -c httpd. When that happens, the logs show:

1
2
3
4
5

2017-03-14 21:25:02,416 [INFO] Running configSets: httpd
2017-03-14 21:25:02,416 [INFO] Running configSet httpd
2017-03-14 21:25:02,418 [INFO] Running config Apache
...
2017-03-14 21:25:05,935 [INFO] Running config PHP

When the AWS CLI CloudFormation call includes –parameters ParameterKey=mode,ParameterValue=db,UsePreviousValue=true, that forces cfn-init to use -c db. When that happens, the logs show:

1
2
3

2017-03-14 21:23:23,153 [INFO] Running configSets: db
2017-03-14 21:23:23,153 [INFO] Running configSet db
2017-03-14 21:23:23,154 [INFO] Running config mariadb

When the AWS CLI CloudFormation call includes –parameters ParameterKey=mode,ParameterValue=all,UsePreviousValue=true, that forces cfn-init to use -c db. When that happens, the logs show:

1
2
3
4
5
6
7

2017-03-14 21:23:03,729 [INFO] Running configSets: all
2017-03-14 21:23:03,729 [INFO] Running configSet all
2017-03-14 21:23:03,731 [INFO] Running config mariadb
...
2017-03-14 21:23:42,218 [INFO] Running config Apache
...
2017-03-14 21:23:45,824 [INFO] Running config PHP

Using this technique, you can maintain just a single CloudFormation template, and use it to create many different combinations of EC2 instances. It is also easier to cut and paste various Config Sets than to extend UserData blocks.

Tags:  AWS  AWS CloudFormation
Categories:  AWS  AWS CloudFormation

See Also

 Top Ten Tags

AWS (43)   Kinesis (9)   Streams (8)   AWS Console (5)   Go (5)   Analytics (4)   Data (4)   database (4)   Amazon DynamoDB (3)   Amazon Elastic Compute Cloud (EC2) (3)  

All Tags (173)

 Top Ten Categories

AWS (38)   Amazon DynamoDB (6)   Amazon Elastic Compute Cloud (EC2) (4)   AWS CloudFormation (4)   Certification (3)   Database (3)   Amazon Athena (2)   Amazon Aurora for Amazon RDS (2)   Amazon CloudFormation (2)   Amazon CloudWatch (2)  

All Categories (72)

Disclaimer

All data and information provided on this site is for informational purposes only. cloudninja.cloud makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this site and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

This is a personal weblog. The opinions expressed here represent my own and not those of my employer. My opinions may change over time.