Using the AWS CLI to show the Kinesis Encryption Status

2018-01-27 / 789 words / 4 minutes

Many organizations have internal guidelines designed to keep their data secure. Typical guidelines include the encryption of various resources such as Amazon Kinesis Data Streams. There are many ways to get this data, including expensive 3rd party tools, but the easiest and cheapest way to show the Kinesis encryption status is probably using the AWS Command Line Interface (CLI).

In order to produce a halfway decent tool for the use of the security team, there are a few requirements:

Display the AWS account name. This might not be important if you only have a single account, but organizations with these security rules typically have multiple AWS accounts, and displaying the name helps avoid confusion. For each Kinesis Stream, we want to show the encryption status. This will ensure we don’t miss any streams. The text should be compact and easy to parse. We’re assuming that the AWS CLI is already installed, and a Linux or Mac OS X machine is being used.

Using the AWS CLI to show the Kinesis Encryption Status The process will be broken into three steps:

Displaying the AWS account name Retrieving the list of Kinesis Streams Retrieving the Kinesis encryption status Displaying the AWS account name The aws iam list-account-aliases is used to retrieve the AWS account name, but the default format is JSON which produces output like:

{ “AccountAliases”: [ “amz20180124” ] } 1 2 3 4 5 { “AccountAliases”: [ “amz20180124” ] } This doesn’t meet our 4th requirement. Even if it did, there is no guarantee that the user has not changed their default output type using aws configure.

To get just the account name (in this case “amz20180124“), we use the command:

aws iam list-account-aliases –query “AccountAliases[0]” –output text

The –query parameter provides a JMESPath string. JMES is a query language for JSON documents, similar to XPath for XML documents. The query is for the 1st element of the AccountAliases array. This notation uses a zero-base for arrays, meaning in array of 3 items, there is an item at position 0, 1 and 2.

Using the –query parameter alone would return literally “amz20180124” (including the quotes). Using –output text removes these quotes.

Retrieving the list of Kinesis Streams In this step we need to retrieve the list of streams for use in a bash for loop. The command to retrieve the list of streams is aws kinesis list-streams but again, the output isn’t useable:

{ “StreamNames”: [ “tmp”, “tmp2”, “tmp3” ] } 1 2 3 4 5 6 7 { “StreamNames”: [ “tmp”, “tmp2”, “tmp3” ] } We use exactly the same techniques here to query the JSON output and convert it to text. The full command is:

aws kinesis list-streams –query “StreamNames” –output text

Retrieving the Kinesis Encryption Status In this step we need to retrieve the encryption status. The command that shows details about a Kinesis Stream is aws kinesis describe-stream –stream-name which generates the following output:

{ “StreamDescription”: { “KeyId”: “alias/aws/dynamodb”, “EncryptionType”: “KMS”, “StreamStatus”: “ACTIVE”, “StreamName”: “tmp2”, “Shards”: [ { “ShardId”: “shardId-000000000000”, “HashKeyRange”: { “EndingHashKey”: “340282366920938463463374607431768211455”, “StartingHashKey”: “0” }, “SequenceNumberRange”: { “StartingSequenceNumber”: “49581147093082945838572051644250419483489399386681638914” } } ], “StreamARN”: “arn:aws:kinesis:us-east-1:187655263883:stream/tmp2”, “EnhancedMonitoring”: [ { “ShardLevelMetrics”: [] } ], “StreamCreationTimestamp”: 1517017093.0, “RetentionPeriodHours”: 24 } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 { “StreamDescription”: { “KeyId”: “alias/aws/dynamodb”, “EncryptionType”: “KMS”, “StreamStatus”: “ACTIVE”, “StreamName”: “tmp2”, “Shards”: [ { “ShardId”: “shardId-000000000000”, “HashKeyRange”: { “EndingHashKey”: “340282366920938463463374607431768211455”, “StartingHashKey”: “0” }, “SequenceNumberRange”: { “StartingSequenceNumber”: “49581147093082945838572051644250419483489399386681638914” } } ], “StreamARN”: “arn:aws:kinesis:us-east-1:187655263883:stream/tmp2”, “EnhancedMonitoring”: [ { “ShardLevelMetrics”: [] } ], “StreamCreationTimestamp”: 1517017093.0, “RetentionPeriodHours”: 24 } } From this output, we need to pull the EncryptionType value from the StreamDescription map. The JMESPath query for that is “StreamDescription.EncryptionType” The complete command is:

aws kinesis describe-stream –stream-name ${stream} –query “StreamDescription.EncryptionType”

NOTE: So, there’s no –output text here? In other examples, using the text output removed the quotes and made the output compatible with a bash for loop. In this example, it causes the output to always be None.

Wrap Up The final code in expanded form:

aws iam list-account-aliases –query “AccountAliases[0]” for stream in aws kinesis list-streams --query "StreamNames" --output text do echo -n “Stream: ${stream}=” aws kinesis describe-stream –stream-name ${stream} –query “StreamDescription.EncryptionType” done 1 2 3 4 5 6 aws iam list-account-aliases –query “AccountAliases[0]” for stream in aws kinesis list-streams --query "StreamNames" --output text do echo -n “Stream: ${stream}=” aws kinesis describe-stream –stream-name ${stream} –query “StreamDescription.EncryptionType” done It is compact enough to be used in an alias:

aws iam list-account-aliases –query “AccountAliases[0]” –output text;for stream in aws kinesis list-streams –query “StreamNames” –output text; do echo -n “Stream: ${stream} = “; aws kinesis describe-stream –stream-name ${stream} –query “StreamDescription.EncryptionType” –output text; done

Tags: AWS AWS CLI Amazon Kinesis Amazon Kinesis Data Streams bash
Categories: AWS Amazon Kinesis Amazon Kinesis Streams Amazon Kinesis Data Streams

Using the AWS CLI to show the Kinesis Encryption Status

See Also