AWS Commands

Account

Account number:

aws sts get-caller-identity \
    --query 'Account' \
    --output text

IP Address Ranges

AWS documentation is here.

curl -s https://ip-ranges.amazonaws.com/ip-ranges.json

ip_ranges.json

cat "$IpRanges" \
    | jq '.prefixes[]
          | select(.region = "us-east-2")
          | .ip_prefix'

Get Region of IP

(Would be cool).

Check IP address against each CIDR range. Would be cool to do this using bit comparisons.

See: https://unix.stackexchange.com/questions/274330/check-ip-is-in-range-of-whitelist-array

Regions

All Regions in Partition

aws ec2 describe-regions \
    | jq -r ".Regions[].RegionName"
#+RESULTS:
eu-north-1
ap-south-1
eu-west-3
eu-west-2
eu-west-1
ap-northeast-3
ap-northeast-2
ap-northeast-1
sa-east-1
ca-central-1
ap-southeast-1
ap-southeast-2
eu-central-1
us-east-1
us-east-2
us-west-1
us-west-2

Number of regions:

echo $AllRegions | wc -w   # Using wc

regions=($AllRegions)      # Or use an array, and
echo ${#regions[*]}        # print array length
#+RESULTS:
21
21

All Regions in All Partitions

Should be in the botocore data files. Also, we can pull them from the IpRanges document. You have to decide whether "GLOBAL" is a "region" in your situation.

cat "$IpRanges" \
    | jq -r '.prefixes[].region' \
    | sort | uniq

Run a Command in All Regions

Linear

Get the number of VPCs in each region the slow and simple way.

for region in $regions; do
    vpcs=($(aws --region $region ec2 describe-vpcs | jq '.Vpcs[].VpcId'))
    echo "$region: ${#vpcs[*]}"
done
#+RESULTS:
eu-north-1: 0
ap-south-1: 0
eu-west-3: 0
eu-west-2: 0
eu-west-1: 0
ap-northeast-3: 1
ap-northeast-2: 0
ap-northeast-1: 0
sa-east-1: 0
ca-central-1: 0
ap-southeast-1: 1
ap-southeast-2: 0
eu-central-1: 0
us-east-1: 2
us-east-2: 3
us-west-1: 0
us-west-2: 0

Parallel

Get the number of VPCs in each region the 🚀 and 😎 (and 🤮) way by starting a subprocess for each region in parallel. Note that ${#regions[@]} is bash syntax for the length of the $regions array.

script=$(cat <<-"EOF"
vpcs=($(aws --region {} ec2 describe-vpcs | jq '.Vpcs[].VpcId'))
echo "{}: ${#vpcs[@]}"
EOF
)

regions=($regions)
printf "%s\n" "${regions[@]}" \
    | xargs -n 1 \
            -P ${#regions[*]} \
            -I {} \
            bash -c "$script"
#+RESULTS:
us-east-2: 8
ca-central-1: 2
eu-south-1: 1
eu-west-3: 1
us-west-2: 3
us-east-1: 97
eu-central-1: 1
us-west-1: 4
eu-north-1: 2
eu-west-2: 3
eu-west-1: 1
sa-east-1: 1
ap-northeast-2: 1
ap-northeast-1: 2
ap-south-1: 1
me-south-1: 1
ap-east-1: 1
ap-northeast-3: 1
af-south-1: 1
ap-southeast-1: 1
ap-southeast-2: 1

ACM

List

aws acm list-certificates \
    --includes "keyUsage=ANY" \
    | jq '.CertificateSummaryList[]'

Describe

aws acm describe-certificate \
    --certificate-arn $CertArn

Get Certificate

aws acm get-certificate \
    --certificate-arn $CertificateArn

Get Certificate PEM File

aws acm get-certificate \
    --certificate-arn $CertificateArn \
    | jq -r '.Certificate' > route_guide_cert.pem
cat appliance_cert.pem

API Gateway V2

List APIs

aws apigatewayv2 get-apis \
    | jq '.Items[]'

CloudFormation

For create/deploy/update, see my cloudformation/README.org.

View Stack Log

aws cloudformation list-stacks \
    | jq -M '.StackSummaries[0].StackId' \
    | xargs aws cloudformation describe-stack-events --stack-name

List Stacks

List only the created stacks

aws cloudformation list-stacks \
    --stack-status-filter CREATE_IN_PROGRESS CREATE_COMPLETE

Stack Outputs

aws cloudformation describe-stacks \
    --stack-name $StackName \
    | jq '.Stacks[].Outputs[]'

Get the value of a single output:

aws cloudformation describe-stacks \
    --stack-name $StackName \
    --query "Stacks[0].Outputs[?OutputKey=='$OutputKey'].OutputValue" \
    --output text

Get Stack Parameters

Get all parameters used to create a particular stack:

aws cloudformation describe-stacks \
    --stack-name $StackName \
    | jq '.Stacks[].Parameters[]'

Get the value of a single parameter:

aws cloudformation describe-stacks \
    --stack-name $StackName \
    --query "Stacks[0].Parameters[?ParameterKey=='$Parameter'].ParameterValue" \
    --output text

Exports

Get value of the exported field with name $ExportName:

aws cloudformation list-exports \
    --query "Exports[?Name=='$ExportName'].Value" \
    --output text

List exports that start with a string:

aws cloudformation list-exports \
    --query "Exports[?starts_with(Name, 'foo-')].[Name, Value]" \
    --output text

Spec Files

URLs are documented here.

CloudWatch

Delete All Alarms

alarms=$(aws cloudwatch describe-alarms \
             | jq -r '.MetricAlarms[].AlarmName')

for alarm in $alarms; do
    aws cloudwatch delete-alarms \
        --alarm-names $alarm;
done

Create Alarm Targeting ASG policy

aws cloudwatch put-metric-alarm \
    --alarm-name AddCapacity \
    --metric-name CPUUtilization \
    --namespace AWS/EC2 \
    --statistic Average \
    --period 120 \
    --threshold 80 \
    --comparison-operator GreaterThanOrEqualToThreshold \
    --dimensions "Name=AutoScalingGroupName,Value=my-asg" \
    --evaluation-periods 2 \
    --alarm-actions $ScalingPolicyArn

CodeDeploy

list-deployments

aws deploy list-deployments --application-name saus

DynamoDB

The paper: Dynamo: Amazon’s Highly Available Key-value Store. This describes a distributed database with a leaderless replication model.

From Designing Data-Intensive Applications by Martin Kleppmann:

Dynamo is not available to users outside of Amazon. Confusingly, AWS offers a hosted database product called DynamoDB, which uses a completely different architecture: it is based on single-leader replication.

List Tables

aws dynamodb  list-tables | jq '.TableNames[]'

Delete Tables Like a Maniac

AWS allows you to delete have 10 delete-table operations running at a time. We can create a pool of 10 processes and continuously pump delete-table commands into it using xargs.

script=$(cat <<"EOF"
aws dynamodb delete-table --table-name {}
aws dynamodb wait table-not-exists --table-name {}
EOF
      )
echo ${tables[*]} \
    | xargs -n 1 -P 10 -I {} sh -c "$scrpt"

TODO: When I originally wrote this, I remember needing to add a tr " " "\n" or something for portability with Linux. Test this out again.

TODO: script used to be one command (using &&); test again after splitting it into two commands.

EC2 AMIs

List AMIs

All of this account's AMIs:

aws ec2 describe-images --owners $Account

All of this account's AMIs with a particular Name tag:

aws ec2 describe-images \
    --owners $account \
    --filters "Name=tag:Name,Values=My AMI"

Get Latest Image

aws ec2 describe-images \
    --owners $Account \
    --filters "Name=state,Values=available" \
    | jq -r '.Images
             | sort_by(.CreationDate)
             | last | .ImageId'

Delete AMIs and EBS Snapshots

List AMIs to delete and set to variable AmisToDelete. Then:

jq -c '.Images
       | sort_by(.CreationDate)
       | .[]
       | {name: .Name, snap: .BlockDeviceMappings[]
       | select(.Ebs != null)
       | .Ebs.SnapshotId}' < $AmisToDelete > images.txt

And then… what did I do?

for i in (cat images.txt | jq); do
    # deregister image
    # delete the snapshot
done

List Accounts That Can Access AMI

aws ec2 describe-image-attribute \
    --image-id $ImageID \
    --attribute launchPermission

EC2 AZs

All AZs in all regions:

for region in $AllRegions; do
    azs=$(aws --region $region \
              ec2 describe-availability-zones \
              | jq -c '[.AvailabilityZones[].ZoneName]')
    printf "$region: $azs\n"
done
#+RESULTS:
eu-north-1: ["eu-north-1a","eu-north-1b","eu-north-1c"]
ap-south-1: ["ap-south-1a","ap-south-1b","ap-south-1c"]
eu-west-3: ["eu-west-3a","eu-west-3b","eu-west-3c"]
eu-west-2: ["eu-west-2a","eu-west-2b","eu-west-2c"]
eu-west-1: ["eu-west-1a","eu-west-1b","eu-west-1c"]
ap-northeast-3: ["ap-northeast-3a","ap-northeast-3b","ap-northeast-3c"]
ap-northeast-2: ["ap-northeast-2a","ap-northeast-2b","ap-northeast-2c","ap-northeast-2d"]
ap-northeast-1: ["ap-northeast-1a","ap-northeast-1c","ap-northeast-1d"]
sa-east-1: ["sa-east-1a","sa-east-1b","sa-east-1c"]
ca-central-1: ["ca-central-1a","ca-central-1b","ca-central-1d"]
ap-southeast-1: ["ap-southeast-1a","ap-southeast-1b","ap-southeast-1c"]
ap-southeast-2: ["ap-southeast-2a","ap-southeast-2b","ap-southeast-2c"]
eu-central-1: ["eu-central-1a","eu-central-1b","eu-central-1c"]
us-east-1: ["us-east-1a","us-east-1b","us-east-1c","us-east-1d","us-east-1e","us-east-1f"]
us-east-2: ["us-east-2a","us-east-2b","us-east-2c"]
us-west-1: ["us-west-1b","us-west-1c"]
us-west-2: ["us-west-2a","us-west-2b","us-west-2c","us-west-2d"]

The Actual AZ

us-east-1b does not mean the same thing in every AWS account. If everyone creates infra in us-east-1b, that infra will actually be in several different AZs, depending on which AZ us-east-1b maps to for each account. To determine the real AZ, you need to look at the Zone ID.

Print the Zone Name and Zone ID for each AZ in a region:

aws ec2 describe-availability-zones \
    | jq -c '.AvailabilityZones[]
             | {"Name": .ZoneName, "Id": .ZoneId}'
#+RESULTS:
{"Name":"us-east-2a","Id":"use2-az1"}
{"Name":"us-east-2b","Id":"use2-az2"}
{"Name":"us-east-2c","Id":"use2-az3"}

EC2 KeyPairs

Create

aws ec2 create-key-pair --key-name $KeyName

Copy private key to ~/.ssh

KeyName=$(echo $KeyPair | jq -r '.KeyName')
KeyFile=~/.ssh/$dir/$KeyName
echo $KeyPair | jq -r '.KeyMaterial' > $KeyFile
chmod 0600 $KeyFile
echo $KeyFile

Describe

aws ec2 describe-key-pairs | jq '.KeyPairs[]'

Delete

aws ec2 delete-key-pair --key-name $KeyName

EC2 Instances

Get Running Instances

aws ec2 describe-instances \
    --query "Reservations[].Instances[?State.name=="running"]"

Get All Instance Tags

aws ec2 describe-instances \
    | jq -c '.Reservations[] | .Instances[] | .Tags[]'

Get Instances with Tag Key

With just a specific Tag key, disregarding value:

aws ec2 describe-instances \
    --filter "Name=tag-key,Values=k8s.io/role/node" \
    | jq '.Reservations[].Instances[]'

Get Instances with Tag Key/Value

With just a specific Tag key:

aws ec2 describe-instances \
    --filter "Name=tag:$TagName,Values=$TagValue" \
    | jq -r '.Reservations[].Instances[].Tags[]
             | select(.Key == "Name")
             | .Value'

With a specific tag key and value

EC2 Transit Gateway

List Routes in a TG Route Table

aws --profile shared-services --region us-east-1 \
    ec2 search-transit-gateway-routes \
    --transit-gateway-route-table-id $TGWRouteTableId \
    --filters "Name=type,Values=propagated" \
    | jq -r '.Routes[].DestinationCidrBlock'

EC2 VPCs

VPCs

aws ec2 describe-vpcs | jq '.Vpcs[]'

The names of VPCs that have a Name tag:

aws ec2 describe-vpcs \
    --filters Name=tag-key,Values=Name \
    | jq -r '.Vpcs[].Tags[]
             | select(.Key == "Name")
             | .Value'

Subnets

aws ec2 describe-subnets  | jq '.Subnets'

SecurityGroups

aws ec2 describe-security-groups \
    | jq '.SecurityGroups'

Private ALB IPs

aws ec2 describe-network-interfaces \
    | jq -r '.NetworkInterfaces[]
          | select(.Attachment.InstanceOwnerId == "amazon-elb")
          | .PrivateIpAddress'

Managed Prefix Lists

aws ec2 describe-managed-prefix-lists \
    | jq '.PrefixLists[]'

#+header :var PrefixListID=""

aws ec2 get-managed-prefix-list-entries \
    --prefix-list-id $PrefixListID \
    | jq '.Entries[]'

EC2 Volumes

List Volumes

aws ec2 describe-volumes \
    --filters Name=tag-key,Values=$Tag \
    | jq -r '.Volumes[].VolumeId'

List Unattached Volumes

aws ec2 describe-volumes \
    --filters Name=status,Values=available \
    | jq -r '.Volumes[].VolumeId'

Delete Volumes

for volume in $Volumes; do
    aws ec2 delete-volume --volume-id $volume
done

ECR

Log in

TODO: $account doesn't work (newline)

ecr=$account.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
pw=$(aws ecr get-login-password)
docker login \
       --username AWS \
       --password "$pw" \
       "https://$ecr"

Create Repository

aws ecr create-repository \
    --repository-name $repoName \
    --tags "Key=Foo,Value=True"

Add Image

Tag image for ECR:

ecr=$account.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
docker tag \
       $image:$tag \
       $ecr/$name:$tag

Push it:

ecr=$account.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
docker push \
       $ecr/$name:$tag

ECS

Fix a CloudFormation deployment where an ECS service fails to stabilize. Taken from this AWS blog post. Just updates the number of ECS service tasks to 0.

aws ecs update-service \
    --cluster $ECS_CLUSTER_NAME \
    --service $ECS_SERVICE_NAME \
    --desired-count 0

EKS

List Clusters

aws eks list-clusters | jq '.clusters[]'

Get Kube Config

KUBECONFIG=~/.kube/ironnet_clusters/$CLUSTER_NAME
aws eks update-kubeconfig \
    --name $CLUSTER_NAME \
    --kubeconfig $KUBECONFIG

Elasticache

List CacheClusters

aws elasticache describe-cache-clusters \
    --show-cache-node-info \
    | jq '.CacheClusters'

List ReplicationGroups

aws elasticache describe-replication-groups
        | jq '.ReplicationGroups'

ElasticBeanstalk

Config Options for Nampespace

aws elasticbeanstalk describe-configuration-options \
    | jq '.Options[]
          | if .Namespace == "aws:autoscaling:launchconfiguration"
            then .
            else null
            end'

ElasticLoadBalancingV2

Describe ALBs

aws elbv2 describe-load-balancers \
    | jq '.LoadBalancers[] | select(.Type == "application")'

Get DNS Name

aws elbv2 describe-load-balancers \
    | jq -r --arg DEPLOYMENT_NAME "$DEPLOYMENT_NAME" \
         '.LoadBalancers[]
          | select(.Type == "application")
          | select(.LoadBalancerName
          | startswith($DEPLOYMENT_NAME))
          | .DNSName'

GlobalAccelerator

GlobalAccelerator is only in us-west-2, so set the region explicitly.

List Accelerators

aws --region us-west-2 \
    globalaccelerator list-accelerators \
    | jq '.Accelerators[]'

List Listeners

aws --region us-west-2 \
    globalaccelerator list-listeners \
    --accelerator-arn $AcceleratorArn

IPs

aws --region us-west-2 \
    globalaccelerator list-accelerators \
    | jq -r '.Accelerators[].IpSets[].IpAddresses[]'

IAM

Create Policy

aws iam create-policy \
    --policy-name $PolicyName \
    --policy-document '{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret"
    ],
    "Resource": [
      "arn:*:secretsmanager:*:*:secret:MySecret"
    ]
  }]
}'

Describe Policy

aws iam get-policy --policy-arn $arn

Assume Role

aws sts assume-role \
    --role-session-name foo \
    --role-arn $RoleArn

List Instance Profiles

aws iam list-instance-profiles \
    | jq '.InstanceProfiles[].InstanceProfileName'

Delete Instance Profile

aws iam delete-instance-profile \
    --instance-profile-name $Name

IMDS

Kinesis

Delete Stream

aws kinesis delete-stream --stream-name $stream

Kinesis Firehose

Describe Stream

aws firehose describe-delivery-stream \
    --delivery-stream-name $StreamName \
    | jq '.DeliveryStreamDescription'

Update Stream S3 Destination Bucket

aws firehose update-destination \
    --delivery-stream-name $StreamName \
    --current-delivery-stream-version-id 1 \
    --destination-id destinationId-000000000001 \
    --extended-s3-destination-update \
    BucketARN=arn:aws:s3:::$BucketName

THEN:

  • Update the Stream's IAM policy to allow it to write to the new location
  • Update the S3 Bucket Policy to allow the Stream's IAM Role to write to it

Update Stream S3 Destination Prefix

aws firehose update-destination \
    --delivery-stream-name $StreamName \
    --current-delivery-stream-version-id 1 \
    --destination-id destinationId-000000000001 \
    --extended-s3-destination-update Prefix=$Prefix

Also may need to update the IAM Policy and the S3 Bucket policy, as above.

Lambda

Invoke function

aws lambda invoke \
    --cli-binary-format raw-in-base64-out \
    --function-name cf-Hello-World \
    /tmp/cats.json

Invoke from a bastion host:

/usr/local/bin/aws lambda invoke \
                   --cli-binary-format raw-in-base64-out \
                   --function-name cf-Hello-World \
                   /tmp/cats.json
cat /tmp/out.json | jq -r '.body' | jq

With payload file:

aws lambda invoke \
    --cli-binary-format raw-in-base64-out \
    --function-name hello-world \
    --payload file://tests/resources/event_alb.json \
    /tmp/out.json

Create Function with Zip File

aws lambda create-function \
    --function-name artifacttool \
    --runtime python2.7 \
    --role $RoleArn \
    --handler lambda_function.lambda_handler \
    --zip-file fileb://$ZipFileName

Update Function code with Zip File

aws lambda update-function-code \
    --function-name artifacttool \
    --zip-file fileb://$ZipFileName

Container Image: Run Interactively

docker run -it --rm \
       --entrypoint bash \
       public.ecr.aws/lambda/python:3.8

Organizations

aws organizations list-roots \
    | jq -r '.Roots[].Id'

Polly

aws polly synthesize-speech \
    --engine standard \
    --output-format mp3 \
    --voice-id Amy \
    --text "All these cats wear hats." \
    /tmp/speech_standard.mp3 \
    && afplay /tmp/speech_standard.mp3

Reosurce Access Manager (RAM)

This needs to be run against the AWS account that actually owns the shared resources. E.g. in a LandingZone environment, probably the shared-services account.

aws ram list-resources --resource-owner SELF

RDS

describe-db-instances

aws rds describe-db-instances \
    --db-instance-identifier "my-service-dev" \
    | jq 'DBInstances[]'

describe-db-engine-versions

aws rds describe-db-engine-versions \
    --engine "postgres" \
    | jq -r '.DBEngineVersions[].EngineVersion'

pending maintenance actions for a DB

aws rds describe-pending-maintenance-actions \
    | jq '.PendingMaintenanceActions[]
          | select(.ResourceIdentifier=="$DB_ARN")'

Route53

Get HostedZoneId from domain name

aws route53 list-hosted-zones \
    | jq -r --arg name $hostedZoneName \
         '.HostedZones[]
          | select(.Name == $name + ".")
          | select(.Config.PrivateZone == false)
          | .Id'

List Records for a HostedZone

aws route53 list-resource-record-sets \
    --hosted-zone-id $hostedZoneId \
    | jq '.ResourceRecordSets[]'

SecretsManager

List Secrets

aws secretsmanager list-secrets \
    | jq -r '.SecretList[].Name'

Filter:

aws secretsmanager list-secrets \
    --filters 'Key=name,Values=foo/bar' \
    | jq -r '.SecretList[].Name'

Get Secret

aws secretsmanager get-secret-value \
    --secret-id $SecretName \
    | jq -r '.SecretString'

Create Secret

aws secretsmanager create-secret \
    --name $SecretName \
    --secret-string $SecretValue

ServiceCatalog

Get Product IDs

aws --profile main servicecatalog search-products-as-admin

Launch Product

Note that user, even Admin user, must be added to the Portfolio users/groups. You can do this with aws servicecatalog associate-principal-with-portfolio

Service Quotas

Get Quota

aws service-quotas list-service-quotas \
    --service-code firehose \
    | jq '.Quotas[] | {QuotaName, Value}'

Get Default Quota

aws service-quotas get-aws-default-service-quota \
    --service-code firehose \
    --quota-code $QuotaCode

S3

Canonical ID

aws s3api get-bucket-acl --bucket $BucketName

Create Bucket

aws s3api create-bucket \
    --bucket $BUCKET_NAME \
    --create-bucket-configuration LocationConstraint=$AWS_DEFAULT_REGION

List Objects

All objects

aws s3api list-objects-v2 \
    --bucket $BucketName \
    | jq '.Contents[]'

10 most recent objects. AWS has no way of doing this filtering server-side, so you need to request all objects and then sort them.

aws s3api list-objects-v2 \
    --bucket $BucketName \
    | jq -r '.Contents
             | sort_by(.LastModified)
             | reverse | first | .Key'

Get object

aws s3api get-object \
    --bucket $BucketName \
    --key $Key ./foo

Get multiple objects

mkdir -p ~/Downloads/foo
aws s3 cp \
    --recursive \
    s3://$BucketName/ \
    ~/Downloads/foo/

Get Bucket Policy

aws s3api get-bucket-policy --bucket $BucketName \
    | jq -r '.Policy' | jq

Delete Buckets

I have functions for this in dotfiles/.functions.fish.

SNS

SNS topics

aws sns list-topics \
    | jq '.Topics[] | .TopicArn'

SQS

List Queues

aws sqs list-queues \
    | jq '.QueueUrls[]'

Read Message

aws --profile prod --region us-east-1 sqs receive-message \
    --queue-url "$q_url" \
    --attribute-names All \
    --message-attribute-names All \
    --max-number-of-messages 2

Send Message

aws sqs send-message \
    --queue-url "$q_url" \
    --message-body '{"Message": "{\"foo\":\"bar\"}"}'

SSM

aws ssm get-parameter \
    --with-decryption \
    --name $name

SSO

Register Client

This command does not require AWS credentials. You only need a region.

aws --region us-east-1 sso-oidc register-client \
    --client-name foo \
    --client-type public

Returns an object like:

{
    "clientId": "abc1234",
    "clientSecret": "a JSON Web Token",
    "clientIdIssuedAt": 1659099242,
    "clientSecretExpiresAt": 1666875242
}

Token

TODO: Need to run aws sso login. Requires block in ~/.aws/config I think.

TODO: Look up token file programatically

token_file=~/.aws/sso/cache/$token_name.json
cat $token_file | jq -r '.accessToken'

List Accounts I Can Access

TODO: SsoToken has a trailing newline!

aws sso list-accounts \
    --access-token $token
aws sso list-account-roles \
    --access-token $token \
    --account-id 820416642700