AWS Commands

Account

Account number:

aws sts get-caller-identity \
    --query 'Account' \
    --output text

IP Address Ranges

AWS documentation is here.

curl -s https://ip-ranges.amazonaws.com/ip-ranges.json

ip_ranges.json

cat "$IpRanges" \
    | jq '.prefixes[]
          | select(.region = "us-east-2")
          | .ip_prefix'

Get Region of IP

(Would be cool).

Check IP address against each CIDR range. Would be cool to do this using bit comparisons.

See: https://unix.stackexchange.com/questions/274330/check-ip-is-in-range-of-whitelist-array

Regions

All Regions in Partition

aws ec2 describe-regions \
    | jq -r ".Regions[].RegionName"

#+RESULTS:

eu-north-1
ap-south-1
eu-west-3
eu-west-2
eu-west-1
ap-northeast-3
ap-northeast-2
ap-northeast-1
sa-east-1
ca-central-1
ap-southeast-1
ap-southeast-2
eu-central-1
us-east-1
us-east-2
us-west-1
us-west-2

Number of regions:

echo $AllRegions | wc -w   # Using wc

regions=($AllRegions)      # Or use an array, and
echo ${#regions[*]}        # print array length

#+RESULTS:

21
21

All Regions in All Partitions

Should be in the botocore data files. Also, we can pull them from the IpRanges document. You have to decide whether "GLOBAL" is a "region" in your situation.

cat "$IpRanges" \
    | jq -r '.prefixes[].region' \
    | sort | uniq

Run a Command in All Regions

Linear

Get the number of VPCs in each region the slow and simple way.

for region in $regions; do
    vpcs=($(aws --region $region ec2 describe-vpcs | jq '.Vpcs[].VpcId'))
    echo "$region: ${#vpcs[*]}"
done

#+RESULTS:

eu-north-1: 0
ap-south-1: 0
eu-west-3: 0
eu-west-2: 0
eu-west-1: 0
ap-northeast-3: 1
ap-northeast-2: 0
ap-northeast-1: 0
sa-east-1: 0
ca-central-1: 0
ap-southeast-1: 1
ap-southeast-2: 0
eu-central-1: 0
us-east-1: 2
us-east-2: 3
us-west-1: 0
us-west-2: 0

Parallel

Get the number of VPCs in each region the 🚀 and 😎 (and 🤮) way by starting a subprocess for each region in parallel. Note that ${#regions[@]} is bash syntax for the length of the $regions array.

script=$(cat <<-"EOF"
vpcs=($(aws --region {} ec2 describe-vpcs | jq '.Vpcs[].VpcId'))
echo "{}: ${#vpcs[@]}"
EOF
)

regions=($regions)
printf "%s\n" "${regions[@]}" \
    | xargs -n 1 \
            -P ${#regions[*]} \
            -I {} \
            bash -c "$script"

#+RESULTS:

us-east-2: 8
ca-central-1: 2
eu-south-1: 1
eu-west-3: 1
us-west-2: 3
us-east-1: 97
eu-central-1: 1
us-west-1: 4
eu-north-1: 2
eu-west-2: 3
eu-west-1: 1
sa-east-1: 1
ap-northeast-2: 1
ap-northeast-1: 2
ap-south-1: 1
me-south-1: 1
ap-east-1: 1
ap-northeast-3: 1
af-south-1: 1
ap-southeast-1: 1
ap-southeast-2: 1

ACM

List

aws acm list-certificates \
    --includes "keyUsage=ANY" \
    | jq '.CertificateSummaryList[]'

Describe

aws acm describe-certificate \
    --certificate-arn $CertArn

Get Certificate

aws acm get-certificate \
    --certificate-arn $CertificateArn

Get Certificate PEM File

aws acm get-certificate \
    --certificate-arn $CertificateArn \
    | jq -r '.Certificate' > route_guide_cert.pem

cat appliance_cert.pem

API Gateway V2

List APIs

aws apigatewayv2 get-apis \
    | jq '.Items[]'

CloudWatch

Delete All Alarms

alarms=$(aws cloudwatch describe-alarms \
             | jq -r '.MetricAlarms[].AlarmName')

for alarm in $alarms; do
    aws cloudwatch delete-alarms \
        --alarm-names $alarm;
done

Create Alarm Targeting ASG policy

aws cloudwatch put-metric-alarm \
    --alarm-name AddCapacity \
    --metric-name CPUUtilization \
    --namespace AWS/EC2 \
    --statistic Average \
    --period 120 \
    --threshold 80 \
    --comparison-operator GreaterThanOrEqualToThreshold \
    --dimensions "Name=AutoScalingGroupName,Value=my-asg" \
    --evaluation-periods 2 \
    --alarm-actions $ScalingPolicyArn

CodeDeploy

list-deployments

aws deploy list-deployments --application-name saus

DynamoDB

The paper: Dynamo: Amazon’s Highly Available Key-value Store. This describes a distributed database with a leaderless replication model.

From Designing Data-Intensive Applications by Martin Kleppmann:

Dynamo is not available to users outside of Amazon. Confusingly, AWS offers a hosted database product called DynamoDB, which uses a completely different architecture: it is based on single-leader replication.

List Tables

aws dynamodb  list-tables | jq '.TableNames[]'

Delete Tables Like a Maniac

AWS allows you to delete have 10 delete-table operations running at a time. We can create a pool of 10 processes and continuously pump delete-table commands into it using xargs.

script=$(cat <<"EOF"
aws dynamodb delete-table --table-name {}
aws dynamodb wait table-not-exists --table-name {}
EOF
      )
echo ${tables[*]} \
    | xargs -n 1 -P 10 -I {} sh -c "$scrpt"

TODO: When I originally wrote this, I remember needing to add a tr " " "\n" or something for portability with Linux. Test this out again.

TODO: script used to be one command (using &&); test again after splitting it into two commands.

EC2 AMIs

List AMIs

All of this account's AMIs:

aws ec2 describe-images --owners $Account

All of this account's AMIs with a particular Name tag:

aws ec2 describe-images \
    --owners $account \
    --filters "Name=tag:Name,Values=My AMI"

Get Latest Image

aws ec2 describe-images \
    --owners $Account \
    --filters "Name=state,Values=available" \
    | jq -r '.Images
             | sort_by(.CreationDate)
             | last | .ImageId'

Delete AMIs and EBS Snapshots

List AMIs to delete and set to variable AmisToDelete. Then:

jq -c '.Images
       | sort_by(.CreationDate)
       | .[]
       | {name: .Name, snap: .BlockDeviceMappings[]
       | select(.Ebs != null)
       | .Ebs.SnapshotId}' < $AmisToDelete > images.txt

And then… what did I do?

for i in (cat images.txt | jq); do
    # deregister image
    # delete the snapshot
done

List Accounts That Can Access AMI

aws ec2 describe-image-attribute \
    --image-id $ImageID \
    --attribute launchPermission

EC2 AZs

All AZs in all regions:

for region in $AllRegions; do
    azs=$(aws --region $region \
              ec2 describe-availability-zones \
              | jq -c '[.AvailabilityZones[].ZoneName]')
    printf "$region: $azs\n"
done

#+RESULTS:

eu-north-1: ["eu-north-1a","eu-north-1b","eu-north-1c"]
ap-south-1: ["ap-south-1a","ap-south-1b","ap-south-1c"]
eu-west-3: ["eu-west-3a","eu-west-3b","eu-west-3c"]
eu-west-2: ["eu-west-2a","eu-west-2b","eu-west-2c"]
eu-west-1: ["eu-west-1a","eu-west-1b","eu-west-1c"]
ap-northeast-3: ["ap-northeast-3a","ap-northeast-3b","ap-northeast-3c"]
ap-northeast-2: ["ap-northeast-2a","ap-northeast-2b","ap-northeast-2c","ap-northeast-2d"]
ap-northeast-1: ["ap-northeast-1a","ap-northeast-1c","ap-northeast-1d"]
sa-east-1: ["sa-east-1a","sa-east-1b","sa-east-1c"]
ca-central-1: ["ca-central-1a","ca-central-1b","ca-central-1d"]
ap-southeast-1: ["ap-southeast-1a","ap-southeast-1b","ap-southeast-1c"]
ap-southeast-2: ["ap-southeast-2a","ap-southeast-2b","ap-southeast-2c"]
eu-central-1: ["eu-central-1a","eu-central-1b","eu-central-1c"]
us-east-1: ["us-east-1a","us-east-1b","us-east-1c","us-east-1d","us-east-1e","us-east-1f"]
us-east-2: ["us-east-2a","us-east-2b","us-east-2c"]
us-west-1: ["us-west-1b","us-west-1c"]
us-west-2: ["us-west-2a","us-west-2b","us-west-2c","us-west-2d"]

us-east-1b does not mean the same thing in every AWS account. If everyone creates infra in us-east-1b, that infra will actually be in several different AZs, depending on which AZ us-east-1b maps to for each account. To determine the real AZ, you need to look at the Zone ID.

Print the Zone Name and Zone ID for each AZ in a region:

aws ec2 describe-availability-zones \
    | jq -c '.AvailabilityZones[]
             | {"Name": .ZoneName, "Id": .ZoneId}'

#+RESULTS:

{"Name":"us-east-2a","Id":"use2-az1"}
{"Name":"us-east-2b","Id":"use2-az2"}
{"Name":"us-east-2c","Id":"use2-az3"}

EC2 KeyPairs

Create

aws ec2 create-key-pair --key-name $KeyName

Copy private key to ~/.ssh

KeyName=$(echo $KeyPair | jq -r '.KeyName')
KeyFile=~/.ssh/$dir/$KeyName
echo $KeyPair | jq -r '.KeyMaterial' > $KeyFile
chmod 0600 $KeyFile
echo $KeyFile

Describe

aws ec2 describe-key-pairs | jq '.KeyPairs[]'

Delete

aws ec2 delete-key-pair --key-name $KeyName

EC2 Instances

Get Running Instances

aws ec2 describe-instances \
    --query "Reservations[].Instances[?State.name=="running"]"

Get All Instance Tags

aws ec2 describe-instances \
    | jq -c '.Reservations[] | .Instances[] | .Tags[]'

Get Instances with Tag Key

With just a specific Tag key, disregarding value:

aws ec2 describe-instances \
    --filter "Name=tag-key,Values=k8s.io/role/node" \
    | jq '.Reservations[].Instances[]'

Get Instances with Tag Key/Value

With just a specific Tag key:

aws ec2 describe-instances \
    --filter "Name=tag:$TagName,Values=$TagValue" \
    | jq -r '.Reservations[].Instances[].Tags[]
             | select(.Key == "Name")
             | .Value'

With a specific tag key and value

EC2 Transit Gateway

List Routes in a TG Route Table

aws --profile shared-services --region us-east-1 \
    ec2 search-transit-gateway-routes \
    --transit-gateway-route-table-id $TGWRouteTableId \
    --filters "Name=type,Values=propagated" \
    | jq -r '.Routes[].DestinationCidrBlock'

EC2 VPCs

VPCs

aws ec2 describe-vpcs | jq '.Vpcs[]'

The names of VPCs that have a Name tag:

aws ec2 describe-vpcs \
    --filters Name=tag-key,Values=Name \
    | jq -r '.Vpcs[].Tags[]
             | select(.Key == "Name")
             | .Value'

Subnets

aws ec2 describe-subnets  | jq '.Subnets'

SecurityGroups

aws ec2 describe-security-groups \
    | jq '.SecurityGroups'

Private ALB IPs

aws ec2 describe-network-interfaces \
    | jq -r '.NetworkInterfaces[]
          | select(.Attachment.InstanceOwnerId == "amazon-elb")
          | .PrivateIpAddress'

Managed Prefix Lists

aws ec2 describe-managed-prefix-lists \
    | jq '.PrefixLists[]'

#+header :var PrefixListID=""

aws ec2 get-managed-prefix-list-entries \
    --prefix-list-id $PrefixListID \
    | jq '.Entries[]'

EC2 Volumes

List Volumes

aws ec2 describe-volumes \
    --filters Name=tag-key,Values=$Tag \
    | jq -r '.Volumes[].VolumeId'

List Unattached Volumes

aws ec2 describe-volumes \
    --filters Name=status,Values=available \
    | jq -r '.Volumes[].VolumeId'

Delete Volumes

for volume in $Volumes; do
    aws ec2 delete-volume --volume-id $volume
done

ECR

Log in

TODO: $account doesn't work (newline)

ecr=$account.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
pw=$(aws ecr get-login-password)
docker login \
       --username AWS \
       --password "$pw" \
       "https://$ecr"

Create Repository

aws ecr create-repository \
    --repository-name $repoName \
    --tags "Key=Foo,Value=True"

Add Image

Tag image for ECR:

ecr=$account.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
docker tag \
       $image:$tag \
       $ecr/$name:$tag

Push it:

ecr=$account.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
docker push \
       $ecr/$name:$tag

ECS

Fix a CloudFormation deployment where an ECS service fails to stabilize. Taken from this AWS blog post. Just updates the number of ECS service tasks to 0.

aws ecs update-service \
    --cluster $ECS_CLUSTER_NAME \
    --service $ECS_SERVICE_NAME \
    --desired-count 0

EKS

List Clusters

aws eks list-clusters | jq '.clusters[]'

Get Kube Config

KUBECONFIG=~/.kube/ironnet_clusters/$CLUSTER_NAME
aws eks update-kubeconfig \
    --name $CLUSTER_NAME \
    --kubeconfig $KUBECONFIG

Elasticache

List CacheClusters

aws elasticache describe-cache-clusters \
    --show-cache-node-info \
    | jq '.CacheClusters'

List ReplicationGroups

aws elasticache describe-replication-groups
        | jq '.ReplicationGroups'

ElasticBeanstalk

Config Options for Nampespace

aws elasticbeanstalk describe-configuration-options \
    | jq '.Options[]
          | if .Namespace == "aws:autoscaling:launchconfiguration"
            then .
            else null
            end'

ElasticLoadBalancingV2

Describe ALBs

aws elbv2 describe-load-balancers \
    | jq '.LoadBalancers[] | select(.Type == "application")'

Get DNS Name

aws elbv2 describe-load-balancers \
    | jq -r --arg DEPLOYMENT_NAME "$DEPLOYMENT_NAME" \
         '.LoadBalancers[]
          | select(.Type == "application")
          | select(.LoadBalancerName
          | startswith($DEPLOYMENT_NAME))
          | .DNSName'

GlobalAccelerator

GlobalAccelerator is only in us-west-2, so set the region explicitly.

List Accelerators

aws --region us-west-2 \
    globalaccelerator list-accelerators \
    | jq '.Accelerators[]'

List Listeners

aws --region us-west-2 \
    globalaccelerator list-listeners \
    --accelerator-arn $AcceleratorArn

IPs

aws --region us-west-2 \
    globalaccelerator list-accelerators \
    | jq -r '.Accelerators[].IpSets[].IpAddresses[]'

IAM

Create Policy

aws iam create-policy \
    --policy-name $PolicyName \
    --policy-document '{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret"
    ],
    "Resource": [
      "arn:*:secretsmanager:*:*:secret:MySecret"
    ]
  }]
}'

Describe Policy

aws iam get-policy --policy-arn $arn

Assume Role

aws sts assume-role \
    --role-session-name foo \
    --role-arn $RoleArn

List Instance Profiles

aws iam list-instance-profiles \
    | jq '.InstanceProfiles[].InstanceProfileName'

Delete Instance Profile

aws iam delete-instance-profile \
    --instance-profile-name $Name

IMDS

http://169.254.169.254/latest/meta-data/

Kinesis

Delete Stream

aws kinesis delete-stream --stream-name $stream

Kinesis Firehose

Describe Stream

aws firehose describe-delivery-stream \
    --delivery-stream-name $StreamName \
    | jq '.DeliveryStreamDescription'

Update Stream S3 Destination Bucket

aws firehose update-destination \
    --delivery-stream-name $StreamName \
    --current-delivery-stream-version-id 1 \
    --destination-id destinationId-000000000001 \
    --extended-s3-destination-update \
    BucketARN=arn:aws:s3:::$BucketName

THEN:

Update the Stream's IAM policy to allow it to write to the new location
Update the S3 Bucket Policy to allow the Stream's IAM Role to write to it

Update Stream S3 Destination Prefix

aws firehose update-destination \
    --delivery-stream-name $StreamName \
    --current-delivery-stream-version-id 1 \
    --destination-id destinationId-000000000001 \
    --extended-s3-destination-update Prefix=$Prefix

Also may need to update the IAM Policy and the S3 Bucket policy, as above.

Lambda

Invoke function

aws lambda invoke \
    --cli-binary-format raw-in-base64-out \
    --function-name cf-Hello-World \
    /tmp/cats.json

Invoke from a bastion host:

/usr/local/bin/aws lambda invoke \
                   --cli-binary-format raw-in-base64-out \
                   --function-name cf-Hello-World \
                   /tmp/cats.json

cat /tmp/out.json | jq -r '.body' | jq

With payload file:

aws lambda invoke \
    --cli-binary-format raw-in-base64-out \
    --function-name hello-world \
    --payload file://tests/resources/event_alb.json \
    /tmp/out.json

Create Function with Zip File

aws lambda create-function \
    --function-name artifacttool \
    --runtime python2.7 \
    --role $RoleArn \
    --handler lambda_function.lambda_handler \
    --zip-file fileb://$ZipFileName

Update Function code with Zip File

aws lambda update-function-code \
    --function-name artifacttool \
    --zip-file fileb://$ZipFileName

Container Image: Run Interactively

docker run -it --rm \
       --entrypoint bash \
       public.ecr.aws/lambda/python:3.8

Organizations

aws organizations list-roots \
    | jq -r '.Roots[].Id'

Polly

aws polly synthesize-speech \
    --engine standard \
    --output-format mp3 \
    --voice-id Amy \
    --text "All these cats wear hats." \
    /tmp/speech_standard.mp3 \
    && afplay /tmp/speech_standard.mp3

Reosurce Access Manager (RAM)

This needs to be run against the AWS account that actually owns the shared resources. E.g. in a LandingZone environment, probably the shared-services account.

aws ram list-resources --resource-owner SELF

RDS

describe-db-instances

aws rds describe-db-instances \
    --db-instance-identifier "my-service-dev" \
    | jq 'DBInstances[]'

describe-db-engine-versions

aws rds describe-db-engine-versions \
    --engine "postgres" \
    | jq -r '.DBEngineVersions[].EngineVersion'

pending maintenance actions for a DB

aws rds describe-pending-maintenance-actions \
    | jq '.PendingMaintenanceActions[]
          | select(.ResourceIdentifier=="$DB_ARN")'

Route53

Get HostedZoneId from domain name

aws route53 list-hosted-zones \
    | jq -r --arg name $hostedZoneName \
         '.HostedZones[]
          | select(.Name == $name + ".")
          | select(.Config.PrivateZone == false)
          | .Id'

List Records for a HostedZone

aws route53 list-resource-record-sets \
    --hosted-zone-id $hostedZoneId \
    | jq '.ResourceRecordSets[]'

SecretsManager

List Secrets

aws secretsmanager list-secrets \
    | jq -r '.SecretList[].Name'

Filter:

aws secretsmanager list-secrets \
    --filters 'Key=name,Values=foo/bar' \
    | jq -r '.SecretList[].Name'

Get Secret

aws secretsmanager get-secret-value \
    --secret-id $SecretName \
    | jq -r '.SecretString'

Create Secret

aws secretsmanager create-secret \
    --name $SecretName \
    --secret-string $SecretValue

ServiceCatalog

Get Product IDs

aws --profile main servicecatalog search-products-as-admin

aws service-quotas list-service-quotas \
    --service-code firehose \
    | jq '.Quotas[] | {QuotaName, Value}'

Get Default Quota

aws service-quotas get-aws-default-service-quota \
    --service-code firehose \
    --quota-code $QuotaCode

S3

Canonical ID

aws s3api get-bucket-acl --bucket $BucketName

Create Bucket

aws s3api create-bucket \
    --bucket $BUCKET_NAME \
    --create-bucket-configuration LocationConstraint=$AWS_DEFAULT_REGION

List Objects

All objects

aws s3api list-objects-v2 \
    --bucket $BucketName \
    | jq '.Contents[]'

10 most recent objects. AWS has no way of doing this filtering server-side, so you need to request all objects and then sort them.

aws s3api list-objects-v2 \
    --bucket $BucketName \
    | jq -r '.Contents
             | sort_by(.LastModified)
             | reverse | first | .Key'

Get object

aws s3api get-object \
    --bucket $BucketName \
    --key $Key ./foo

Get multiple objects

mkdir -p ~/Downloads/foo
aws s3 cp \
    --recursive \
    s3://$BucketName/ \
    ~/Downloads/foo/

Get Bucket Policy

aws s3api get-bucket-policy --bucket $BucketName \
    | jq -r '.Policy' | jq

aws sns list-topics \
    | jq '.Topics[] | .TopicArn'

SQS

List Queues

aws sqs list-queues \
    | jq '.QueueUrls[]'

Read Message

aws --profile prod --region us-east-1 sqs receive-message \
    --queue-url "$q_url" \
    --attribute-names All \
    --message-attribute-names All \
    --max-number-of-messages 2

Send Message

aws sqs send-message \
    --queue-url "$q_url" \
    --message-body '{"Message": "{\"foo\":\"bar\"}"}'

SSM

aws ssm get-parameter \
    --with-decryption \
    --name $name

SSO

Register Client

This command does not require AWS credentials. You only need a region.

aws --region us-east-1 sso-oidc register-client \
    --client-name foo \
    --client-type public

Returns an object like:

{
    "clientId": "abc1234",
    "clientSecret": "a JSON Web Token",
    "clientIdIssuedAt": 1659099242,
    "clientSecretExpiresAt": 1666875242
}

Token

TODO: Need to run aws sso login. Requires block in ~/.aws/config I think.

TODO: Look up token file programatically

token_file=~/.aws/sso/cache/$token_name.json
cat $token_file | jq -r '.accessToken'

List Accounts I Can Access

TODO: SsoToken has a trailing newline!

aws sso list-accounts \
    --access-token $token

aws sso list-account-roles \
    --access-token $token \
    --account-id 820416642700