Tag Archives: AWS CLI

Amazon EC2 Application Status Checks: A Native Health Check for the Application Layer

An EC2 instance can show 2/2 status checks passed while the application on it has been dead for twenty minutes. The hypervisor is fine, the kernel booted, the network stack answers pings — none of which tells you whether nginx stopped accepting connections or your app is stuck returning 500s to every request. That gap, between “the instance is up” and “the thing running on it actually works,” is what application status checks now cover natively, as of August 10.

EC2 Application Status Checks!

The Gap It Fills

EC2 has run two automatic checks on every instance for years: a system status check watching the AWS hardware underneath, and an instance status check watching the guest OS and its network config. An attached EBS check came later for volume reachability. All three are automatic — you can’t turn them off or point them anywhere — and none of them look inside the application. A crashed worker process or a web server that stopped listening is invisible to EC2, because the instance itself still looks perfectly healthy.

Closing that gap used to mean building it yourself: an ALB target group health check if you had a load balancer, a cron job and a custom CloudWatch metric if you didn’t. Application status checks are AWS’s native answer to the second case, a fourth, opt-in check that speaks HTTP directly to your application.

How It Works

You define a check with a protocol (HTTP or HTTPS), a port, a path, and a status code matcher — the same shape as an ALB health check. Associate it with instances by ID or by tag, including aws:autoscaling:groupName to cover an entire Auto Scaling group in one call, and EC2 pings that port and path every 60 seconds. Fixed interval, no faster, no slower.

aws ec2 create-application-status-check \
    --protocol https --port 443 --path "/health" \
    --status-code-matcher "200"

Two consecutive failures flip the check to impaired, two successes flip it back to ok, both configurable, along with a 6-second timeout and a 300-second grace period after launch so a slow-starting app doesn’t get replaced before it’s ready. Set that grace period too low and Auto Scaling will happily replace instances that just needed more time to boot. Full parameter reference is in the Amazon EC2 User Guide.

Where the Checks Actually Run

Worth knowing before you roll this out: check traffic doesn’t come from some anonymous AWS endpoint. AWS provisions a managed network interface inside your own VPC, one per subnet-and-security-group combination, and traffic runs from within your VPC over AWS’s internal network, never the public internet. That managed ENI doesn’t count against your instance’s ENI limit, but it does count against your account’s ENIs-per-VPC quota, worth checking before a fleet-wide rollout across a lot of subnet-and-security-group combinations.

Check flow

The step people miss: CloudWatch gets a metric per check plus one aggregate per instance, and it’s only the aggregate that Auto Scaling watches. You can also specify the source subnet and security group yourself instead of letting AWS choose, useful if your network has segmentation rules a security review would care about.

App status check details

Auto Scaling, and the Deploy Trap

Auto Scaling groups already pull health signals from EC2, ELB, VPC Lattice, EBS, and custom checks. Application status checks slot in as one more. Associate the check with the group’s instances, and Auto Scaling replaces anything reporting impaired — no extra configuration needed. This earns its keep most on workloads that never sat behind a load balancer at all: backend workers, queue consumers, anything that never had a native AWS health signal before.

The catch: a deployment or restart makes the check fail too, since the app genuinely isn’t answering for a few seconds. Auto Scaling can’t tell “expected restart” from “actual crash,” so an unplanned deploy across a fleet can trigger a replacement storm you caused yourself. AWS gives you three ways to prevent that:

ApproachWhen to use it
Suppress the checkA known, bounded maintenance window
Exclude from aggregationValidating a new check without risking replacements
DisassociatePermanent removal

For routine deploys, automate suppression from your pipeline’s pre- and post-deploy hooks:

aws ec2 enable-application-status-check-suppression \
    --instance-ids i-0123456789abcdef0 --duration-seconds 3600

Put this in the deployment runbook before you need it, not after Auto Scaling has replaced half a fleet mid-release.

Two Gotchas Worth Knowing

When a check fails, look at the reason code before touching application logs. Redirects are a common false failure: health checks don’t follow 301s or 302s, so a /health path that redirects will fail unless you add that code to your matcher. Two other behaviors catch people off guard. The health check goes out over HTTP/2, so a minimal server that only speaks HTTP/1.1 in cleartext can fail a check that a plain curl would pass. And the HTTPS check never validates the server’s certificate, so a passing check tells you nothing about certificate validity.

What It Costs

Pricing: $0.01 per hour per managed ENI, per Availability Zone, plus standard CloudWatch pricing for the metrics. That cost tracks unique subnet-and-security-group combinations, not instance count. One Auto Scaling group in one subnet is one ENI per AZ, essentially free. A fleet spread across many subnets for segmentation reasons will rack up more ENIs than the instance count suggests, so it’s worth estimating that number before assuming this costs nothing. Default account quotas (50 checks, 5,000 targets) are generous for most teams and adjust automatically except for the targets limit, which needs a manual request.

Where This Fits

Application status checks aren’t a replacement for real observability. There’s no tracing, no latency data, just “did this path return the code I said meant healthy.” What they replace is whatever duct tape your team already built to answer that one question: a cron job and a custom metric, a sidecar pinging itself, a script from years ago nobody wants to touch. If that’s your current setup, especially for anything that doesn’t sit behind a load balancer, it’s worth an afternoon to pilot this with the check set to excluded before it can page anyone.

Securing AWS credentials in WSL using aws-vault

This article will walk you through step-by-step procedures to secure AWS CLI credentials using the aws-vault tool.

aws-vault configuration in WSL

Why aws-vault?

Configuring the AWS CLI saves the AWS access key and secret key in plain text format under ~/.aws/credentials file. This is a security concern since the location is known, anybody can scan/look for the keys, and credentials can be compromised.

Hence, to store AWS keys more securely, we need some tool/software to keep the keys in encrypted format rather than plain text. We are discussing the aws-vault open-source tool widely used to secure AWS keys.

Also read: Setting up WSL for Sysadmin work.

How to install and configure aws-vault in WSL?

We will use the pass as a secure backend for aws-vault while configuring it in WSL as the pass is a native Linux password manager. The list of compatible secure backends for aws-vault can be found here.

Install pass utility in WSL. You can leverage package manager depending on your Linux flavour.

# apt install pass

The next step is to generate the gpg key. You will be prompted to enter a password for the key in this process. This password is necessary to unlock password storage for fetching the passwords within.

$ gpg --generate-key
...
GnuPG needs to construct a user ID to identify your key.

Real name: shrikant
Email address: info@kerneltalks.com
You selected this USER-ID:
    "shrikant <info@kerneltalks.com>"

Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
...

pub   rsa3072 2022-09-25 [SC] [expires: 2024-09-24]
      110577AFEE906D84224719BCD0F123xxxxxxxxxx
uid                      shrikant <info@kerneltalks.com>
sub   rsa3072 2022-09-25 [E] [expires: 2024-09-24]

Initiate the new password storage along with gpg key encryption. Use gpg key created in the previous step in the below command –

$ pass init -p aws-vault 110577AFEE906D84224719BCD0F123xxxxxxxxxx
mkdir: created directory '/root/.password-store/aws-vault'
Password store initialized for 110577AFEE906D84224719BCD0F123xxxxxxxxxx (aws-vault)

Now, the pass backend is ready to use with aws-vault.

Install aws-vault. Simply grab the latest release from here. Move and rename it to one of the binary directories and assign execute permission.

$ wget https://github.com/99designs/aws-vault/releases/download/v6.6.0/aws-vault-linux-amd64
$ mv aws-vault-linux-amd64 /usr/local/sbin/aws-vault
$ chmod +x /usr/local/sbin/aws-vault

You have pass configured and aws-vault installed. Your system is ready to save the AWS credentials in aws-vault.

By default, aws-vault considers keyctl backend. Export the below variables for configuring it with pass backend. It’s good to export them in the login profile of WSL so that they will be exported for all your future shell sessions.

$ export AWS_VAULT_BACKEND=pass
$ export AWS_VAULT_PASS_PASSWORD_STORE_DIR=/root/.password-store/aws-vault

Add your AWS credentials in aws-vault with the below command –

# aws-vault add my-aws-account
Enter Access Key ID: XXXXXXXXXXXXX
Enter Secret Access Key: 
Added credentials to profile "my-aws-account" in vault

my-aws-account is an account alias/name for identification purposes only.

Verify if aws-vault saved the credentials in pass password storage.

$ pass show
Password Store
└── aws-vault
    └── my-aws-account

At this point, you have successfully secured your AWS credentials, and you can safely remove ~/.aws/credentials file. If you have more than one AWS account configured in the credentials file, add all of them into aws-vault before deleting it.

pass saves the password storage in ~/.password-store in encrypted format.

$ ls -lrt ~/.password-store/aws-vault/
total 8
-rw------- 1 root root   41 Sep 25 22:01 .gpg-id
-rw------- 1 root root  831 Sep 25 14:22 my-aws-account.gpg

How to use aws-vault?

When you want to run any command using AWS CLI, you can execute it like this –

$ aws-vault exec &lt;profile-name> -- aws &lt;cli-command>
# aws-vault exec my-aws-account -- aws s3 ls

Here: my-aws-account is the AWS profile name. aws-vault also reads the ~/.aws/config file to figure out AWS profiles configured on the machine.

You can authenticate properly with credentials stored securely in the encrypted form! aws-vault also handles the IAM role and MFA prompts defined in AWS profiles.

Error troubleshooting

gpg: decryption failed: No secret key

pass password storage is locked and needs to be unlocked. Run pass show <password-name> , and it should prompt for the gpg key password.

$ pass show my-aws-account
┌───────────────────────────────────────────────────────────────┐
│ Please enter the passphrase to unlock the OpenPGP secret key: │
│ "shrikant <info@kerneltalks.com>"                             │
│ 3072-bit RSA key, ID 0DB81DDFxxxxxxxx,                        │
│ created 2022-09-25 (main key ID D0F123Bxxxxxxxxx).            │
│                                                               │
│                                                               │
│ Passphrase: _________________________________________________ │
│                                                               │
│         <OK>                                   <Cancel>       │
└───────────────────────────────────────────────────────────────┘

Now, try the aws-vault command, and it should work fine.


aws-vault: error: Specified keyring backend not available, try --help

Ensure you have exported both parameters mentioned in the above configuration steps so that aws-vault will use the pass backend.


aws-vault: error: exec: Failed to get credentials for kerneltalks-test: operation error STS: GetSessionToken, https response error StatusCode: 403, RequestID: e514d0e5-b8b6-4144-8616-05061eeed00c, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

It points to the wrong access/secret key combination added to the secure vault. Delete the respective key from the vault and re-add it. If you lost your secret key, then re-create new pair from the AWS console and add it to the key-vault.

$ aws-vault remove kerneltalks-test
Delete credentials for profile "kerneltalks-test"? (y|N) y
Deleted credentials.
$aws-vault add kerneltalks-test
Enter Access Key ID: xxxxxxxxxx
Enter Secret Access Key: 
Added credentials to profile "kerneltalks-test" in vault

aws-vault: error: rotate: Error creating a new access key: operation error IAM: CreateAccessKey, https response error StatusCode: 403, RequestID: 2392xxxx-x452-4bd4-80c3-452403baxxxx, api error InvalidClientTokenId: The security token included in the request is invalid

If the profile is enabled with MFA and you are trying to run some IAM modification, then you should encounter the above error. Use --no-session flag with the command. Read in-depth regarding the no-session flag here.

How to configure switching IAM roles in AWS CLI?

A short howto on configuring AWS CLI to switch roles

AWS CLI Switch Roles configuration

Requirement:

You have one AWS account that needs to switch roles before executing things on AWS. It’s an easy method on AWS console, but how to switch roles in AWS CLI.

Solution:

Let’s consider the below setup-

  • AWS IAM account with programmatic access – user101
  • Same IAM account having sts:AsumeRole permissions.
  • AWS IAM role for above said IAM user to assume (same or cross-account)- role101

Start with configuring the AWS CLI in a standard way.

$ aws configure --profile user101
AWS Access Key ID [None]: AKIAQX3SNXZGUQFOSK4T
AWS Secret Access Key [None]: 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
Default region name [us-east-1]: us-east-1
Default output format [None]: json

Note: It is not a good practice to keep AWS credentials in a plain text format. Keep them in a secured encrypted way using aws-auth.

Now, at this point, you must have an AWS credentials file created in the home directory.

$ cd ~/.aws
$ cat credentials
[user101]
aws_access_key_id = AKIAQX3SNXZGUQFOSK4T
aws_secret_access_key = 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
region = us-east-1
output = json

You need to edit the above credentials file to add IAM role details. Append the below configuration in the file.

If you are working with AWS Gov Cloud make sure the ARNs has proper AWS Partition defined. E.g. arm:aws-us-gov:x:x:…..
[role101]
role_arn = arn:aws:iam::xxxxxxxxx:role/role101
output = json
source_profile = user101

where –

  • role101 is a Role identifier. You can choose as per your choice.
  • Mention the correct IAM role ARN
  • source_profile should use the profile identifier of the user who will assume this role. In our case, its user101.

Save the file, and you are ready to go.

Test configurations –

$ aws sts get-caller-identity
{
    "UserId": "AIDAQX3SNXZG3Z2AXNIMJ",
    "Account": "xxxxxxxxx",
    "Arn": "arn:aws:iam::xxxxxxxxx:user/user101"
}

$ aws sts get-caller-identity --profile role101
{
    "UserId": "AROAQX3SNXZG6KL4YENFZ:botocore-session-1631087792",
    "Account": "xxxxxxxxx",
    "Arn": "arn:aws:sts::xxxxxxxxx:assumed-role/role101/botocore-session-1631087792"
}

You can see this by using --profile role101 we are assuming the IAM role role101 for the user user101.

AWS CLI configuration for switching roles using MFA

Note: If you are on Windows and using GitBash, refer to configuring GitBash for MFA prompts. It works perfectly in Powershell.

In some cases, your AWS environment must have MFA restrictions in place where the user user101 must have MFA enabled to switch to the role role101. In such a scenario, your role profile in credentials files should include MFA device ARN as well like below –

[role101]
role_arn = arn:aws:iam::xxxxxxxxx:role/role101
mfa_serial = arn:aws:iam::xxxxxxxxx:mfa/user101
output = json
source_profile = user101

where –

mfa_serial is the ARN of the MFA device of user101.

You will be prompted to supply the MFA code whenever you use profile role101 in AWS CLI commands.

$ aws sts get-caller-identity --profile role101
Enter MFA code for arn:aws:iam::xxxxxxxxx:mfa/user101:
{
    "UserId": "AROAQX3SNXZG6KL4YENFZ:botocore-session-1631089277",
    "Account": "xxxxxxxxx",
    "Arn": "arn:aws:sts::xxxxxxxxx:assumed-role/role101/botocore-session-1631089277"
}