Tag Archives: AWS

Self-Managed S3 Buckets for Lambda Code: You Finally Own the Artifact

Picture a platform team running a few hundred Lambda functions across a handful of accounts. Every function has several published versions kept around for safe rollbacks. One morning a deploy fails with a quota error, and the culprit isn’t the new function at all. It’s the 75 GB account-level limit on function and layer code storage, quietly filled up by copies of packages you thought lived in your own S3 bucket. If you’ve ever had to explain to a security reviewer why your deployment artifacts also sit in an internal bucket you can’t see, encrypt, or tag, this one is for you.

AWS has added a way to point Lambda at your own S3 bucket and have it read your code from there directly, with no hidden second copy.

What actually changed

There’s a new function configuration setting called S3ObjectStorageMode. It has two values.

The default is COPY, which behaves exactly like Lambda always has. If you don’t set the field at all, you get COPY, so nothing about your existing functions changes on its own.
The new value is REFERENCE. Set it when you create or update a function, and Lambda stops copying your .zip package into its internal storage. Instead it keeps a reference to your S3 object and reads the code from your bucket when it needs it. Your object becomes the one canonical artifact.

The feature works in all AWS standard regions where Lambda is available, and there’s no extra charge for it beyond the normal S3 storage and request costs you’d pay anyway.

How it works, old way and new

Under the old model (now COPY), the flow looks like this: you upload a .zip to your bucket, call CreateFunction or UpdateFunctionCode with the bucket name and object key, and Lambda pulls that artifact into a service-managed bucket. It builds the optimized, runnable version of your function from that internal copy. The catch is that this copy counts against your 75 GB account quota, and you have no say over how it’s stored.

With REFERENCE, that copy step disappears. Lambda records where your object lives and reads it directly. Two things fall out of that. First, your package no longer counts toward the 75 GB limit, because there’s no Lambda-side copy to count. Second, creating and updating functions gets faster, since Lambda skips the copy-into-internal-bucket step. AWS describes this as a faster time to first invoke for new functions and after updates.

Architecture

The diagram below contrasts the two modes and sketches the multi-account pattern that, in my experience, is the real reason to adopt this.

Two modes for S3 buckets of Lambda code.

In a COPY deployment, the artifact exists twice: once in your bucket, once inside Lambda. In REFERENCE mode there’s a single object, and your function holds a pointer to it. Extend that to an organization and the shape gets interesting: put every artifact in one bucket in a central “artifact” or shared-services account, then grant s3:GetObject to each workload account’s Lambda execution role through the bucket policy. Now one bucket is the source of truth for what’s deployed everywhere, with one place to enforce encryption, versioning, and retention.

When to reach for it

CI/CD and artifact management. Your pipeline uploads a package once, and the function references that same object. One set of lifecycle rules and access controls covers everything, and a rollback becomes “point the function at the previous S3 object version.”
Multi-account, multi-team setups. Centralize artifacts in one account, hand out cross-account s3:GetObject via bucket policies, and keep a single inventory of what code runs where.
Disaster recovery. Because you own the bucket, you can turn on Cross-Region Replication (CRR) or Same-Region Replication (SRR), and pair it with S3 Versioning and Object Lock to keep a tamper-resistant archive that survives an accidental delete or a corrupted deploy. See the S3 replication docs and S3 lifecycle examples.
Quota and compliance pressure. If you’re brushing up against 75 GB, or you need your own encryption, access logging, Object Lock, or compliance tags on the artifact, this gives you that control.

When to leave it alone

For plenty of workloads, COPY is fine and simpler. If you aren’t near the quota, don’t need custom encryption or tagging on the artifact, and have no DR requirement on your deployment packages, there’s little reason to change anything. The default exists for a reason.

Security notes

The trade in REFERENCE mode is control for responsibility. You now own the bucket’s posture: its encryption, access policies, lifecycle transitions, and audit trail are yours to configure and to get right.
For cross-account access, you grant s3:GetObject to the calling function’s execution role in the bucket policy.
Versioning plus Object Lock is the combination worth setting up early if you care about a durable, tamper-proof code archive.
The full IAM and bucket-policy details live in the Lambda developer guide.

Pricing

No additional charge for the feature. You pay the standard S3 storage and request costs for the object, which you were largely paying already if your artifacts lived in S3.

Limitations and open questions

It doesn’t state whether REFERENCE needs KMS decrypt permissions beyond s3:GetObject when you use customer-managed encryption (It should), exactly how a referenced object version is pinned for rollback, or what happens to a live function if the referenced object is later deleted or modified. Any latency effect from reading code directly at runtime isn’t discussed either. Confirm these against the Lambda console and the developer guide before you roll it out widely.

Bottom line

This is a small setting with an outsized effect for teams operating at scale. If the 75 GB quota or “we can’t audit that copy” has ever slowed you down, S3ObjectStorageMode: REFERENCE is worth a look. Start on a non-critical function, get your bucket policy and versioning right, and expand from there. Original announcement on the AWS Compute Blog.

Lambda MicroVMs: When Functions Aren’t Enough and EC2 Is Too Much

Leave a reply

Picture this: you’re building a browser-based notebook where data analysts paste in Python, load a 3 GB dataframe, generate a few charts, then wander off to a meeting. Ninety minutes later they come back and expect their kernel, their variables, and their half-finished plot to still be sitting there.

Now try to build that on what AWS gave you before June 2026. Lambda? Fifteen-minute execution ceiling, no persistent process between invocations, no way to hold that dataframe in memory across the analyst’s coffee break. ECS or EC2? Sure, but now you’re running per-user containers or VMs, paying for idle capacity, and writing your own scheduler to reap dead sessions. Fargate gets you closer, but you still own the isolation story when the code being run was generated by an LLM you can’t fully trust.

This is the gap Lambda MicroVMs is aimed at.

What actually launched

AWS Lambda MicroVMs is a new compute primitive that exposes the Firecracker virtualization layer (the same one that has always run underneath Lambda) as something you can address directly. You get per-instance hardware isolation, snapshot-based startup, and — this is the part that matters — the ability to keep a single execution environment alive for an entire working session rather than a single request.

Alongside it, AWS shipped a companion resource called the Lambda Network Connector (LNC), which is how you attach a MicroVM to a private VPC when you need it to reach a database or an internal API.

How it actually works

Two resource types, and once you understand these the rest falls into place:

MicroVM image: a versioned artifact you build from a Dockerfile. When you create one, the service runs your Dockerfile, boots your application inside a MicroVM, and takes a Firecracker snapshot of the memory and disk state. Think of it as a “warm” image — dependencies already imported, JIT already warmed, whatever init your app does already done.
MicroVM: an instance launched from that image. Because it’s restored from a snapshot rather than cold-booted, it comes up close to instantly.

Each MicroVM gets its own HTTPS endpoint, and that endpoint speaks to individual ports on the guest — plain HTTPS, WebSockets, and gRPC all work, so you connect to it the same way you’d connect to any container you were running yourself.

On sizing: the default baseline is 2 GB of memory and 1 vCPU. You can configure that up to 8 GB and 4 vCPUs at launch, with vCPU pinned to memory at a 2:1 ratio (memory in GB is double the vCPU count). From whatever baseline you pick, a MicroVM will auto-scale vertically up to 4x during peak demand. Horizontally, the service claims you can launch several hundred MicroVMs inside a minute.

Sessions can run from a few minutes up to eight hours. Egress to the public internet works out of the box; VPC access requires the LNC.

Architecture

See the diagram below. The pattern is the per-session model: user requests come into your control plane, the control plane looks up whether that user already has a live MicroVM, and either routes traffic to the existing HTTPS endpoint or launches a new instance from a MicroVM image. When the user goes idle, you decide the lifecycle — keep it warm, snapshot it, or let it go.

The interesting design question isn’t really “how do I launch a MicroVM” — it’s “who owns the session-to-endpoint mapping, and what’s my policy when the user disconnects?” You will end up building a small state machine. Plan for it.

When this is the right tool

Browser-based IDEs, notebooks, and the current wave of vibe-coding platforms — anywhere users bring their own code and expect their environment to feel persistent.
Analytics platforms running user- or LLM-generated queries where the working set is large and the session is long.
AI coding assistants that iterate on generated code and want to hold context between iterations, including RL-style loops that spin environments up and down to compare execution paths.
Security and vulnerability scanning where you need real isolation between scans and sometimes elevated OS privileges inside the guest.
CI/CD build and test runners where each job wants a clean, isolated box that starts fast.

When it probably isn’t

The launch material doesn’t call out anti-patterns directly, so treat this as my read rather than AWS’s guidance:

If your workload is stateless per-invocation and short, regular Lambda is still simpler and cheaper reasoning-wise.
If you need more than 8 GB of memory or 4 vCPUs per instance, you’re outside the MicroVM baseline envelope and you should look at Fargate or EC2.
If your sessions genuinely need to outlive eight hours without a snapshot/resume, this isn’t your primitive either.

Security notes worth reading twice

The isolation story is genuinely strong — hardware-level, one guest per user or job, which is the whole point of using Firecracker as a primitive rather than just a container runtime. That’s what makes it defensible as a sandbox for untrusted or model-generated code.

Two things to design around, though.

First, outbound internet access is on by default. If you’re running untrusted code, you almost certainly want egress controls, and the launch material doesn’t spell out how granular those are — treat this as a question to answer before you go to production.
Second, private VPC connectivity is not a checkbox; it requires configuring an LNC. Factor that into your networking design early, not late.

Pricing

Lambda MicroVMs are priced per instance-second. Check the Lambda pricing page (MicroVMs tab) before you commit to a design, especially for long-lived sessions — an eight-hour idle MicroVM has very different economics from a 200 ms function invocation.

Limitations to keep on the whiteboard

2 GB / 1 vCPU baseline, 8 GB / 4 vCPU max, 2:1 memory-to-vCPU ratio.
Vertical auto-scale capped at 4x baseline.
Horizontal scale advertised as “several hundred per minute” during spikes — quantify this against your own burst profile before betting on it.
Sessions in the “few minutes to 8 hours” range; the exact hard upper bound isn’t explicitly stated.
VPC access is opt-in via LNC.

Wrapping up

The most useful way to think about Lambda MicroVMs is that AWS unbundled Firecracker from Lambda’s request/response model and let you address it directly, while keeping the parts of Lambda that were actually pleasant — no capacity planning, no patching, no scheduler to write. If you’ve been building per-user sandboxes on top of ECS or bare EC2 and feeling like you were re-implementing Firecracker badly, this is the primitive to evaluate.

Start with the product page and the developer guide, and if snapshot-based startup is new to you, the older SnapStart post is worth a read for the mental model. Networking details live in the LNC docs, and quotas are on the service limits page. And refer to this AWS Blog which walk you through AWS MicroVms.

Know about Amazon GuardDuty Investigation: AI-powered security analysis

Leave a reply

Security teams routinely burn hours triaging a single GuardDuty finding. The workflow is familiar: pull the finding, pivot into CloudTrail, cross-reference VPC flow logs, chase the principal across accounts, map the behavior to a known technique, and finally decide whether the alert is a real incident or noise. Multiply that across an organization producing dozens or hundreds of findings a day, and investigation backlog becomes the bottleneck — not detection.

Amazon GuardDuty is now addressing that bottleneck directly. The GuardDuty investigation is available in public preview and performs on-demand, AI-powered investigations of GuardDuty findings, returning a structured assessment that a human analyst would otherwise assemble by hand.

What changed

The investigation agent adds a new capability to Amazon GuardDuty: rather than only surfacing findings, GuardDuty can now investigate them for you. Each investigation produces a structured output that includes a risk level, a confidence score, a natural-language summary, investigation details, MITRE ATT&CK technique mappings, resource mappings, and prioritized recommended actions — including specific AWS CLI commands the analyst can execute.

The agent is accessible through the AWS Management Console, the AWS CLI, AWS APIs, AWS SDKs, and the AWS MCP server — meaning it slots into both console-driven workflows and agentic security tooling built on top of the Agent Toolkit for AWS.

How it works

An investigation is initiated by an caller with appropriate permissions through the GuardDuty console, by calling the CreateInvestigation API, or via the AWS CLI. The target can be:

a specific GuardDuty finding
a member account
an entire AWS organization

CLI and API callers can supply a free-form natural-language trigger prompt of up to 2,048 characters to steer the investigation — for example, focusing the agent on a particular hypothesis or scope.

Under the hood, the agent correlates findings and evidence and returns a structured assessment. Results are retrieved with GetInvestigation, and prior investigations can be enumerated with ListInvestigations.

Processing uses cross-Region inference via the Cross-Region Inference Service (CRIS). Investigation data remains stored in the Region where the investigation was created, but inference and summary generation may occur in another Region within the same geography, transmitted over Amazon’s encrypted network.

Please go through this very detailed AWS blog that walks you through the whole investigation process – Amazon GuardDuty investigation agent: on-demand AI-powered threat assessment.

Architecture

The accompanying diagram shows the request flow: an administrator triggers an investigation from the console, CLI, SDK, API, or an MCP-connected agent; GuardDuty accepts the request in the origin Region; CRIS performs inference in-geography; and the structured assessment is returned to the caller while the underlying investigation data stays resident in the origin Region.

When to use it

The investigation agent fits naturally when you want to:

Triage a single GuardDuty finding without a manual pivot chase
Assess security posture across an entire AWS organization
Reduce manual correlation of evidence spread across CloudTrail, VPC flow logs, and other GuardDuty data sources
Wire GuardDuty investigations into AI-assisted security workflows through the AWS MCP server
Produce on-demand, structured threat assessments consumable by downstream automation

When not to use it

The agent is not the right tool if:

GuardDuty is not enabled in the account or organization
You are operating in a Region that does not support the preview
You need a member account to view another member’s or the administrator’s investigations, which is not permitted

Security considerations

Architecturally, three points matter for a review.

First, cross-Region inference. Investigation data is stored only in the origin Region, but processing and summary results may traverse another Region in the same geography. If your data-residency posture is scoped to a single Region rather than a geography, this is worth an explicit review before enabling the feature in regulated workloads. Please refer geography and their inference regions here.

Second, transport. Data moves across Amazon’s internal, encrypted network — but the residency point above still stands independently of encryption in transit.

Third, authorization. Investigations honor the existing GuardDuty authorization model. Callers can investigate only accounts they are already authorized to access. The three IAM actions to allow on principals that will drive investigations are:

guardduty:CreateInvestigation
guardduty:GetInvestigation
guardduty:ListInvestigations

Scope these on the administrator account role that runs the SOC’s investigation workflow, not broadly.

Pricing

In Preview mode, GuardDuty Investigations are made available at free of cost. Confirm current pricing in the GuardDuty product page before enabling at scale once its GA.

Limitations

The feature is in public preview.
GuardDuty must already be enabled, the account must be in a supported Region, and only administrator accounts can create investigations in admin as well as member accounts.
Member accounts cannot access investigations belonging to peer members or to the administrator.
Cross-Region inference behavior described above also applies.
During preview, 10 investigations/account/day with total limit of 100 investigations/account.Failed investigations do not count toward these quotas.
This feature is available only in the following 10 commercial AWS Regions: US East (N. Virginia), US East (Ohio), US West (Oregon), Canada (Central), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Paris), Europe (Stockholm), and Asia Pacific (Tokyo).
The trigger prompt is capped at 2,048 characters when invoked through API/CLI.

Verify latest limitations against the GuardDuty investigation documentation before committing to a design.

Conclusion

The investigation agent shifts GuardDuty from a detection surface to a triage surface. For teams whose incident response cost is dominated by evidence correlation rather than detection, that is the interesting move. Preview status means the mechanics — Regions, quotas, latency, retention — need to be validated against your environment before it lands in a runbook, but the shape of the workflow is clear enough to start piloting against a defined scope, most sensibly a single administrator account or a bounded set of high-signal findings. Details and getting-started guidance are on the AWS Security Blog and in the GuardDuty documentation.

Exploring the Latest AWS Console-to-Code Feature

Leave a reply

On November 2023 AWS announced the Preview going live for the new feature AWS Console-to-Code. Two months later, in this blog, we will explore this feature, learn about how to use it, what are the limitations, etc.

What is the AWS Console-to-Code feature?

It’s the latest feature from AWS made available in the EC2 console that leverages Generative AI to convert the actions performed on the AWS EC2 console into the IaC (infrastructure as Code) code! It’s a stepping stone towards IaC creation methods in the world of AWS cloud.

The actions carried out on the AWS console during the current session are monitored by the feature in the background. These recorded actions are then made available to the user to select up to 5 of these actions, along with their preferred language. AWS then utilizes its Generative AI capabilities to automatically generate code that replicates the objectives achieved through manual actions on the console.

It also generates the AWS CLI command alongside the IaC code.

The usefulness of the AWS Console-to-Code feature

With the current list of limitations and the preview stage, this feature might not be a game changer but it does have potential in the future. The AWS Console-to-Code feature will surely help developers and administrators to get the IaC skeleton quickly to start from and speed up the IaC coding with less effort.

This feature simplifies the process of generating AWS CLI commands, eliminating the need to constantly consult documentation and manually construct commands with the correct arguments. As a result, it accelerates automation deliveries with reduced effort.

By the way, there is no additional cost to use Console-to-Code so it doesn’t hurt to employ it for initial IaC drafting!

Limitation of AWS Console-to-Code feature

Currently, it’s in the ‘Preview’ phase.
Only available in North Virginia (us-east-1) region as of today.
It can generate IaC code in the listed types and languages only –
- CDK: Java
- CDK: Python
- CDK: TypeScript
- CloudFoprmation: JSON
- CloudFoprmation: YAML
It does not retain data across sessions. The actions that are performed in the current session are made available for Code Generation. Meaning if you refresh the browser page, it resets the action list and starts recording afresh.
Up to 5 actions can be selected to generate code.
Actions from the EC2 console only are recorded. However, I observed even a few actions like Security Group creation or Volume listing, etc. are not being recorded.

How to use the AWS Console-to-Code feature

Login to the EC2 console and select region N. Virginia (us-east-1)
On the left-hand side menu, ensure you have a Console-to-Code link.
Perform some actions in the EC2 console like launching an instance, etc.
Navigate to Console-to-Code by clicking on the link in the left-hand side menu.
It will present you with a list of recorded actions. Select one or a maximum of 5 actions for which you want to generate code. You can even filter the recorded actions as per their Type:
- Show read-only: Read-only events like Describe*
- Show mutating: Events that modified/created/deleted or altered the AWS resources.
Click on the drop-down menu and select the type and language for the code.

It should start generating code.
After code generation, you have an option to copy or download it. You can also copy the AWS CLI command on the same page.

*Python code generated by AWS Console-to-Code*

It also provides the generated code’s explanation at the bottom of the code.

How to install Cluster Autoscaler on AWS EKS

Leave a reply

A quick rundown on how to install Cluster Autoscaler on AWS EKS.

What is Cluster Autoscaler (CA)

Cluster Autoscaler is not a new word in the Kubernetes world. It’s a program that scales out or scales in the Kubernetes cluster as per capacity demands. It is available on Github here.

For scale-out action, it looks for any unschedulable pods in the cluster and scale-out to make sure they can be scheduled. If CA is running with default settings, then it checks every 10 seconds. So basically it detects and acts for scale-out in 10 secs.

For scale in action it watches nodes for their utilization and any underutilized node will be elected for scale in. The elected node will have to remain in an un-needed state for 10 minutes for CA to terminate it.

CA on AWS EKS

As you know now, CA’s core functionality is spawning new nodes or terminating the un-needed ones, it’s essential it must be having underlying infrastructure access to perform these actions.

In AWS EKS, Kubernetes nodes are EC2 or FARGATE compute. Hence, Cluster Autoscaler running on EKS clusters should be having access to respective service APIs to perform scale out and scale in. It can be achieved by creating an IAM role with appropriate IAM policies attached to it.

Cluster Autoscaler should be running in a separate namespace (kube-system by default) on the same EKS cluster as a Kubernetes deployment. Let’s look at the installation

How to install Cluster Autoscaler on AWS EKS

Creating IAM role

IAM role of Autoscaler needs to have an IAM policy attached to it with the below permissions –

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "sts:AssumeRole",
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "ec2:DescribeLaunchTemplateVersions"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

You will need to use this policy ARN in eksctl command. Also, make sure you have an IAM OIDC provider associated with your EKS cluster. Read more in detail here.

As mentioned above, we need to have an IAM role in a place that can be leveraged by Cluster Autoscaler to perform resource creation or termination on AWS services like EC2. It can be done manually, but it’s recommended to perform it using eksctl command for its comfort and perfection! It takes care of trust relationship policy and related conditions while setting up a role. If you do not prefer eksctl then refer to this document to create it using AWS CLI or console.

You need to run it from the terminal where AWS CLI is configured.

# eksctl create iamserviceaccount --cluster=<CLUSTER-NAME> --namespace=<NAMESPACE> --name=cluster-autoscaler --attach-policy-arn=<MANAGED-POLICY-ARN> --override-existing-serviceaccounts --region=<CLUSTER-REGION> --approve

where –

CLUSTER-NAME: Name of the EKS Cluster
NAMESPACE: ns under which you plan to run CA. Preference: kube-system
CLUSTER-REGION: Region in which EKS Cluster is running
MANAGED-POLICY-ARN: IAM policy ARN created for this role

# eksctl create iamserviceaccount --cluster=blog-cluster --namespace=kube-system --name=cluster-autoscaler --attach-policy-arn=arn:aws:iam::xxxxxxxxxx:policy/blog-eks-policy --override-existing-serviceaccounts --region=us-east-1 --approve
2022-01-26 13:45:11 [&#x2139;]  eksctl version 0.80.0
2022-01-26 13:45:11 [&#x2139;]  using region us-east-1
2022-01-26 13:45:13 [&#x2139;]  1 iamserviceaccount (kube-system/cluster-autoscaler) was included (based on the include/exclude rules)
2022-01-26 13:45:13 [!]  metadata of serviceaccounts that exist in Kubernetes will be updated, as --override-existing-serviceaccounts was set
2022-01-26 13:45:13 [&#x2139;]  1 task: {
    2 sequential sub-tasks: {
        create IAM role for serviceaccount "kube-system/cluster-autoscaler",
        create serviceaccount "kube-system/cluster-autoscaler",
    } }2022-01-26 13:45:13 [&#x2139;]  building iamserviceaccount stack "eksctl-blog-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler"
2022-01-26 13:45:14 [&#x2139;]  deploying stack "eksctl-blog-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler"
2022-01-26 13:45:14 [&#x2139;]  waiting for CloudFormation stack "eksctl-blog-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler"
2022-01-26 13:45:33 [&#x2139;]  waiting for CloudFormation stack "eksctl-blog-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler"
2022-01-26 13:45:50 [&#x2139;]  waiting for CloudFormation stack "eksctl-blog-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler"
2022-01-26 13:45:52 [&#x2139;]  created serviceaccount "kube-system/cluster-autoscaler"

The above command prepares the JSON CloudFormation template and deploys it in the same region. You can visit the CloudFormation console and check it.

Installation

If you choose to run CA in different namespace by defining custom namespace in manifest file, then replace kube-system with appropriate namespace name in all below commands.

Download and prepare your Kubernetes to manifest file.

# curl -o cluster-autoscaler-autodiscover.yaml https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml
# sed -i 's/<YOUR CLUSTER NAME>/cluster-name/g' cluster-autoscaler-autodiscover.yaml

Replace cluster-name with EKS cluster name.

Apply the manifest to your EKS cluster. Make sure you have the proper context set for your kubectl command so that kubectl is targeted to the expected EKS cluster.

# kubectl apply -f cluster-autoscaler-autodiscover.yaml
serviceaccount/cluster-autoscaler configured
clusterrole.rbac.authorization.k8s.io/cluster-autoscaler created
role.rbac.authorization.k8s.io/cluster-autoscaler created
clusterrolebinding.rbac.authorization.k8s.io/cluster-autoscaler created
rolebinding.rbac.authorization.k8s.io/cluster-autoscaler created
deployment.apps/cluster-autoscaler created

Add annotation to cluster-autoscaler service account with ARN of the IAM role we created in the first step. Replace ROLE-ARN with IAM role arn.

# kubectl annotate serviceaccount cluster-autoscaler -n kube-system eks.amazonaws.com/role-arn=<ROLE-ARN>

$ kubectl annotate serviceaccount cluster-autoscaler -n kube-system eks.amazonaws.com/role-arn=arn:aws:iam::xxxxxxxxxx:role/eksctl-blog-cluster-addon-iamserviceaccount-Role1-1X55OI558WHXF --overwrite=true
serviceaccount/cluster-autoscaler annotated

Patch CA for adding eviction related annotation

# kubectl patch deployment cluster-autoscaler -n kube-system -p '{"spec":{"template":{"metadata":{"annotations":{"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"}}}}}'
deployment.apps/cluster-autoscaler patched

Edit CA container command to accommodate below two arguments –

--balance-similar-node-groups
--skip-nodes-with-system-pods=false

# NEW="        - --balance-similar-node-groups\n        - --skip-nodes-with-system-pods=false"
# kubectl get -n kube-system deployment.apps/cluster-autoscaler -o yaml | awk "/- --node-group-auto-discovery/{print;print \"$NEW\";next}1" > autoscaler-patch.yaml
# kubectl patch deployment.apps/cluster-autoscaler -n kube-system --patch "$(cat autoscaler-patch.yaml)"
deployment.apps/cluster-autoscaler patched

Make sure the CA container image is the latest one in your deployment definition. If not you can choose a new image by running –

# kubectl set image deployment cluster-autoscaler -n kube-system cluster-autoscaler=k8s.gcr.io/autoscaling/cluster-autoscaler:vX.Y.Z

Replace X.Y.Z with the latest version.

$ kubectl set image deployment cluster-autoscaler -n kube-system cluster-autoscaler=k8s.gcr.io/autoscaling/cluster-autoscaler:v1.21.1
deployment.apps/cluster-autoscaler image updated

Verification

Cluster Autoscaler installation is now complete. Verify the logs to make sure Cluster Autoscaler is not throwing any errors.

# kubectl -n kube-system logs -f deployment.apps/cluster-autoscaler