For years, running EKS meant accepting that the control plane was a black box you paid for but couldn’t tune. Want the scheduler to pack pods tighter to cut node count? Write your own scheduler plugin or run a second scheduler alongside the default one. Want HPA to react faster than the stock 15-second loop? You couldn’t — that number was baked into every managed cluster, full stop. Want shorter event retention so a CI-heavy namespace doesn’t bloat etcd? Not your call.
Control Plane configurable parameters
AWS announced in mid-August 2026 that this is no longer true. EKS now exposes a set of advanced configuration parameters across the scheduler, controller manager, and API server, set directly through CreateCluster and UpdateClusterConfig. No sidecar schedulers, no forked control plane, no support ticket asking AWS nicely. If you run a shared EKS platform serving multiple teams — which is exactly the kind of cluster where these defaults start to hurt — this is worth a proper look rather than a skim of the release notes.
What actually changed
Four parameters, across three components:
Component
Parameter
Range
Default
Needs Provisioned Control Plane
kube-scheduler
nodeResourcesFit.scoringStrategy
LeastAllocated, MostAllocated
LeastAllocated
No
kube-controller-manager
horizontalPodAutoscalerSyncPeriod
10s–15s
15s
Yes
kube-apiserver
eventTtl
10m–60m
60m
No
kube-apiserver
serviceNodePortRange
10260–32767
30000–32767
No
That’s a narrow slice of what upstream Kubernetes actually lets you configure, and I’d guess it stays narrow for a while — AWS is clearly starting with the parameters that are safe to expose without risking control plane stability, not opening the floodgates. Each one is set through a component-specific config block (kubeSchedulerConfig, kubeControllerManagerConfig, kubeApiServerConfig), works on Kubernetes 1.31+, and shows up in describe-cluster output along with every default you haven’t touched. Changes go through a rolling update of the control plane, so don’t expect them instant — budget a few minutes and poll with DescribeUpdate or aws eks wait cluster-active if you’re scripting it.
Scheduler: MostAllocated is the interesting one
This is the parameter I’d actually reach for first on a shared platform. By default, Kubernetes scores nodes with LeastAllocated, which spreads pods thin across your fleet so every node keeps headroom. That’s a reasonable default when you don’t know your workloads. It’s also exactly why clusters running steady-state services end up with fifteen half-empty nodes instead of ten well-used ones.
Switch to MostAllocated and the scheduler starts favoring nodes that are already carrying load, packing new pods onto them instead of spreading out. Combine that with Karpenter or Cluster Autoscaler consolidation and lightly used nodes actually get reclaimed over time — this is where the cost story lives, not in the scheduling decision itself.
You can also weight which resources drive scoring:
Worth being precise here because the behavior is easy to misread. Weights are relative, not absolute — gpu: 100 next to cpu: 1 doesn’t mean CPU stops mattering, it means CPU only breaks ties once GPU availability is identical across candidates. The sharper gotcha is that omitting a resource from the list isn’t the same as giving it a low weight — leave memory out entirely and it’s excluded from scoring altogether, not just deprioritized. If you’re running mixed CPU and GPU node groups, that distinction is the difference between a config that behaves as intended and one that quietly ignores memory pressure.
Two things won’t change no matter which strategy you pick. First, the scheduler never moves pods that are already running — flipping the strategy only affects future placement, so if you’re trying to consolidate an already-packed cluster, you still need to evict or roll the workloads yourself. Second, filtering behavior is untouched; a pod that genuinely doesn’t fit on a node still won’t land there. And the honest trade-off with MostAllocated: you’re concentrating blast radius. Pack workloads onto fewer nodes and losing one of them, or an AZ event taking out a chunk of your fleet, now affects more pods per incident. On a cluster running twenty teams’ workloads, that’s not a hypothetical — I’d pilot this on a subset of node groups before flipping it cluster-wide.
HPA sync period: faster reaction, but only if you’ve earned it
Shortening horizontalPodAutoscalerSyncPeriod from the default 15 seconds down to 10 sounds like a free win — your workloads scale out sooner after a traffic spike. It isn’t free, and AWS gates it behind Provisioned Control Plane for a good reason: the HPA controller has to reconcile every HorizontalPodAutoscaler object in the cluster within that window, and shortening the window from 15s to 10s cuts the number of HPA objects your control plane can keep up with by roughly a third.
Here’s the part that’ll bite someone: EKS doesn’t validate the sync period against how many HPA objects you actually have. Set it to 10s on a cluster with more HPAs than that tier supports, and the update succeeds. Nothing errors. What happens instead is quieter and worse — some HPA objects stop getting reconciled on schedule, autoscaling starts responding slower than before you touched anything, and there’s no alarm or event telling you why. Before anyone on your team shortens this, run:
kubectl get hpa --all-namespaces --no-headers | wc -l
and check that number against your scaling tier’s supported count. If you inherit a cluster where someone already set this and autoscaling feels sluggish, that command is your first move, not a Datadog dashboard.
One more thing worth flagging to anyone managing the control plane scaling tier: once horizontalPodAutoscalerSyncPeriod is off default, you can’t move that cluster back from Provisioned to Standard mode. You have to reset the sync period to 15s first, then downgrade the tier. It’s a small detail, but it’s the kind of thing that turns into a surprised Slack message during a cost-optimization pass six months from now.
Event TTL and NodePort range: smaller blast radius, still worth reading the fine print
eventTtl is the simplest lever here — how long the API server holds onto Kubernetes events before deleting them, tunable from the default 60 minutes down to 10. On a cluster running CI/CD, batch, or AI workloads that generate thousands of events an hour, that’s real etcd pressure and real API server list latency you’re trimming. The catch: a shorter TTL only applies to events created after the change. Existing events keep whatever retention was active when they were written, so the storage benefit shows up gradually as the old events age out — not the moment UpdateClusterConfig returns. And once an event’s gone, it’s gone; if you lean on kubectl get events for postmortems, make sure something durable is already scraping events externally before you shorten this.
serviceNodePortRange is the one I’d reach for during a lift-and-shift rather than day-to-day tuning. Legacy apps that expect services on specific fixed ports outside Kubernetes’ default 30000–32767 window used to force a choice: rewrite the app, or bolt on a proxy. Now you can widen or shift the range (bounded to 10260–32767, to stay clear of kubelet/kube-proxy health ports on the low end and the Linux ephemeral port range on the high end) and let the migrated app keep its original ports. Just remember it’s cluster-wide and non-retroactive — narrowing the range doesn’t kick out services already holding an out-of-range port, but recreating one of those services will fail to get that port back.
Where this actually earns a place in your platform
None of these four parameters change the availability or performance envelope of your control plane — AWS is explicit that clusters keep the same SLAs regardless. What they change is whether your platform team keeps working around upstream Kubernetes defaults with sidecar tooling, or just configures the thing directly and gets it recorded in CloudTrail like any other cluster change. On a shared EKS platform, the honest advice is: pilot the scheduler strategy on a low-risk node group first, treat the HPA sync period as something you validate with a headcount check rather than assume is safe, and use event TTL as a lever specifically for your noisiest CI or batch namespaces rather than a blanket cluster setting. Terraform and ACK support isn’t there yet — it’s Console, CLI, SDKs, CloudFormation, and CDK for now — so if your provisioning is Terraform-first, you’re looking at a temporary CLI-driven exception until that support lands.
The bigger signal here isn’t really any one parameter. It’s that AWS is starting to treat the EKS control plane as something you configure, not just something you consume. Worth watching what gets added to this list next.
By this point in the series you’ve picked a partition key, chosen EKS or ECS as the substrate, worked out the networking, built in shuffle sharding and bake time, and made peace with what it all costs. What’s left is the least glamorous problem and the one that actually determines whether any of the above survives contact with a real team: how do you provision and update dozens of near-identical cells without dozens of near-identical ways for them to drift apart.
Shipping cells!
Infrastructure as code: the stamp pattern
The core idea is simple to state and easy to get subtly wrong in practice: write one module that describes a cell, and apply it once per cell with different inputs. A hands-on Terraform walkthrough of this pattern frames it as a blueprint you stamp out repeatedly, alongside a separately-built router layer that isn’t part of any individual cell’s module at all.
A minimal shape for that module looks something like this:
module "cell" {
source = "./modules/cell"
for_each = var.cells
cell_id = each.key
account_id = each.value.account_id
region = each.value.region
az = each.value.az
tenant_range = each.value.tenant_range
}
variable "cells" {
type = map(object({
account_id = string
region = string
az = string
tenant_range = string
}))
}
The detail that matters more than the module code itself is state isolation. Each cell’s Terraform state needs to be genuinely separate — a distinct backend key per cell, not just a shared workspace inside one state file — so that a plan or apply against cell-07 can’t accidentally touch cell-03. A shared state file for “all cells” quietly reintroduces exactly the coupling the whole architecture exists to avoid, just at the tooling layer instead of the runtime layer.
CDK works the same way conceptually, even though the syntax differs. AWS’s own reference implementation structures this as three separate stacks — one for shared ECR repositories, one for the routing components, and one for the cell blueprint, which gets deployed repeatedly, once per cell. Terraform, CDK, or CloudFormation, the stamp pattern is the same idea wearing different clothes.
A cell registry as the single source of truth
Once you’re past a handful of cells, you want one place that lists every cell that exists — its account, region, AZ, tenant range, and lifecycle status (active, draining, newly provisioned). This can be as simple as a cells.yaml checked into the same repo as your Terraform, or as robust as a DynamoDB table if other systems need to query it at runtime. What matters isn’t the storage mechanism, it’s that this registry becomes the one input that drives both your infrastructure provisioning and your application deployment generator — because the moment those two read from different sources of truth, you’ve created a gap where they can disagree about how many cells exist.
Cell registry
Deploying applications with ArgoCD
On the application side, ArgoCD’s ApplicationSet controller is built for exactly this fan-out. A list or git-file generator, sourced from the same cell registry, produces one child Application per cell (or per cell-and-app combination), rather than hand-maintaining one YAML file per cell per application.
Platform-wide baseline components — an ingress controller, cert-manager, the monitoring agents every cell needs regardless of tenant — are a good fit for a separate App-of-Apps hierarchy, kept distinct from the per-cell, per-tenant applications so a change to the shared baseline doesn’t get tangled up with a single cell’s release train.
Sync waves are worth using deliberately here: router configuration and any cell-registration step should land before the cell’s own workloads sync, not after, or you risk a cell coming online with traffic already routed to it before its services exist. And each cell’s rollout should carry its own canary and bake time — tying back to the deployment discipline from earlier in this series — rather than one canary standing in for the whole fleet. A fleet-wide canary defeats the purpose of having cells in the first place; the point was that a bad release only ever touches one of them at a time.
The habit that actually keeps this from falling apart
None of this tooling prevents drift on its own. What prevents drift is treating the cell registry as a change that goes through the same pull request and review path as any other production change, and making sure both the Terraform plan and the ArgoCD ApplicationSet read from that same file. When someone adds cell-12 to the registry, that one commit should be the trigger for both the infrastructure to get stamped out and the applications to start deploying into it — not two separate manual steps that someone has to remember to keep in sync.
That’s really the throughline of this whole series. Cell-based architecture starts as a resiliency idea — smaller blast radius, contained failure — but by the time it’s running in production it’s mostly a delivery-engineering discipline. The isolation only holds up if your pipeline can safely stamp out cell N+1 in an afternoon, deploy into it with the same confidence as cell N, and tear it down just as cleanly if it turns out you didn’t need it. If your tooling can’t do that yet, the resilience story is still theoretical — and that’s a good place to start the next iteration of the design, rather than the last one.
Cell-based architecture gets pitched almost entirely as a reliability pattern, and it is one — the AWS Well-Architected Framework files it directly under the Reliability pillar’s bulkhead guidance. But it touches all six pillars once you actually build it, and at least two of them — cost and sustainability — tend to get discovered the hard way, after the architecture is already in production and the bill has arrived.
Well architected and cost in Cell Based Arch!
Running it through all six pillars
Reliability is the pillar cells were designed for, and it’s the easy one: smaller, isolated fault domains, contained blast radius, and a natural unit for fractional deployment.
Operational excellence cuts both ways. Cells give you a clean unit for canary releases and bake time, which is a genuine operational win — but every cell you add is another entry in the on-call runbook, another set of dashboards, another thing that can silently drift out of configuration with its siblings if your automation isn’t keeping up.
Security benefits in the same shape reliability does: a leaked credential or a misconfigured IAM policy inside one cell doesn’t automatically hand over the rest of the estate. The cost is that you now have N sets of IAM roles, N certificates, and N network boundaries to keep patched and rotated instead of one.
Cost optimization is where the trade-off gets sharp, and it’s covered in more detail below.
Performance efficiency tends to improve with smaller cells, because a noisy neighbor problem shrinks to the size of one cell instead of the whole platform. AWS’s ad-serving case is a clean illustration in the other direction: when tenants shared in-memory state on common infrastructure, one tenant with a large dataset could trigger memory pressure that degraded everyone sharing that heap, which is precisely the failure mode dedicated per-tenant compute was brought in to solve.
Sustainability is the pillar cell-based architecture can quietly work against. Fixed-cost resources replicated per cell — a NAT gateway sized for peak, a node pool that never scales below some minimum — sit idle a lot of the time by design, since isolation means you can’t pack unrelated tenants’ spare capacity together. Idle reserved capacity is wasted energy whether or not the AWS bill makes that obvious.
The cost math that catches people off guard
A few numbers are worth having in your head before you commit to a cell count.
An EKS control plane costs roughly $0.10 an hour, which works out to around $73 a month — and that’s before a single pod runs. Ten cell-per-cluster EKS deployments means roughly $730 a month in control-plane fees alone, on top of whatever compute those clusters actually run. ECS doesn’t have this problem in the same way — a cluster is a free logical construct, and billing starts at the task — which is a real reason cluster-per-tenant is a more forgiving pattern on ECS than on EKS.
AWS PrivateLink interface endpoints run close to $7.30 a month each plus data transfer, which is trivial shared once across a tier of cells and a real line item if you accidentally provision one per cell instead of pre-wiring it once at the tier level — the difference between those two choices is exactly what drove an 80 percent cut in network configuration overhead in AWS’s own ad-serving redesign.
And the starkest number in that same case study isn’t a unit price at all — it’s utilization. Before the redesign, the isolated-per-tenant fleet ran at roughly 3 percent average CPU and 19 percent average memory, with servers idle more than 98 percent of the time. That’s the cost of strict isolation taken to its logical extreme: every tenant’s capacity has to be sized for that tenant’s peak, and peaks, by definition, don’t happen most of the time.
Karpenter and the bin-packing tension
If you’re on EKS, Karpenter earns its cost savings mainly through consolidation — packing workloads onto fewer, better-utilized nodes and leaning on Spot capacity where it’s safe to. Karpenter documentation and the practical write-ups from teams running it at scale are consistent on this: it works best with a large pool of mixed workloads to bin-pack across dozens of nodes with varied sizes, not a handful of strictly-separated node pools each sized for one tenant.
Cell isolation deliberately works against that. If every cell gets its own NodePool to preserve the isolation boundary, you’re intentionally giving up some of Karpenter’s consolidation opportunity in exchange for the guarantee that one cell’s workload can’t crowd out another’s. That’s a legitimate, conscious trade — not a mistake — but it’s worth naming explicitly rather than discovering it as an unexplained line item during a cost review.
What this means for how you size cells
Everything above points back to the same conclusion from earlier in this series: start with fewer, larger cells, and shrink them as your tooling matures. A few practical habits follow directly from the cost picture:
Bundle genuinely fixed-cost resources — a VPC, a NAT gateway — across a batch of cells where the blast radius trade-off allows it, rather than one per cell.
Tier tenants by traffic profile instead of mapping every tenant to its own cell 1:1; group similar-sized tenants together and reserve full dedicated cells for the ones that actually need them.
Revisit your cell count and size against the AWS Well-Architected Tool and the SaaS Lens periodically — this isn’t a decision you make once and forget, it’s one that should move as your tenant mix and automation both change.
There’s no version of cell-based architecture that’s free. The honest framing is that you’re buying blast-radius containment with a mix of duplicated fixed costs, reduced bin-packing efficiency, and extra operational surface — and the job of a good design is making sure you’re paying for isolation you actually need, not isolation you inherited from copying someone else’s cell count. The last post in this series covers the part that makes all of the above sustainable day to day: provisioning cells with infrastructure as code and shipping application changes into them with ArgoCD.
Splitting a system into cells doesn’t automatically make it resilient. If you still push a bad deployment to every cell at once, or if the same unlucky customer keeps landing on the same overloaded set of cells as everyone else, the isolation on your architecture diagram isn’t doing much for you in production. The pattern only pays off once you pair it with two more things: shuffle sharding, and real discipline about how changes roll out.
Resiliency in Cell Based Arch!
Shuffle sharding: spreading the overlap thin
Cells alone reduce blast radius by dividing customers into groups. Shuffle sharding goes a step further by giving each customer (or request) a near-unique combination of cells or nodes, so that any two customers only partially overlap, and the odds of two customers sharing the exact same full set of resources drop fast as your pool of cells grows. AWS describes this in more depth in Shuffle Sharding: Massive and Magical Fault Isolation, and the cell-based architecture FAQ is direct about the boundary: shuffle sharding works within a cell, but by definition a cell shouldn’t share state across cell lines, so don’t let shuffle sharding become an excuse to blur the boundary you just built.
The AZ-as-cell-boundary example from the Journey to Cloud-Native series makes the payoff concrete. With customers shuffle-sharded across three AZ-aligned EKS cells, a black-swan event that would have hit 100 percent of the application without cells is capped at roughly a third of capacity — and shuffle sharding also gives you a scaling lever for a single customer whose traffic spikes unexpectedly, since their load doesn’t have to be absorbed by one fixed cell.
Bake time and fractional deployment
The other half of real resilience is how you ship changes. A cell-based system gives you a natural unit for fractional deployment — roll a change to one cell, watch it, then the next, rather than everywhere at once. What actually makes this work is bake time: a deliberate pause after each incremental step, long enough to know whether the change caused trouble before promoting it further. Depending on the system, that might be fifteen minutes or several hours — the right duration is whatever it takes for you to trust the signal, not a fixed number copied from someone else’s runbook.
The discipline that goes with it is equally important: roll back at the first sign the release is destabilizing things, rather than pushing forward hoping it stabilizes on its own. A fractional failure contained to one cell is a good outcome. A full rollout that later needs a full rollback is the outcome cells were supposed to prevent. Stateful changes deserve extra caution here specifically because they’re harder to walk back — a schema migration that can’t be un-run is a one-way door, cell-based or not.
Release Flow
AWS’s own hyperscale teams lean on progressive delivery tooling for exactly this — Argo Rollouts alongside EKS, watching response time and error rate as the promotion signal, described in the same Journey to Cloud-Native post. We’ll come back to wiring this into ArgoCD directly in the last post of this series.
Monitoring: per-cell, and in aggregate
Splitting one system into many cells multiplies your monitoring surface by the same factor. Each cell needs its own health signal — white-box metrics from inside the application, black-box checks from outside it, and business metrics that catch problems the infrastructure metrics miss entirely, like a quiet drop in successful checkouts. On top of that, you need an aggregate view that rolls individual cells up into “how many cells are healthy right now,” because a dashboard with forty individual cell panels and no summary is not a dashboard anyone can act on during an incident.
Tag everything with a cell identifier from the start — logs, traces, and metrics alike — so that when something does go wrong, correlating the failure back to a specific cell is a query, not an investigation. On EKS, Karpenter’s topology spread support and AWS’s Zonal Shift capability are worth pairing with AZ-aligned cells specifically, since they let you push pods away from an impaired zone and cordon affected nodes without hand-rolling that logic yourselves.
Multi-cell transactions: avoid them if you can
Sooner or later someone will ask for a request or a report that needs data from more than one cell. Handle this as a deliberate, external coordination step — a map-reduce-style service that calls each cell through its normal public interface, never by reaching into a cell’s internals directly — and treat every one of these you add as a real complexity cost, not a free feature. Each cross-cell dependency is a small crack in the isolation boundary you built the rest of this architecture to protect. Some are unavoidable. Most are worth pushing back on.
Resilience, in a cell-based system, isn’t a property you get from the architecture diagram. It’s the sum of a routing layer that fails safely, a shuffle-sharding strategy that keeps blast radius genuinely small, and a deployment process that treats bake time as non-negotiable. Get the diagram right and skip the process, and you’ve built an expensive way to still take a full outage. Next in the series: what all of this actually costs, and how it maps onto the Well-Architected Framework beyond just reliability.
Every cell you design is, by intent, isolated from every other cell. The one piece of the system that can’t fully follow that rule is the network path that decides which cell a piece of traffic reaches in the first place. Get the routing layer wrong and you’ve rebuilt the single point of failure you were trying to eliminate — just one layer higher up.
Networking in Cell Based Arch!
VPC topology: per-cell, shared, or AZ-as-boundary
There’s no single correct way to lay out VPCs against cells, and the right answer depends on which boundary you’re actually defending. A VPC per cell gives the cleanest isolation — a NACL misconfiguration or route table mistake in one VPC can’t reach another — but it multiplies NAT gateways, VPC endpoints, and IP address planning by the number of cells you run. A shared VPC with dedicated subnets per cell is cheaper to run and easier on IP address management, at the cost of some shared blast radius at the VPC level itself.
A third option skips the VPC question and uses the Availability Zone as the cell boundary directly. AWS’s own Journey to Cloud-Native architecture write-up describes exactly this: pods aligned into cell groups on EKS, spread across three AZs, with only the routing layer — Route 53, DynamoDB, and the load balancer — shared across the whole system. The payoff is concrete: without cells, an initial bad event could take out the entire application; spread across three AZ-aligned cells, the worst case for that same initial hit caps at roughly a third of total capacity.
Whether you also want an AWS account as an isolation boundary is a separate, harder call. Multiple accounts contain the damage from leaked credentials or an account hitting a service quota, but account sprawl brings its own tax — every account needs to be onboarded into billing, monitoring, and security tooling, and that overhead doesn’t disappear just because Organizations makes account creation easy.
Route 53 weighted routing, worked through
For AWS-native cell routing, Route 53 weighted routing does a lot of the heavy lifting cheaply. Each cell (or infra group within a cell) gets a weighted alias record behind one stable DNS name, and adding capacity is just adding another weighted record — existing clients never need a DNS change. A record for a new cell might look like this:
Turning on health evaluation matters here — it’s what keeps Route 53 from sending traffic to a cell whose ALB has gone unhealthy. A hosted zone supports up to 10,000 weighted records, so this scales to a lot of cells before you need to think about it again.
The ALB quotas that quietly cap your design
Two unglamorous Application Load Balancer quotas end up shaping infra-group size before any architecture diagram does: a maximum of 100 target groups per load balancer, and a maximum of 5 target groups per listener rule. Work through the arithmetic and one ALB, configured with reasonably granular listener rules, comfortably serves on the order of 50 tenant or cell targets before you need a second ALB — which is exactly the trigger AWS Ads used to decide when to add a new infra group rather than scale the existing one vertically. Knowing this number before you design the routing layer saves a redesign later.
PrivateLink for shared dependencies — and its cost
Cells rarely live in complete isolation from the rest of your platform; they usually need to reach a handful of centrally-owned services — a shared cache, an identity provider, a billing system. AWS PrivateLink is the standard way to expose those without opening them to the public internet or wiring up VPC peering per cell. The trick that actually pays off is pre-wiring the interface endpoints once, at tier or platform creation time, rather than per tenant — AWS Ads reports an 80 percent cut in network configuration overhead from doing exactly that. Each interface endpoint runs a modest, predictable cost — roughly $7.30 a month plus data transfer — which is close to noise once it’s shared across dozens of cells, and a line item worth tracking once it’s multiplied per cell instead.
Where a Transit Gateway shows up to connect layers of cells to each other, treat it carefully. A single TGW sitting at the center of every cell-to-cell path becomes exactly the kind of shared point of failure the whole architecture was designed to avoid. If cross-layer traffic is limited to a small, known set of backend cells, bilateral connections are often the safer default over a hub-and-spoke TGW.
Where VPC Lattice fits
Amazon VPC Lattice is worth knowing about for the service-to-service leg of this problem specifically. It gives services a logical, IAM-authorized identity instead of an IP address, and it works across EC2, ECS, EKS, and Lambda targets in different accounts and VPCs without hand-built peering or Transit Gateway routes. That makes it a good fit for the case where a cell’s workloads need to reach a small set of centrally-owned services elsewhere in the organization — but it’s not a cell router. It solves reachability between services, not the “which cell does this customer belong to” assignment problem, which still needs the DynamoDB-and-DNS pattern from the routing layer.
Certificates: the boring single point of failure
One detail that’s easy to skip and expensive to skip: give each cell its own TLS certificate with its own expiration date rather than sharing one certificate estate-wide. A single shared certificate that expires or gets misconfigured is a shared blast radius, no different in kind from a shared VPC route table — it just doesn’t look like one until it fails.
Networking is the layer of a cell-based system you actually want to be a little boring — predictable quotas, pre-wired connectivity, health-checked routing, and nothing clever sitting in the critical path that doesn’t need to be there. Boring networking is what makes the resiliency claims in the next post actually hold up under a real failure, rather than just on the architecture diagram.
Once you’ve picked a partition key, the next decision is what actually runs inside a cell. For containerized workloads that’s an EKS-or-ECS question, and it’s worth more thought than “we already know Kubernetes.” The substrate decision determines how expensive isolation is per cell, and cost per cell is a number you’re about to multiply by every cell you’ll ever run.
When you have a choice to make!
EKS cells: namespace, or full cluster?
Inside EKS, a cell can live at two different depths. A namespace-per-cell model, with resource quotas and network policies enforcing the boundary, is cheap — one control plane serves every cell, and you’re mostly paying for the compute the pods consume. The catch is that the control plane itself becomes a shared blast radius domain again. A CRD conflict, an API server issue, or a cluster-wide add-on misbehaving can still touch every cell in that cluster, which undermines a chunk of the reason you wanted cells in the first place.
Cluster-per-cell removes that shared surface entirely, at a real price: an EKS control plane fee for every cluster, a separate node group or Karpenter deployment per cluster, and its own set of add-ons, IAM roles, and networking to keep patched. AWS’s own containerized cell-based reference design uses this model — pods are grouped into cell groups inside EKS, with topology-aware routing hints helping keep traffic within a cell’s Availability Zone rather than crossing between them. That Journey to Cloud-Native architecture post is worth reading end to end if you’re leaning EKS.
ECS cells: cluster-per-tenant is nearly free, until it isn’t
ECS clusters cost nothing to create — the meter starts when a task runs, which makes cluster-per-tenant a genuinely cheap way to get hard compute isolation without touching a Kubernetes control plane at all. It’s a well-worn SaaS isolation pattern for exactly that reason.
It’s also a pattern that can quietly become the problem. Amazon’s ad-serving platform ran an early cellular design that allocated a dedicated AWS account, Application Load Balancer, and ECS cluster to each tenant. Documented in AWS’s write-up on building hybrid multi-tenant architecture for stateful services, that setup hit real limits at a surprisingly small scale: just 18 clients spread across four AWS Regions already needed 181 separate targets to manage, onboarding a new client took roughly 52 days end to end, and average CPU utilization across the fleet sat around 3 percent. Strict isolation had bought accuracy, but the servers spent over 98 percent of their time waiting on requests that never came, because each tenant’s dedicated capacity had to be sized for its peak, not its average.
The fix wasn’t to abandon per-tenant isolation — it was to stop mapping tenants one-to-one onto cells. AWS Ads moved to a three-level hierarchy: a tier groups tenants with similar traffic profiles, a cell is the AWS account boundary within a tier, and an infra group — one VPC, one ALB, a set of per-tenant ECS clusters — is the reusable unit inside a cell. Onboarding became a configuration change against pre-wired infrastructure instead of a multi-week build, dropping to about 7 days. The lesson generalizes past ECS: your isolation boundary and your scaling unit don’t have to be the same thing.
The building blocks that show up in almost every cell design
Regardless of EKS or ECS, the same handful of AWS services tend to appear in the architecture diagram:
Purpose
Typical service
Container compute
Amazon EKS or Amazon ECS, often on Fargate or with Karpenter-managed EC2
Traffic routing to cells
Amazon Route 53 weighted or DNS-based routing, Application/Network Load Balancer
Cross-account, cross-compute service reachability
Amazon VPC Lattice, AWS PrivateLink
Cell-assignment state
Amazon DynamoDB — fast, highly available, a natural fit for “which cell does this customer belong to” lookups
Container images
Amazon ECR
Observability per cell
CloudWatch Container Insights, Amazon Managed Service for Prometheus, Amazon Managed Grafana
Infrastructure as code (Terraform or CDK) plus ArgoCD for the app layer
The DynamoDB piece deserves a callout. AWS’s own hyperscale teams have mapped customers to cells with a hash function whose result is stored in DynamoDB, precisely because that table needs to answer “which cell?” reliably under load without becoming a bottleneck itself. It’s a small, boring piece of infrastructure carrying an outsized amount of trust.
Router patterns, briefly
Every design needs something that decides which cell a given request goes to, and there are three broad ways to build it, each with a different failure profile. A load-balancer-style router sits in the request path permanently, forwarding every packet — simple for clients, but now in the critical path for every transaction. A hand-off router authenticates the client once and returns the cell’s address, after which the client talks to the cell directly; this removes the router from the ongoing request path but pushes some logic into the client. A DNS-based router leans entirely on the DNS system’s own reliability, at the cost of DNS-level control over routing decisions. None of these is strictly better — which one fits depends on how much control you need over routing versus how simple you want the client side to be. We’ll dig into this properly, including where a control-plane/data-plane split helps, in the networking post.
Which one would I reach for
If I’m starting fresh on containers and the workload doesn’t need OS-level tuning, I lean ECS cluster-per-cell for the early cells. It’s cheaper to stand up and tear down, and the limits you bump into are AWS service quotas rather than Kubernetes control-plane behavior you have to reason about separately. I’ll reach for EKS cluster-per-cell once the team already runs Kubernetes elsewhere and wants one operational model across cell and non-cell workloads — the consistency is worth the extra control-plane bill at that point. Namespace-per-cell inside a single EKS cluster is the one I’d be most cautious about recommending for anything claiming strong isolation; it’s cheap, but you’re still betting the whole cluster’s control plane won’t be the thing that takes every cell down at once.
The substrate decision sets your cost floor. The networking layer sets your failure floor — which is where we’re headed next.
A dropped table. A malformed feature flag. A load balancer configuration that fails open instead of closed. None of these need a regional outage to hurt you. They only need a blast radius big enough to reach every one of your customers at once. Multi-AZ and multi-Region deployments protect you from AWS having a bad day. They do nothing for the day your own deployment pipeline has one.
Where it begins!
That gap is what cell-based architecture is built to close, and it’s the reason AWS has used the pattern internally for over a decade before writing it down as public guidance. Before touching a Terraform file or an EKS console, the job is mostly a thinking exercise: deciding what a “cell” is for your system, what goes inside one, and how big is too big. Get that wrong and no amount of Kubernetes or containers expertise will save you later.
What a cell actually is
A cell-based architecture partitions a system into a set of independent, self-contained replicas, each serving only a slice of the overall client base. AWS’s guidance on reducing scope of impact with cell-based architecture frames this as an extension of the same fault isolation AWS already applies at the Availability Zone and Region level, just moved down into your own workload.
The two things that make a cell a cell, rather than just another replica behind a load balancer, are isolation and partitioned state. Every cell runs standalone, with no runtime dependency on any other cell, and the data it owns isn’t replicated elsewhere. A thin routing layer is the only thing that knows all the cells exist — clients (or their traffic) get assigned to exactly one cell and stay there. AWS’s cell-based architecture guidance on GitHub puts it plainly: if a bad actor or a bug wipes a database inside one cell holding a tenth of your users, you’ve lost a tenth of your data, not all of it, and you can restore a tenth of a database a lot faster than the whole thing.
This is also why it sits inside the Reliability pillar of the AWS Well-Architected Framework, specifically as an expanded form of the bulkhead architecture best practice. It’s explicitly called out as guidance for workloads that need extreme levels of resilience, not a default starting point for every workload you own.
Decision flow for Cell Based Arch!
Where cell boundaries come from
The first real decision — before compute, before networking — is the partition key. What is the unit of the system that gets isolated? Tenant ID, customer account, geography, and traffic tier are the usual candidates, and the choice cascades into everything downstream: how routing state is modeled, how the data layer gets sharded, and later, how your Terraform and GitOps repos get structured. Pick this wrong and you’ll be re-partitioning a live system later, which is exactly the kind of one-way-door change the whole pattern is meant to avoid.
The second decision is where the isolation boundary actually sits. An AWS Availability Zone can be the cell boundary, as AWS’s own architecture teams have done for containerized workloads. An AWS account can be the boundary, which limits blast radius from compromised credentials or account-level service quotas, at the cost of more accounts to govern and bill. Or the boundary can sit lower, at the Kubernetes namespace or ECS cluster level, inside a shared account. None of these is universally correct — it depends on what failure you’re actually trying to contain.
Sizing cells: the trade-off nobody gets to skip
Smaller cells reduce blast radius, since each one carries fewer customers. They’re also easier to test, easier to reason about, and individually simpler to operate. But more cells means more of everything else — more routing entries, more monitoring dashboards, more deployment pipelines running in parallel — and that operational surface area doesn’t shrink just because each cell is small.
Larger cells go the other way. Fewer moving parts, better economics per customer, less to monitor — and a bigger blast radius when something does go wrong inside one. There’s no formula that spits out the “right” cell size; it’s a genuine trade-off, and the common pattern is to start with a small number of large cells and shrink them over time as your automation and tooling mature enough to absorb the extra operational load. Jumping straight to hundreds of tiny cells before your deployment tooling can handle it just moves the failure mode from “blast radius” to “operational chaos.”
Is this even the right tool for your workload
Here’s the opinion part: most teams don’t need this. If you’re running a single EKS cluster for an internal tool with a handful of customers, cell-based architecture will cost you more in engineering time than any outage it prevents. It earns its place when a single failure hitting all customers simultaneously is genuinely unacceptable — hyperscale SaaS, regulated industries with strict blast-radius requirements, or platforms where a black-swan event (a sudden traffic spike from one tenant, a bad config push, a compromised credential) has to be contained by design, not by hoping your canary process catches it in time.
A planning checklist before you open the AWS console
Identify the partition key your cells will be organized around.
Decide the isolation boundary: namespace, cluster, Availability Zone, or AWS account.
Decide where cell-assignment state lives and who owns the routing layer.
Decide which resources are centralized (accepting shared blast radius) versus replicated per cell (accepting cost).
Set an initial target cell size, and the trigger condition for splitting or adding another one.
That last point matters more than it looks. Without an explicit trigger — a tenant count, a request-rate threshold, an AWS service quota you’re approaching — cell sizing decisions get made reactively, usually during an incident, which is the worst possible time to be making them.
The rest of this series works through the pieces that hang off these decisions: which AWS container services and building blocks actually implement a cell on EKS or ECS, how the networking layer routes traffic without becoming a single point of failure itself, how shuffle sharding and deployment discipline make the isolation real rather than theoretical, what this costs against the Well-Architected pillars, and how to provision and deploy dozens of these things without losing your mind. Next up: choosing between EKS and ECS as your cell’s compute substrate.
An EC2 instance can show 2/2 status checks passed while the application on it has been dead for twenty minutes. The hypervisor is fine, the kernel booted, the network stack answers pings — none of which tells you whether nginx stopped accepting connections or your app is stuck returning 500s to every request. That gap, between “the instance is up” and “the thing running on it actually works,” is what application status checks now cover natively, as of August 10.
EC2 Application Status Checks!
The Gap It Fills
EC2 has run two automatic checks on every instance for years: a system status check watching the AWS hardware underneath, and an instance status check watching the guest OS and its network config. An attached EBS check came later for volume reachability. All three are automatic — you can’t turn them off or point them anywhere — and none of them look inside the application. A crashed worker process or a web server that stopped listening is invisible to EC2, because the instance itself still looks perfectly healthy.
Closing that gap used to mean building it yourself: an ALB target group health check if you had a load balancer, a cron job and a custom CloudWatch metric if you didn’t. Application status checks are AWS’s native answer to the second case, a fourth, opt-in check that speaks HTTP directly to your application.
How It Works
You define a check with a protocol (HTTP or HTTPS), a port, a path, and a status code matcher — the same shape as an ALB health check. Associate it with instances by ID or by tag, including aws:autoscaling:groupName to cover an entire Auto Scaling group in one call, and EC2 pings that port and path every 60 seconds. Fixed interval, no faster, no slower.
Two consecutive failures flip the check to impaired, two successes flip it back to ok, both configurable, along with a 6-second timeout and a 300-second grace period after launch so a slow-starting app doesn’t get replaced before it’s ready. Set that grace period too low and Auto Scaling will happily replace instances that just needed more time to boot. Full parameter reference is in the Amazon EC2 User Guide.
Where the Checks Actually Run
Worth knowing before you roll this out: check traffic doesn’t come from some anonymous AWS endpoint. AWS provisions a managed network interface inside your own VPC, one per subnet-and-security-group combination, and traffic runs from within your VPC over AWS’s internal network, never the public internet. That managed ENI doesn’t count against your instance’s ENI limit, but it does count against your account’s ENIs-per-VPC quota, worth checking before a fleet-wide rollout across a lot of subnet-and-security-group combinations.
Check flow
The step people miss: CloudWatch gets a metric per check plus one aggregate per instance, and it’s only the aggregate that Auto Scaling watches. You can also specify the source subnet and security group yourself instead of letting AWS choose, useful if your network has segmentation rules a security review would care about.
App status check details
Auto Scaling, and the Deploy Trap
Auto Scaling groups already pull health signals from EC2, ELB, VPC Lattice, EBS, and custom checks. Application status checks slot in as one more. Associate the check with the group’s instances, and Auto Scaling replaces anything reporting impaired — no extra configuration needed. This earns its keep most on workloads that never sat behind a load balancer at all: backend workers, queue consumers, anything that never had a native AWS health signal before.
The catch: a deployment or restart makes the check fail too, since the app genuinely isn’t answering for a few seconds. Auto Scaling can’t tell “expected restart” from “actual crash,” so an unplanned deploy across a fleet can trigger a replacement storm you caused yourself. AWS gives you three ways to prevent that:
Approach
When to use it
Suppress the check
A known, bounded maintenance window
Exclude from aggregation
Validating a new check without risking replacements
Disassociate
Permanent removal
For routine deploys, automate suppression from your pipeline’s pre- and post-deploy hooks:
Put this in the deployment runbook before you need it, not after Auto Scaling has replaced half a fleet mid-release.
Two Gotchas Worth Knowing
When a check fails, look at the reason code before touching application logs. Redirects are a common false failure: health checks don’t follow 301s or 302s, so a /health path that redirects will fail unless you add that code to your matcher. Two other behaviors catch people off guard. The health check goes out over HTTP/2, so a minimal server that only speaks HTTP/1.1 in cleartext can fail a check that a plain curl would pass. And the HTTPS check never validates the server’s certificate, so a passing check tells you nothing about certificate validity.
What It Costs
Pricing: $0.01 per hour per managed ENI, per Availability Zone, plus standard CloudWatch pricing for the metrics. That cost tracks unique subnet-and-security-group combinations, not instance count. One Auto Scaling group in one subnet is one ENI per AZ, essentially free. A fleet spread across many subnets for segmentation reasons will rack up more ENIs than the instance count suggests, so it’s worth estimating that number before assuming this costs nothing. Default account quotas (50 checks, 5,000 targets) are generous for most teams and adjust automatically except for the targets limit, which needs a manual request.
Where This Fits
Application status checks aren’t a replacement for real observability. There’s no tracing, no latency data, just “did this path return the code I said meant healthy.” What they replace is whatever duct tape your team already built to answer that one question: a cron job and a custom metric, a sidecar pinging itself, a script from years ago nobody wants to touch. If that’s your current setup, especially for anything that doesn’t sit behind a load balancer, it’s worth an afternoon to pilot this with the check set to excluded before it can page anyone.
The credential leaks that actually make it into a postmortem tend to follow the same shape. A secret that was supposed to exist in exactly one place ends up existing in four: a .tfvars file someone forgot to delete, a CI log that echoed an environment variable during a failed-deploy debugging session, a state file with the value sitting in plaintext because Terraform needed it to create a resource, and the running container, which is the only place it was actually meant to be.
Secrets Management!
None of the tooling in this space is really about generating or storing a string securely. That part’s not hard. What’s hard is controlling how many places that string touches on its way from “created” to “used by a workload,” and closing off every one of those places except the last.
The map
A secret’s actual path through a modern deployment crosses four systems, and each one leaks it in its own particular way: the infrastructure-as-code tool that provisions the store, the CI/CD or GitOps engine that ships the workload, the store itself, and the runtime platform the workload runs on.
Secret Management Flow
Almost every incident I’ve dug into traces back to one of those four boundaries being noticeably weaker than the other three. Hardening one while ignoring the rest doesn’t close the leak, it just relocates it.
Infrastructure as code: the tool that provisions the vault, imperfectly
CloudFormation and Terraform dominate this layer for AWS-centric shops, and they fail in genuinely different ways.
CloudFormation’s answer is dynamic references: a string like {{resolve:secretsmanager:secret-id:SecretString:json-key}} that CloudFormation resolves at deploy time instead of you typing a password into a template. Pair that with the NoEcho attribute on the parameter and the value won’t print in the console or in describe-stacks. It’s a solid pattern with one sharp edge: NoEcho hides the value, it doesn’t encrypt it, and it offers no protection against someone who already has stack-read permissions. CloudFormation also won’t resolve a dynamic reference inside resource metadata like AWS::CloudFormation::Init, because that would print the secret straight to the console anyway. It’s a small but telling detail about where the real exposure risk actually sits. One advantage CloudFormation has by default: there’s no separate state file for you to secure, because the state lives inside the service.
Terraform’s version of this problem is structural rather than incidental. Mark a variable sensitive = true and Terraform redacts it from your terminal, your plan output, and your CI logs. It does nothing to the state file. Any sensitive value that becomes a resource attribute — a database password, an environment variable on a Lambda function — lands in terraform.tfstate in plain text regardless of how you flagged it, because Terraform needs the real value there to detect drift. Explaining that distinction to a team that assumed the sensitive flag was doing more than it actually does is a conversation most Terraform users eventually have.
For years the honest answer was: encrypt your state backend, restrict who can read it, and treat the state file itself as sensitive data. That’s still correct, but it’s no longer the whole story. Terraform 1.10 introduced ephemeral values, and 1.11 extended that to write-only arguments on managed resources, and together they’re an actual fix rather than a mitigation. An ephemeral resource fetches a value (a secret from Vault, a freshly generated password) and Terraform uses it during the apply without ever writing it to plan or state. A write-only argument, password_wo instead of password, accepts that ephemeral value directly on a resource like aws_db_instance, and the value is gone the moment the apply finishes. If you’re standing up new Terraform configurations this year, there’s not much reason to keep pulling secrets through a data source and hoping your state encryption is strong enough. The one thing worth planning for: migrating an existing attribute to its _wo form can trigger a resource replacement depending on the provider, so treat that migration like a credential rotation and test it somewhere that isn’t production.
CI/CD and GitOps are solving two different secret problems
People use “CI/CD” and “GitOps” almost interchangeably, but they hand secrets to infrastructure in opposite directions, and that difference matters more than which specific tool you’ve standardized on.
A CI/CD pipeline (GitHub Actions, Jenkins, Azure DevOps, or Spacelift running your Terraform) is push-based. Something happens, a runner spins up somewhere outside your infrastructure, that runner needs credentials to reach into your cloud account and your secret store, and then it pushes a change. The secret problem here is entirely about the runner: how does this ephemeral, often shared compute prove who it is without you handing it a long-lived key it might leak in a stray log line?
GitOps (Argo CD, Flux) is pull-based. An agent already lives inside the cluster and continuously reconciles actual state against what’s declared in Git. There’s no discrete pipeline-run moment to inject a secret from outside; the reconciler has to fetch it itself, from inside the cluster, and Git can never hold the plaintext value because Git is exactly where everyone with repo access can see it. That’s a different problem than “authenticate a runner,” which is why GitOps secret tooling looks nothing like CI/CD secret tooling.
On the CI/CD side, most of the industry has converged on the same answer: stop handing runners long-lived cloud credentials at all. GitHub Actions federates over OIDC: the workflow requests a short-lived token from GitHub’s own identity provider, and an IAM role’s trust policy accepts that token only from specific repositories, branches, or environments. No access key sits in GitHub Secrets waiting to be exfiltrated by a compromised dependency. Azure DevOps gets most of the way there with federated service connections, though the pattern I still see most often is a variable group linked to an Azure Key Vault: the group stores secret names rather than values and pulls the live value from the vault at queue time, so a vault outage fails the run early instead of mid-deploy. One thing that catches people off guard the first time: a variable group can’t link to a Key Vault that uses Azure RBAC for its permission model — the vault has to be running the older access-policy model instead. If secrets change often enough that a queue-time pull isn’t fresh, the AzureKeyVault@2 task reads the vault mid-run instead, at the cost of an extra step in every pipeline.
Jenkins is the odd one out because it predates all of this, and the right way to hand it secrets depends entirely on where the agents actually run. Agents on Kubernetes should use Vault’s Kubernetes auth method and let the pod’s own service account token do the authenticating. Agents on long-lived VMs typically use AppRole instead. The Vault plugin for Jenkins handles the retrieval and masks secrets in the build log, but “the plugin masks it” is a claim worth being a little skeptical of. An advisory against the plugin found that masking silently failed for secrets printed from shell steps on an agent when a specific durable-task logging mode was enabled; a workaround shipped in a related credentials plugin, but the underlying issue was still open at the time the advisory was published. The lesson generalizes past this one CVE: log masking is a convenience feature, not a security boundary, and you should assume that anything a build step can print, eventually it will.
Spacelift, which is closer to purpose-built CI/CD for Terraform than a general-purpose runner, handles this with contexts: reusable bundles of environment variables and mounted files attached to one or many stacks. Mark a variable or a mounted file as secret and it’s hidden from the UI and the API, visible only to the run itself. It composes nicely with Terraform’s own write-only arguments — Spacelift keeps the value out of its dashboard, Terraform keeps it out of state, and neither tool has to do the other’s job.
On the GitOps side, three patterns cover almost every cluster I’ve come across, and they trade off differently enough that picking one is a real decision rather than a coin flip.
Approach
Where the secret lives
Rotation
Best fit
Sealed Secrets
Encrypted in Git, decrypted only by an in-cluster controller
Manual, no external store to poll
Small teams with no existing secret store, comfortable with cluster-key lock-in
SOPS
Encrypted in Git via KMS or age
Manual re-encryption on change
Config that spans Kubernetes and non-Kubernetes systems
External Secrets Operator
Referenced in Git; the real value stays in Vault, Secrets Manager, or Key Vault
Automatic, on a sync interval
Teams that already run a real secret store and want rotation to just work
I’ve moved teams off Sealed Secrets more than once, and it’s rarely about the encryption being weak. It’s that the cluster’s private key becomes a single point of failure people forget about until they’re migrating clusters and discover nothing sealed two years ago can be unsealed anywhere else. External Secrets Operator avoids that by keeping Git as a set of pointers instead of a vault substitute, which is honestly the more defensible GitOps posture: Git should describe which secret a workload needs, not contain it in any form.
The stores themselves: what you’re actually paying for
AWS Secrets Manager, Azure Key Vault, and HashiCorp Vault all do the core job (encrypted storage, access control, an audit trail) well enough that the choice usually comes down to which cloud you’re already committed to, plus one real conceptual difference.
Secrets Manager and Key Vault are, underneath the branding, secured key-value stores with a rotation feature bolted on. That’s not a criticism. For a team running in one cloud, storing mostly static credentials and rotating them on a schedule, that’s exactly the right amount of tool, and you get IAM or Entra ID integration for free instead of standing up a separate identity story.
Vault’s dynamic secrets are the actual differentiator, and they change the security model rather than just the storage location. Instead of storing a long-lived database password and rotating it on a schedule, Vault’s database engine creates a brand-new database user with a short lease every time a workload asks for one, and revokes it the moment the lease expires or the workload shuts down. Most of the time, there is no standing credential to steal, because most of the time there isn’t a credential at all. That’s a genuinely stronger posture, but it comes with real operational weight: an engine to configure per database type, lease-renewal logic your applications need to handle, and a Vault cluster that now needs its own high availability, because your workloads can’t start without it. I’d only take that on if you’re actually going to use dynamic secrets for something: databases, cloud IAM, PKI certificates. If you’re going to end up storing mostly static values and checking a rotation box, you’ve built a more complicated Secrets Manager, not a better one.
Getting the secret into a running container
This is where the container platform’s own identity model decides how much of the pain above you actually feel.
ECS keeps it simple by splitting responsibility across two IAM roles that get conflated constantly the first few times someone sets this up: the execution role, which the ECS agent assumes to pull the image from ECR and to resolve any secrets block in the task definition (calling Secrets Manager or SSM Parameter Store before the container even starts), and the task role, which the application itself assumes at runtime for everything else. Put the secret permission on the task role instead of the execution role and the container just fails to start, with an error message that doesn’t always point at which role is missing what.
EKS and AKS have more moving parts because Kubernetes wasn’t originally built with cloud IAM in mind, so there’s a federation step before you even reach the secret. On EKS, that step used to mean IRSA exclusively (mapping a Kubernetes service account to an IAM role through the cluster’s OIDC provider), and IRSA still works and remains the practical choice for Fargate and Windows node groups. But EKS Pod Identity, generally available since late 2023, has become the default recommendation for new EC2-based clusters: no per-cluster OIDC provider to wire up, a universal trust policy instead of one keyed to each cluster’s issuer URL, and IAM session tags for namespace and service account applied automatically. Once that identity exists, the actual secret retrieval is usually External Secrets Operator syncing a value from Secrets Manager or Vault into a native Kubernetes Secret on an interval, or the Secrets Store CSI driver mounting it as a volume without ever materializing a Kubernetes Secret object. That’s a real difference if you’re trying to keep secrets out of etcd entirely. AKS follows the same shape with its own managed identity and CSI driver combination, including autorotation that polls Key Vault every two minutes by default and updates the mounted file and any synced Secret without a pod restart.
If you’re running EKS on mostly EC2 node groups and still wiring up IRSA for new workloads out of habit, that’s worth a second look. Pod Identity isn’t dramatically more secure by itself, but it removes an entire category of trust-policy maintenance that scales badly past one or two clusters.
Rotation: the step most projects only half-finish
Storing a secret securely and rotating it are different problems, and plenty of “secrets management” initiatives quietly solve only the first one.
AWS Secrets Manager’s rotation runs through a Lambda function that Secrets Manager calls in four steps on a schedule, and the mechanics explain why it can be close to zero-downtime instead of a coordinated cutover.
AWS Secret Rotation
Secrets Manager tracks versions with staging labels instead of overwriting anything in place. AWSCURRENT is whatever your application is using right now. A new version is created and labeled AWSPENDING while createSecret and setSecret do the actual work of generating a new credential and pushing it to the database or service, and only once testSecret confirms the new credential logs in does finishSecret flip the labels: AWSPENDING becomes the new AWSCURRENT, and the old current version becomes AWSPREVIOUS. Nothing reading the secret mid-rotation ever sees a half-updated value, because the label that matters doesn’t move until the new credential is proven to work.
Vault handles rotation by mostly avoiding the need for it. Dynamic secrets carry a short lease instead of a rotation schedule, so “rotation” for a database credential is really Vault refusing to hand out a lease longer than its TTL and issuing a fresh one on the next request. That’s arguably the cleaner model, but it only covers secrets Vault generates itself; anything stored as a static key-value pair rotates exactly as manually as it would anywhere else.
The IaC layer has its own rotation trap, and both major tools share it. CloudFormation’s dynamic references and Terraform’s data-source lookups both resolve a secret’s value once, at deploy time, and neither one watches the store for changes afterward. Pin a CloudFormation dynamic reference to a specific version-id and a background rotation in Secrets Manager will never touch your stack, not until you push a template change that touches the resource. That’s exactly why AWS’s own guidance is to use versionless references, so a stack update always picks up whatever AWSCURRENT happens to be. It’s a small detail that’s generated more than one confused ticket asking why a rotation “didn’t take.”
Hardening: shrinking the blast radius
Everything above assumes the access paths are already reasonably tight. Hardening is about making sure a compromised credential, a leaked log, or an overprivileged role can’t turn into anything worse than it already is.
Network path matters more than people give it credit for. Secrets Manager supports a VPC interface endpoint, and once that endpoint exists, a secret’s resource policy can add an aws:SourceVpce condition that denies any request not arriving through it, collapsing the attack surface from “anyone with the right IAM permissions from anywhere” down to “traffic that physically transited this one endpoint inside the VPC.” AWS also recommends BlockPublicPolicy: true on any identity allowed to attach resource policies, which uses AWS’s own automated reasoning to reject a resource policy granting broad or public access before it ever takes effect, rather than relying on someone catching it in review.
Identity is the other half of this. Every federation pattern described above, and every IRSA or Pod Identity association, exists for the same reason: a short-lived, narrowly scoped credential that expires on its own beats a long-lived one that has to be remembered, stored, and eventually rotated by a person. A workload or pipeline still authenticating with a static access key at this point is usually not a deliberate choice, it’s just the oldest unresolved thing in the account.
And then there’s the secret that never should have left a laptop. Gitleaks and TruffleHog cover this from two angles. Gitleaks runs fast, offline pattern matching that’s cheap enough as a pre-commit hook to block a leak before it’s even recorded in history. TruffleHog goes further and verifies whether a detected credential is still live with a real, read-only call against the provider it belongs to, which matters when triaging which of the dozens of things a full history scan just turned up are actual emergencies. Running both, one at commit time and one on a schedule against full history, catches more than either alone would. It’s worth taking seriously specifically because of how coding assistants have changed the failure mode: a GitGuardian report earlier this year found AI-assisted commits leaking secrets at roughly double the rate of human-typed ones, which tracks. Pasting a working credential into a prompt or a generated config file is a much easier way to leak one than typing it from memory ever was.
None of these controls substitute for each other. Scoping a secret to a VPC endpoint does nothing if the IAM role allowed to read it is wide open. Rotating on a schedule doesn’t help if the previous version is still sitting in three CI logs somewhere. The actual work is closing every path on the map from the start of this post, not just the one that happens to look most broken today.
If there’s one rule that generalizes
Count the number of places a secret’s plaintext value could theoretically be read by something other than the workload that needs it: a state file, a build log, a Kubernetes Secret sitting in etcd, a debug terraform output, a .env pasted into a chat message. Every tool in this post is, underneath its specific syntax, either reducing that count or increasing it. Write-only arguments reduce it. A Sealed Secret sitting in a public repo doesn’t touch it at all, which is the point. A sensitive = true flag that a team believes is doing more than it actually is increases it, quietly, until someone finds out the hard way. Pick tools by asking which direction they move that number, and most of the decisions in this post get easier to make on their own.
AWS App Mesh shuts down on September 30, 2026. Not “enters maintenance mode” — the console and the resources stop working. If you’re one of the teams that bet on it in 2019 for EKS service-to-service traffic, you’ve got a couple of months left, and AWS has been pointing everyone at the same replacement for ECS workloads and EKS workloads alike: Amazon VPC Lattice.
That deadline is a decent excuse to actually understand what Lattice is, because it’s not just an App Mesh clone. It quietly replaces a chunk of what Transit Gateway and PrivateLink do too, and it changes some assumptions network engineers have had baked in since VPC peering existed. Worth twenty minutes even if you have no App Mesh to migrate.
What VPC Lattice actually is
Strip away the marketing and VPC Lattice is a managed, regional application-layer proxy that sits between your consumers and your services, handles service discovery, applies IAM-based authorization, and routes requests — all without you deploying a load balancer, a sidecar, or a peering connection. AWS’s own description is that it manages network connectivity and application layer routing between services across different VPCs and AWS accounts, and that’s a fair summary, if a little dry.
The mental model that clicked for me: think of it as a load balancer that doesn’t live in any one VPC. You publish a service into a service network (a logical boundary, not a piece of infrastructure you provision), and any VPC or account associated with that service network can reach it by DNS name — no route tables, no CIDR planning, no peering mesh growing into an unmanageable tangle.
Lattice Service Network.
A service network can span accounts via AWS Resource Access Manager, so a platform team can own the network and share it out to application teams without those teams ever touching a route table. That’s the part that made this interesting to me as an architect rather than as a networking hobbyist — it moves network topology out of the app team’s problem space entirely.
What it’s quietly replacing
Nobody built this in a vacuum. Every one of these had a gap Lattice was built to close.
VPC peering doesn’t scale past a certain account count — it’s point-to-point, so N accounts means something close to N² connections, and overlapping CIDRs break it outright. Lattice’s data plane assigns consumers a link-local address and doesn’t care what your VPC CIDR looks like, overlapping or not.
Transit Gateway solved the peering-mesh problem at layer 3/4 — it’s still the right tool for routing IP traffic between VPCs and on-premises networks at scale, and I wouldn’t rip one out to replace it with Lattice. But TGW has no concept of an HTTP request, a path, or an identity. It moves packets, not requests. If your actual problem is “service A needs to call /v2/orders on service B and I want to enforce that with IAM,” TGW can’t help you get there on its own.
AWS PrivateLink is the closest sibling — it’s also a proxy-based, non-peering way to expose a service across accounts. The difference is PrivateLink is fundamentally point-to-point: each consumer VPC needs its own interface endpoint, and there’s no native request-level routing or IAM authorization baked into the data path. Lattice centralizes that into one service network that many consumers attach to, and layers on HTTP-aware routing rules on top.
AWS App Mesh is the one with the deadline. It gave you real service-mesh features — client-side retries, circuit breaking, fine control — but it did that with an Envoy sidecar next to every task, which is real operational weight: certificate rotation, sidecar upgrades, resource overhead per pod. Lattice deliberately drops the sidecar. You lose some of App Mesh’s client-side sophistication in the trade — Lattice’s routing and auth decisions happen server-side, not in a proxy next to your code — but for most teams that’s a fair trade for not operating a mesh control plane.
VPC Peering
Transit Gateway
PrivateLink
VPC Lattice
OSI layer
L3
L3/L4
L3/L4
L7 (app layer)
Cross-account model
1:1 mesh
Hub-and-spoke
1:1 endpoint per consumer
Many-to-many via service network
CIDR overlap tolerant
No
No
Yes
Yes
Identity-aware auth
No
No
No (SG/NACL only)
Yes (IAM/SigV4)
Request-level routing
No
No
No
Yes (path, method, header, weighted)
Sidecars required
No
No
No
No
How it’s actually built
Four concepts do the work. A service is the logical front door — listeners, rules, target groups — conceptually a load balancer, but one that isn’t tied to a single VPC. A listener is a protocol and port (HTTP, HTTPS, or TLS_PASSTHROUGH). A rule matches on path, HTTP method, or header and forwards to a target group, which can hold EC2 instances, IP addresses, Lambda functions, or an Application Load Balancer, and — as of the more recent releases — ECS and Fargate tasks natively alongside EC2 and EKS.
Worth calling out explicitly: Lattice terminates HTTPS itself using an ACM-managed certificate, which is convenient, but it only does server-side TLS. If you need mutual TLS, you have to use a TLS_PASSTHROUGH listener and let the target negotiate the client certificate — Lattice’s own data plane won’t do mTLS termination for you.
The newer half of the story is VPC Resources and Resource Gateways, which let you expose non-HTTP TCP endpoints — a database, an internal domain name, an on-premises system reachable over Direct Connect or VPN — through the same service network model, complete with the same IAM controls. This is the part that pushes Lattice past “App Mesh replacement” and into “how do I expose an RDS instance to another account without a peering connection or a bastion.” It’s genuinely useful, and it’s the feature I’d bring up first if a platform team asked me why this deserves attention beyond the App Mesh deadline.
For Kubernetes specifically, the AWS Gateway API Controller implements the open-source Kubernetes Gateway API and translates Gateway and HTTPRoute objects into Lattice service network objects behind the scenes. If your EKS team already writes Gateway API manifests, adopting Lattice barely changes their workflow — it’s the controller that’s doing the AWS-specific work.
Authorization runs on IAM. Auth policies attached at the service network or service level use standard IAM policy JSON, and clients authenticate with SigV4-signed requests, the same signing scheme every other AWS API call uses. This is a genuine strength if your org already centers identity around IAM roles, and a genuine adoption cost if you have legacy clients that have never had to sign a request in their life. Budget time for that conversation — it comes up in almost every migration writeup I’ve read.
When I’d actually reach for it
When Lattice makes sense.
Lattice earns its place when the actual requirement is cross-account or cross-VPC service exposure with identity-aware access control and some request-level routing, and you don’t want to own a control plane to get it. New microservice builds, platform teams centralizing how application teams expose services to each other, EKS shops trying to get off App Mesh before the clock runs out, and anyone who’s been maintaining a PrivateLink endpoint-per-consumer sprawl are the clearest fits.
I’d think twice before using it as a wholesale Transit Gateway replacement. It’s not built for raw IP-layer routing at TGW’s scale, and — as I’ll get to in pricing — the per-service hourly charge adds up fast if you’re running hundreds of services with modest traffic each. I’d also pause if you need genuine client-side mesh behavior: retries with backoff, circuit breakers, fault injection for chaos testing. That logic sits server-side in Lattice, which is simpler to operate but less flexible than a sidecar that runs next to your code.
Pros and cons, plainly
The upside is real: no sidecars to patch, no peering mesh to keep untangled, IAM auth you already understand, weighted routing for blue/green and canary out of the box, and a service directory that gives you an actual inventory of what’s exposed to what. Observability is baked in too — access logs and CloudWatch metrics per service without instrumenting anything yourself.
The downside is mostly about maturity and rigidity. It’s newer than TGW or PrivateLink, so you’ll hit rough edges — G2 reviewers flag protocol support currently limited to HTTP, HTTPS, and gRPC, and region coverage, while it’s grown a lot since GA, still isn’t everywhere. Security group design has a real gotcha: targets need to allow inbound traffic from the Lattice association security group, not from the consumer’s VPC CIDR, and that trips people up in early deployments because it looks wrong at first glance. And moving to IAM/SigV4 auth is a genuine client-side change — nothing you can skip past.
Pricing — the part that actually decides adoption
Three dimensions drive the bill: an hourly charge per service, a per-GB data processing charge, and a per-request (or per-connection, for TLS_PASSTHROUGH) charge above a free tier. In US East (N. Virginia), that’s $0.025 per service-hour, $0.025 per GB processed, and $0.10 per million requests beyond the first 300,000 free per hour. Compare that to PrivateLink at roughly $0.01 per hour per AZ plus $0.01/GB with volume discounts, and it’s clear Lattice costs more per unit — you’re paying for the extra L7 intelligence and the simpler operating model, not for cheaper bytes.
Run the numbers on a single, modestly busy service and it’s not scary: one service processing 100 GB and 200,000 requests an hour for a month lands around $18–20 in hourly charges alone before data and requests, and a heavier example AWS publishes — one service with HTTPS and TLS listeners together processing 2,100 GB and millions of requests a month — comes out to roughly $268 a month. The bill gets serious at fleet scale. One cloud architect’s published comparison modeled 200 services across 100 accounts pushing 30TB a month: Transit Gateway came out around $4,250, Lattice around $4,840 — roughly a 14% premium, driven almost entirely by the per-service hourly charge rather than data processing, which was actually cheaper for TGW in that scenario. If you’re running that many low-traffic services, the fixed hourly cost per service matters more than the per-GB rate.
VPC Resources (the database/on-prem exposure feature) bill separately and use a tiered per-GB rate: $0.01/GB for the first petabyte in a Region each month, dropping to $0.006 and then $0.004/GB at higher volumes, plus a small hourly charge per resource. Worth modeling separately from your service traffic if you’re planning to route a lot of data through it.
Quotas worth knowing before you design around them
These are the ones that actually shape an architecture, not just trivia:
Quota
Default
Adjustable
Services per Region
2,000
Yes
Service networks per Region
50
Yes
Service associations per service network
500
Yes
VPC associations per service network
500
Yes
Target groups per service
10
Yes
Targets per target group
1,000
Yes
Listeners per service
2
Yes
Rules per listener
10
Yes
Requests/sec per service per AZ
10,000
Contact your SA/TAM
Bandwidth per service per AZ
10 Gbps
Contact your SA/TAM
Connection idle timeout (HTTP/gRPC)
1 minute
Contact your SA/TAM
Max connection lifetime
10 minutes
Fixed
Service networks a VPC can associate with directly
1
— (use service-network VPC endpoints for more)
Full, current numbers are on the official quotas page — check it before you design, not after you hit a wall. The one that catches people off guard most is the one-service-network-per-VPC-via-direct-association limit; if you need a VPC in more than one service network, you’re routing through service-network VPC endpoints instead, which is an extra layer to plan for.
A few concrete use cases
A platform team centralizing how forty application teams expose internal APIs to each other, replacing forty sets of ad-hoc security-group rules and a slowly decaying peering mesh with one service network and a consistent IAM auth policy. A company running canary deployments where 5% of production traffic gets weighted onto a new service version before a full cutover — Lattice’s weighted target groups do this natively, no custom load balancer logic required. An EKS shop migrating off App Mesh before the September deadline, using the Gateway API Controller so app teams keep writing the same HTTPRoute manifests they already know. And the resource-gateway pattern: exposing a shared RDS instance in a data-platform account to a dozen consuming accounts without provisioning a PrivateLink endpoint in every one of them.
Why this belongs on your radar even without the App Mesh deadline
The App Mesh shutdown is the forcing function, but the more interesting shift is architectural: AWS is pulling network topology out of individual VPCs and into a service-oriented abstraction that’s owned centrally and consumed by reference. That’s a meaningful change to how you’d write a network architecture standard or review an ADR — the question stops being “how do these two VPCs route to each other” and starts being “which service network does this belong to, and what’s the IAM policy governing who can call it.” If you own architecture review or network standards for your org, that’s worth getting ahead of before it shows up in someone else’s design doc and you’re reviewing it cold.
It’s not a wholesale replacement for Transit Gateway, and it’s not free — budget the per-service hourly charge honestly before you commit a few hundred services to it. But for the specific problem it targets, cross-account and cross-VPC service exposure with identity-aware access control, it’s a cleaner answer than anything that came before it, and worth prototyping now rather than in month eleven of an App Mesh migration deadline.
