Tag Archives: karpenter

Cell-Based Architecture on AWS, Part 5: Well-Architected and the Real Cost of Cells

Cell-based architecture gets pitched almost entirely as a reliability pattern, and it is one — the AWS Well-Architected Framework files it directly under the Reliability pillar’s bulkhead guidance. But it touches all six pillars once you actually build it, and at least two of them — cost and sustainability — tend to get discovered the hard way, after the architecture is already in production and the bill has arrived.

Well architected and cost in Cell Based Arch!

Running it through all six pillars

Reliability is the pillar cells were designed for, and it’s the easy one: smaller, isolated fault domains, contained blast radius, and a natural unit for fractional deployment.

Operational excellence cuts both ways. Cells give you a clean unit for canary releases and bake time, which is a genuine operational win — but every cell you add is another entry in the on-call runbook, another set of dashboards, another thing that can silently drift out of configuration with its siblings if your automation isn’t keeping up.

Security benefits in the same shape reliability does: a leaked credential or a misconfigured IAM policy inside one cell doesn’t automatically hand over the rest of the estate. The cost is that you now have N sets of IAM roles, N certificates, and N network boundaries to keep patched and rotated instead of one.

Cost optimization is where the trade-off gets sharp, and it’s covered in more detail below.

Performance efficiency tends to improve with smaller cells, because a noisy neighbor problem shrinks to the size of one cell instead of the whole platform. AWS’s ad-serving case is a clean illustration in the other direction: when tenants shared in-memory state on common infrastructure, one tenant with a large dataset could trigger memory pressure that degraded everyone sharing that heap, which is precisely the failure mode dedicated per-tenant compute was brought in to solve.

Sustainability is the pillar cell-based architecture can quietly work against. Fixed-cost resources replicated per cell — a NAT gateway sized for peak, a node pool that never scales below some minimum — sit idle a lot of the time by design, since isolation means you can’t pack unrelated tenants’ spare capacity together. Idle reserved capacity is wasted energy whether or not the AWS bill makes that obvious.

The cost math that catches people off guard

A few numbers are worth having in your head before you commit to a cell count.

An EKS control plane costs roughly $0.10 an hour, which works out to around $73 a month — and that’s before a single pod runs. Ten cell-per-cluster EKS deployments means roughly $730 a month in control-plane fees alone, on top of whatever compute those clusters actually run. ECS doesn’t have this problem in the same way — a cluster is a free logical construct, and billing starts at the task — which is a real reason cluster-per-tenant is a more forgiving pattern on ECS than on EKS.

AWS PrivateLink interface endpoints run close to $7.30 a month each plus data transfer, which is trivial shared once across a tier of cells and a real line item if you accidentally provision one per cell instead of pre-wiring it once at the tier level — the difference between those two choices is exactly what drove an 80 percent cut in network configuration overhead in AWS’s own ad-serving redesign.

And the starkest number in that same case study isn’t a unit price at all — it’s utilization. Before the redesign, the isolated-per-tenant fleet ran at roughly 3 percent average CPU and 19 percent average memory, with servers idle more than 98 percent of the time. That’s the cost of strict isolation taken to its logical extreme: every tenant’s capacity has to be sized for that tenant’s peak, and peaks, by definition, don’t happen most of the time.

Karpenter and the bin-packing tension

If you’re on EKS, Karpenter earns its cost savings mainly through consolidation — packing workloads onto fewer, better-utilized nodes and leaning on Spot capacity where it’s safe to. Karpenter documentation and the practical write-ups from teams running it at scale are consistent on this: it works best with a large pool of mixed workloads to bin-pack across dozens of nodes with varied sizes, not a handful of strictly-separated node pools each sized for one tenant.

Cell isolation deliberately works against that. If every cell gets its own NodePool to preserve the isolation boundary, you’re intentionally giving up some of Karpenter’s consolidation opportunity in exchange for the guarantee that one cell’s workload can’t crowd out another’s. That’s a legitimate, conscious trade — not a mistake — but it’s worth naming explicitly rather than discovering it as an unexplained line item during a cost review.

What this means for how you size cells

Everything above points back to the same conclusion from earlier in this series: start with fewer, larger cells, and shrink them as your tooling matures. A few practical habits follow directly from the cost picture:

  • Bundle genuinely fixed-cost resources — a VPC, a NAT gateway — across a batch of cells where the blast radius trade-off allows it, rather than one per cell.
  • Tier tenants by traffic profile instead of mapping every tenant to its own cell 1:1; group similar-sized tenants together and reserve full dedicated cells for the ones that actually need them.
  • Revisit your cell count and size against the AWS Well-Architected Tool and the SaaS Lens periodically — this isn’t a decision you make once and forget, it’s one that should move as your tenant mix and automation both change.

There’s no version of cell-based architecture that’s free. The honest framing is that you’re buying blast-radius containment with a mix of duplicated fixed costs, reduced bin-packing efficiency, and extra operational surface — and the job of a good design is making sure you’re paying for isolation you actually need, not isolation you inherited from copying someone else’s cell count. The last post in this series covers the part that makes all of the above sustainable day to day: provisioning cells with infrastructure as code and shipping application changes into them with ArgoCD.

Cell-Based Architecture on AWS, Part 2: EKS or ECS for Your Cells, and the Services That Hold Them Together

Once you’ve picked a partition key, the next decision is what actually runs inside a cell. For containerized workloads that’s an EKS-or-ECS question, and it’s worth more thought than “we already know Kubernetes.” The substrate decision determines how expensive isolation is per cell, and cost per cell is a number you’re about to multiply by every cell you’ll ever run.

When you have a choice to make!

EKS cells: namespace, or full cluster?

Inside EKS, a cell can live at two different depths. A namespace-per-cell model, with resource quotas and network policies enforcing the boundary, is cheap — one control plane serves every cell, and you’re mostly paying for the compute the pods consume. The catch is that the control plane itself becomes a shared blast radius domain again. A CRD conflict, an API server issue, or a cluster-wide add-on misbehaving can still touch every cell in that cluster, which undermines a chunk of the reason you wanted cells in the first place.

Cluster-per-cell removes that shared surface entirely, at a real price: an EKS control plane fee for every cluster, a separate node group or Karpenter deployment per cluster, and its own set of add-ons, IAM roles, and networking to keep patched. AWS’s own containerized cell-based reference design uses this model — pods are grouped into cell groups inside EKS, with topology-aware routing hints helping keep traffic within a cell’s Availability Zone rather than crossing between them. That Journey to Cloud-Native architecture post is worth reading end to end if you’re leaning EKS.

ECS cells: cluster-per-tenant is nearly free, until it isn’t

ECS clusters cost nothing to create — the meter starts when a task runs, which makes cluster-per-tenant a genuinely cheap way to get hard compute isolation without touching a Kubernetes control plane at all. It’s a well-worn SaaS isolation pattern for exactly that reason.

It’s also a pattern that can quietly become the problem. Amazon’s ad-serving platform ran an early cellular design that allocated a dedicated AWS account, Application Load Balancer, and ECS cluster to each tenant. Documented in AWS’s write-up on building hybrid multi-tenant architecture for stateful services, that setup hit real limits at a surprisingly small scale: just 18 clients spread across four AWS Regions already needed 181 separate targets to manage, onboarding a new client took roughly 52 days end to end, and average CPU utilization across the fleet sat around 3 percent. Strict isolation had bought accuracy, but the servers spent over 98 percent of their time waiting on requests that never came, because each tenant’s dedicated capacity had to be sized for its peak, not its average.

The fix wasn’t to abandon per-tenant isolation — it was to stop mapping tenants one-to-one onto cells. AWS Ads moved to a three-level hierarchy: a tier groups tenants with similar traffic profiles, a cell is the AWS account boundary within a tier, and an infra group — one VPC, one ALB, a set of per-tenant ECS clusters — is the reusable unit inside a cell. Onboarding became a configuration change against pre-wired infrastructure instead of a multi-week build, dropping to about 7 days. The lesson generalizes past ECS: your isolation boundary and your scaling unit don’t have to be the same thing.

The building blocks that show up in almost every cell design

Regardless of EKS or ECS, the same handful of AWS services tend to appear in the architecture diagram:

PurposeTypical service
Container computeAmazon EKS or Amazon ECS, often on Fargate or with Karpenter-managed EC2
Traffic routing to cellsAmazon Route 53 weighted or DNS-based routing, Application/Network Load Balancer
Cross-account, cross-compute service reachabilityAmazon VPC Lattice, AWS PrivateLink
Cell-assignment stateAmazon DynamoDB — fast, highly available, a natural fit for “which cell does this customer belong to” lookups
Container imagesAmazon ECR
Observability per cellCloudWatch Container Insights, Amazon Managed Service for Prometheus, Amazon Managed Grafana
Cost visibility per cell/tenantKubecost against Amazon Managed Prometheus for multi-cluster cost monitoring
DeploymentInfrastructure as code (Terraform or CDK) plus ArgoCD for the app layer

The DynamoDB piece deserves a callout. AWS’s own hyperscale teams have mapped customers to cells with a hash function whose result is stored in DynamoDB, precisely because that table needs to answer “which cell?” reliably under load without becoming a bottleneck itself. It’s a small, boring piece of infrastructure carrying an outsized amount of trust.

Router patterns, briefly

Every design needs something that decides which cell a given request goes to, and there are three broad ways to build it, each with a different failure profile. A load-balancer-style router sits in the request path permanently, forwarding every packet — simple for clients, but now in the critical path for every transaction. A hand-off router authenticates the client once and returns the cell’s address, after which the client talks to the cell directly; this removes the router from the ongoing request path but pushes some logic into the client. A DNS-based router leans entirely on the DNS system’s own reliability, at the cost of DNS-level control over routing decisions. None of these is strictly better — which one fits depends on how much control you need over routing versus how simple you want the client side to be. We’ll dig into this properly, including where a control-plane/data-plane split helps, in the networking post.

Which one would I reach for

If I’m starting fresh on containers and the workload doesn’t need OS-level tuning, I lean ECS cluster-per-cell for the early cells. It’s cheaper to stand up and tear down, and the limits you bump into are AWS service quotas rather than Kubernetes control-plane behavior you have to reason about separately. I’ll reach for EKS cluster-per-cell once the team already runs Kubernetes elsewhere and wants one operational model across cell and non-cell workloads — the consistency is worth the extra control-plane bill at that point. Namespace-per-cell inside a single EKS cluster is the one I’d be most cautious about recommending for anything claiming strong isolation; it’s cheap, but you’re still betting the whole cluster’s control plane won’t be the thing that takes every cell down at once.

The substrate decision sets your cost floor. The networking layer sets your failure floor — which is where we’re headed next.