Tag Archives: Amazon EKS

EKS Finally Lets You Touch the Control Plane Knobs You’ve Been Faking with Workarounds

For years, running EKS meant accepting that the control plane was a black box you paid for but couldn’t tune. Want the scheduler to pack pods tighter to cut node count? Write your own scheduler plugin or run a second scheduler alongside the default one. Want HPA to react faster than the stock 15-second loop? You couldn’t — that number was baked into every managed cluster, full stop. Want shorter event retention so a CI-heavy namespace doesn’t bloat etcd? Not your call.

Control Plane configurable parameters

AWS announced in mid-August 2026 that this is no longer true. EKS now exposes a set of advanced configuration parameters across the scheduler, controller manager, and API server, set directly through CreateCluster and UpdateClusterConfig. No sidecar schedulers, no forked control plane, no support ticket asking AWS nicely. If you run a shared EKS platform serving multiple teams — which is exactly the kind of cluster where these defaults start to hurt — this is worth a proper look rather than a skim of the release notes.

What actually changed

Four parameters, across three components:

ComponentParameterRangeDefaultNeeds Provisioned Control Plane
kube-schedulernodeResourcesFit.scoringStrategyLeastAllocated, MostAllocatedLeastAllocatedNo
kube-controller-managerhorizontalPodAutoscalerSyncPeriod10s15s15sYes
kube-apiservereventTtl10m60m60mNo
kube-apiserverserviceNodePortRange10260327673000032767No

That’s a narrow slice of what upstream Kubernetes actually lets you configure, and I’d guess it stays narrow for a while — AWS is clearly starting with the parameters that are safe to expose without risking control plane stability, not opening the floodgates. Each one is set through a component-specific config block (kubeSchedulerConfig, kubeControllerManagerConfig, kubeApiServerConfig), works on Kubernetes 1.31+, and shows up in describe-cluster output along with every default you haven’t touched. Changes go through a rolling update of the control plane, so don’t expect them instant — budget a few minutes and poll with DescribeUpdate or aws eks wait cluster-active if you’re scripting it.

Scheduler: MostAllocated is the interesting one

This is the parameter I’d actually reach for first on a shared platform. By default, Kubernetes scores nodes with LeastAllocated, which spreads pods thin across your fleet so every node keeps headroom. That’s a reasonable default when you don’t know your workloads. It’s also exactly why clusters running steady-state services end up with fifteen half-empty nodes instead of ten well-used ones.

Switch to MostAllocated and the scheduler starts favoring nodes that are already carrying load, packing new pods onto them instead of spreading out. Combine that with Karpenter or Cluster Autoscaler consolidation and lightly used nodes actually get reclaimed over time — this is where the cost story lives, not in the scheduling decision itself.

You can also weight which resources drive scoring:

nodeResourcesFit:
  scoringStrategy:
    type: MostAllocated
    resources:
      - name: cpu
        weight: 1
      - name: nvidia.com/gpu
        weight: 100

Worth being precise here because the behavior is easy to misread. Weights are relative, not absolute — gpu: 100 next to cpu: 1 doesn’t mean CPU stops mattering, it means CPU only breaks ties once GPU availability is identical across candidates. The sharper gotcha is that omitting a resource from the list isn’t the same as giving it a low weight — leave memory out entirely and it’s excluded from scoring altogether, not just deprioritized. If you’re running mixed CPU and GPU node groups, that distinction is the difference between a config that behaves as intended and one that quietly ignores memory pressure.

Two things won’t change no matter which strategy you pick. First, the scheduler never moves pods that are already running — flipping the strategy only affects future placement, so if you’re trying to consolidate an already-packed cluster, you still need to evict or roll the workloads yourself. Second, filtering behavior is untouched; a pod that genuinely doesn’t fit on a node still won’t land there. And the honest trade-off with MostAllocated: you’re concentrating blast radius. Pack workloads onto fewer nodes and losing one of them, or an AZ event taking out a chunk of your fleet, now affects more pods per incident. On a cluster running twenty teams’ workloads, that’s not a hypothetical — I’d pilot this on a subset of node groups before flipping it cluster-wide.

HPA sync period: faster reaction, but only if you’ve earned it

Shortening horizontalPodAutoscalerSyncPeriod from the default 15 seconds down to 10 sounds like a free win — your workloads scale out sooner after a traffic spike. It isn’t free, and AWS gates it behind Provisioned Control Plane for a good reason: the HPA controller has to reconcile every HorizontalPodAutoscaler object in the cluster within that window, and shortening the window from 15s to 10s cuts the number of HPA objects your control plane can keep up with by roughly a third.

Here’s the part that’ll bite someone: EKS doesn’t validate the sync period against how many HPA objects you actually have. Set it to 10s on a cluster with more HPAs than that tier supports, and the update succeeds. Nothing errors. What happens instead is quieter and worse — some HPA objects stop getting reconciled on schedule, autoscaling starts responding slower than before you touched anything, and there’s no alarm or event telling you why. Before anyone on your team shortens this, run:

kubectl get hpa --all-namespaces --no-headers | wc -l

and check that number against your scaling tier’s supported count. If you inherit a cluster where someone already set this and autoscaling feels sluggish, that command is your first move, not a Datadog dashboard.

One more thing worth flagging to anyone managing the control plane scaling tier: once horizontalPodAutoscalerSyncPeriod is off default, you can’t move that cluster back from Provisioned to Standard mode. You have to reset the sync period to 15s first, then downgrade the tier. It’s a small detail, but it’s the kind of thing that turns into a surprised Slack message during a cost-optimization pass six months from now.

Event TTL and NodePort range: smaller blast radius, still worth reading the fine print

eventTtl is the simplest lever here — how long the API server holds onto Kubernetes events before deleting them, tunable from the default 60 minutes down to 10. On a cluster running CI/CD, batch, or AI workloads that generate thousands of events an hour, that’s real etcd pressure and real API server list latency you’re trimming. The catch: a shorter TTL only applies to events created after the change. Existing events keep whatever retention was active when they were written, so the storage benefit shows up gradually as the old events age out — not the moment UpdateClusterConfig returns. And once an event’s gone, it’s gone; if you lean on kubectl get events for postmortems, make sure something durable is already scraping events externally before you shorten this.

serviceNodePortRange is the one I’d reach for during a lift-and-shift rather than day-to-day tuning. Legacy apps that expect services on specific fixed ports outside Kubernetes’ default 30000–32767 window used to force a choice: rewrite the app, or bolt on a proxy. Now you can widen or shift the range (bounded to 1026032767, to stay clear of kubelet/kube-proxy health ports on the low end and the Linux ephemeral port range on the high end) and let the migrated app keep its original ports. Just remember it’s cluster-wide and non-retroactive — narrowing the range doesn’t kick out services already holding an out-of-range port, but recreating one of those services will fail to get that port back.

Where this actually earns a place in your platform

None of these four parameters change the availability or performance envelope of your control plane — AWS is explicit that clusters keep the same SLAs regardless. What they change is whether your platform team keeps working around upstream Kubernetes defaults with sidecar tooling, or just configures the thing directly and gets it recorded in CloudTrail like any other cluster change. On a shared EKS platform, the honest advice is: pilot the scheduler strategy on a low-risk node group first, treat the HPA sync period as something you validate with a headcount check rather than assume is safe, and use event TTL as a lever specifically for your noisiest CI or batch namespaces rather than a blanket cluster setting. Terraform and ACK support isn’t there yet — it’s Console, CLI, SDKs, CloudFormation, and CDK for now — so if your provisioning is Terraform-first, you’re looking at a temporary CLI-driven exception until that support lands.

The bigger signal here isn’t really any one parameter. It’s that AWS is starting to treat the EKS control plane as something you configure, not just something you consume. Worth watching what gets added to this list next.

Amazon CloudWatch Managed Prometheus Collectors: Retiring the Self-Managed OTel Collector

Ask anyone who has wired Prometheus metrics into CloudWatch what the real bottleneck was, and it was never Prometheus. It was the collector. You’d stand up an OpenTelemetry Collector as a Deployment or a DaemonSet, hand-tune its scrape config, guess at memory limits for cardinality you hadn’t measured yet, and then spend the next year treating it like one more piece of cluster infrastructure that needed patching, right-sizing, and on-call attention of its own.

AWS Managed Prometheus Collectors

On July 31, AWS announced managed Prometheus collectors for Amazon CloudWatch, a fully managed, agentless scraper for Amazon EKS, Amazon EC2, Amazon ECS, Amazon MSK, and Amazon OpenSearch Service. You supply a scrape configuration and a way to reach your resources, and CloudWatch takes it from there: provisioning the collector, scaling it, and keeping it running. What it scrapes arrives in OpenTelemetry format, queryable with PromQL right next to your AWS vended metrics.

That’s the announcement. The more useful question, if you’ve spent any time in AWS’s observability stack already, is what’s actually new here versus what’s just wearing a new badge.

The new part is the destination, not the collector

This isn’t new collector technology. AWS shipped this same agentless scraper (automatic target discovery, no in-cluster agent, multi-AZ, metrics that never leave your VPC) as the Amazon Managed Service for Prometheus collector back in November 2023, initially scoped to EKS and writing into an AMP workspace. That lineage is visible directly in the API today: you still create one of these with the CreateScraper operation, under the aws amp CLI namespace, not some new CloudWatch-specific command. What’s new in this announcement is the destination and the source list. Alongside an AMP workspace, the same managed collector can now write into a CloudWatch dataset, and the supported sources have grown from EKS-only to include EC2, ECS, MSK, and OpenSearch.

Here’s an EKS scraper creation, adapted from AWS’s setup docs:

aws amp create-scraper \
  --alias "eks-metrics-scraper" \
  --source eksConfiguration="{clusterArn='arn:aws:eks:us-west-2:111122223333:cluster/prod-cluster', \
    securityGroupIds=['sg-0123456789abcdef0'], \
    subnetIds=['subnet-0aaa111','subnet-0bbb222']}" \
  --scrape-configuration configurationBlob=$(cat eks-scrape-config.yaml | base64 -w 0) \
  --destination cloudWatchConfiguration="{datasetArn='arn:aws:cloudwatch:us-west-2:111122223333:dataset/default'}"

Swap that final --destination flag for an ampConfiguration with a workspace ARN and you’re back to the exact collector AWS shipped in 2023. If you’re already running AMP collectors for Grafana dashboards, that’s the useful takeaway: this is an incremental destination option on infrastructure you may already trust, not a new product to evaluate from a cold start.

It also builds on something CloudWatch shipped only six weeks earlier: native OTLP ingestion with PromQL querying, launched June 9. That release is what made CloudWatch a legitimate destination for Prometheus-shaped metrics at all, with per-GB pricing and curated Container Insights dashboards for EKS. Managed Prometheus collectors are the agentless front door to that same pipeline, extended past what Container Insights auto-instruments.

How a scrape actually reaches CloudWatch

The mechanics will feel familiar if you’ve used any managed AWS scraper before. You hand the collector a set of subnets, and it creates an Elastic Network Interface in each one. It scrapes your targets over those ENIs using OTLP, then delivers the result to your CloudWatch dataset through a VPC endpoint. Nothing crosses the public internet.

What isn’t obvious from the diagram is that target discovery differs meaningfully by source, and that’s what determines how much of an existing scrape config actually survives the move:

  • EKS uses Kubernetes service discovery against the cluster API, plus control-plane metrics (kube-scheduler and kube-controller-manager, exposed starting at Kubernetes 1.28).
  • ECS uses DNS-based discovery through AWS Cloud Map, so tasks register once and get picked up without a static IP list.
  • EC2 is static_configs against private IPs, no different from pointing a self-managed setup at a fixed target list.
  • MSK uses DNS-based discovery against the cluster’s own bootstrap DNS name, which resolves to every broker and survives broker replacement without reconfiguration.

MSK is worth a closer look because it needs a prerequisite step: enabling Open Monitoring on the cluster, which exposes a JMX Exporter on port 11001 and a Node Exporter on port 11002. A scrape config covering both looks like this:

global:
  scrape_interval: 60s
  external_labels:
    cluster_name: my-msk-cluster

scrape_configs:
  - job_name: 'msk-jmx'
    dns_sd_configs:
      - names: ['my-cluster.abc123.c4.kafka.us-west-2.amazonaws.com']
        type: A
        port: 11001
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

  - job_name: 'msk-node'
    dns_sd_configs:
      - names: ['my-cluster.abc123.c4.kafka.us-west-2.amazonaws.com']
        type: A
        port: 11002
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

That gets you broker-level Kafka metrics (topic throughput, consumer lag, under-replicated partitions) from the JMX exporter, and host-level CPU, memory, and disk metrics from Node Exporter, both queryable with PromQL once they land. Two things to check before you plan around this: MSK Serverless and MSK Express aren’t supported, and public access combined with KRaft metadata mode is excluded too.

The scrape config is Prometheus-compatible, not Prometheus

This is worth reading closely before you migrate anything, because “Prometheus-compatible YAML” undersells how much is actually missing. The supported configuration surface covers global settings, scrape_configs with static_configs and dns_sd_configs, and relabeling. What isn’t there matters more than what is:

  • Minimum scrape interval is 30 seconds. An SLO built on 10 or 15-second scrapes doesn’t fit here.
  • No file_sd_configs, and no Consul, Eureka, or other external service discovery. If your target list comes from anywhere other than Kubernetes, Cloud Map, a static list, or MSK’s broker DNS, this collector can’t reach it.
  • remote_write and remote_read don’t apply, because delivering to CloudWatch is the whole job. A fan-out to a second backend doesn’t carry over.
  • Configuration blobs cap out at 256 KB, base64-encoded.

None of this is really a flaw. It’s the standard trade-off of a managed service. AWS narrowed the surface to what it can operate reliably at scale, and the narrowing happens to remove exactly the parts of Prometheus configuration that are hardest to run safely inside someone else’s control plane. But “copy your existing prometheus.yml over” is rarely literally true. Diff your current scrape configs against this list before you commit to a cutover date, not after.

What ships automatically, and what doesn’t

EKS and MSK both get a curated dashboard the moment metrics start flowing — EKS OTel and MSK OTel in the CloudWatch console — plus attribute enrichment on every metric: account, region, an inferred unit, and for EKS specifically the cluster name and ARN. That enrichment is what makes PromQL filtering pleasant instead of an exercise in label archaeology. The EC2/ECS path doesn’t come with an automatic dashboard, which is a reasonable trade given how varied EC2/ECS metric shapes are in practice, but it’s worth knowing before cutover rather than after.

Operationally, a managed collector vends its own logs to CloudWatch Logs covering target discovery, scrape successes and failures, and configuration errors, and you manage the scraper itself with aws amp list-scrapers, describe-scraper, and delete-scraper. Deleting one tears down its ENIs, and that cleanup takes a few minutes, so don’t script an immediate re-create into the same subnet.

Cross-account guidance has shifted too. Instead of the role-chaining pattern AMP has historically used for cross-account scrapers, AWS’s current recommendation for both EKS and MSK is CloudWatch metric centralization: replicate the metrics to a monitoring account rather than granting the scraper cross-account reach in the first place.

What this actually costs

Pricing has two components that scale independently, and it’s worth keeping them separate. The collector itself is billed hourly, similar in shape to an ENI or a NAT gateway, though the specific per-hour rate isn’t broken out yet on AWS’s public pricing page as of this writing. What is confirmed is that everything the collector delivers rides on standard CloudWatch OpenTelemetry ingestion pricing: currently $0.50 per GB for general OTel metric publishing, a flat rate that bundles in fifteen months of storage with no separate per-metric or per-API charge. A typical data point with ten to fifteen attributes runs 300 to 600 bytes, for a sense of scale. Console PromQL queries (Query Studio, dashboards) are free; programmatic PromQL queries cost $0.01 per million samples scanned.

That per-GB model is a real shift in how you’d think about cost here. Classic CloudWatch metrics billed per unique metric-dimension combination, so cardinality was the thing to fear. Under OTel ingestion, cardinality by itself is free; what costs money is the serialized size of each data point, which grows with the number and length of the labels attached to it. Fifteen short labels on a metric barely move the needle regardless of series count. A handful of long, high-cardinality values move it fast: unbounded request IDs, verbose ARNs repeated on every point. If you’re porting scrape configs over from a self-managed setup, that’s the moment to prune labels you’ve been carrying out of habit rather than actual query need.

One more line item: scraping itself can rack up VPC data transfer charges between the collector and your targets, separate from CloudWatch ingestion. AWS’s own suggestion is to enable gzip on your /metrics endpoints, which cuts transfer cost without changing what actually gets ingested.

Where I’d actually reach for this

If your team is already CloudWatch-centric and has been putting off a real Prometheus setup mainly because nobody wanted to own collector infrastructure, this closes that gap cleanly. For EKS and MSK metrics specifically, it’s a solid default: alarms and dashboards in the same place as everything else, without standing up Grafana and an AMP workspace just to get there.

I’d still keep a self-managed collector, or the AMP-workspace version of this same scraper, in a handful of specific situations: sub-30-second scrape requirements, target discovery through Consul, Eureka, or file-based service discovery, or a remote_write fan-out to more than one backend because you’re mid-migration or running a deliberately multi-vendor setup on purpose. None of those are rare for a mature platform team, so this doesn’t replace the self-managed path outright. It’s a genuinely good default for the common case, which happens to be most of it.

Availability is broad but not universal — it’s live everywhere CloudWatch’s OTLP endpoint already exists, minus Asia Pacific (New Zealand) for now. Worth checking before you write the migration runbook, not after you’ve scheduled it.