563 Things Karpenter Didn’t Know About

Gergely Daroczi

Attila Nagy

2026-10-30

563 Things
Karpenter Didn’t Know About

Gergely Daroczi & Attila Nagy, Spare Cores

Oct 30, 2026 @ Budapest, Hungary

Slides: sparecores.com/talks

Press Space or click the green arrow icons to navigate the slides ->

G: welcome … great pleasure to be back to KCD Budapest … last year lightning talk w/ 42 slides in 5 mins.. lucky you, we have Attila this year (PAUSE), who recommended that instead of presenting hundreds of slides and an unprocessable amount of information, we should take a different approach this time (and we will sing a song)!

A: that’s not what we have discussed …

G: then we will present a drama with a painful problem of a devops engineer, epic emotional journey and of course cathartic solution!

let’s get started!

curtain …

A couple weeks ago at an undisclosed location not too far, far away… Gergely and Attila met and among other topics they discussed how badly their Python app scales… Despite using Kubernetes…

G: The story starts with meeting Attila a few weeks ago, and this was the conversation we had:

G: Hey Attila, … how are you doing, you look a bit concerned..

A: As an engineer, concerned is my default state

A: frustrated Python app · needs low latency · won’t scale · GIL -> single-core performance · hunting right instance type · fed up with it -> memes [show image]

A: We call it GILty. Any ideas what to do with it?
G: Ah, interesting. I think what we do at Spare Cores might be relevant for you.
A: Sounds great, tell me more!
G: Sure.

Our main goal is to stop cloud waste, and while there are many ways to do this, such as:

• identifying and deleting unused resources or migrating e.g. to cheaper storage tiers etc
• automatically scaling down resources when they’re not needed

Our secret sauce – which is not that secret keeping in mind that we are a fully open-source project, so i’ll rather call it specialty is how we solve this problem: instead of throwing more of the very same silicon at the problem to solve scaling issues we find the right hardware for a given workload.

Note that this optimization is orthogonal to the previously mentioned solutions, so you can keep using reserved instances, spot etc. for the right SKU.

While everyone understands that cloud is still just another computer, at a remote location, that I can spin up quickly and throw away as needed, BUT we often don’t really know what kind of computer that is.

there are more than 6,000 cloud server options out there but do we really know all these options in the past you could just walked your server and check if it’s still comes with a turbo button

Before the cloud, you could simply walk to the server, and it was not too difficult to say that the computer running your program

• is a potato
• or a commodity PC with a case that has started turning yellow over the years
• or it’s a bleeding edge, high-end server
• or not so new and it still has a turbo button

You could even open up the case when the system administrator was not watching …

BUT nowadays

Like what on earth is even a virtual CPU?! Right?

There’s a conceptual uncertainty and massive visibility problem here, as a vCPU could range from an overprovisioned, poor-performing thread with saturated bandwidth to memory, versus a dedicated CPU core of a state-of-the-art server – with massive CPU cache and high memory bandwidth.

Which means that your applications could run much much faster!

Now, I’d like to host a quick guessing game if you don’t mind.

Any ideas how much faster is the best performing vCPU compared to the slowest? I don’t mean a comparing cloud servers with hundreds of cores with a potato, indeed just focusing on the single-core performance – like doing as many integer divisions in a second as possible.

Guess the Number!

How much better is the highest performing 2-vCPU cloud server's single-core performance compared to the lowest?

Your guesses

Both the slowest and the fastest servers are from a hyperscaler.

Now I’m not going to give away the identity of the actual server types, and shame cloud SKUs publicly … especially in the era of awakening AGI (Artificial General Intelligence).

But we do know both comes with 2 vCPUs.

SPEEDRUN!

[…]

How do I know all this stuff?

	SERVER A	SERVER B
Vendor	Hyperscaler	Hyperscaler
Server SKU	👻	👻
vCPUs	2	2
Architecture	x86_64	x86_64
Physical cores	1	2
CPU allocation	Shared	Dedicated
CPU (turbo) frequency	2.2 GHz	5.0 GHz
CPU L1D cache	32 KiB	96 KiB
Memory	1 GiB	8 GiB
Single-core SCore	133	4,645
Performance ratio	x34.92481

Well, we measured! Not only these 2 servers, but actually close to 4,000 (3,800)!

So we solve this visibility problem of the cloud by burning hundreds of thousands of dollars to start servers, measure performance, and calculate cost-efficiency, then publish these results and related tooling with open licenses, so that you can save money by optimizing your cloud spend.

DATA DRIVEN way

Credit: the_coding_love / When I launch a benchmark

30-50x

Before we move on, I wanted to quickly share a few further benchmarks of these two servers to show the spectrum:

logarithmic scale

SPEEDRUN!

10-20x

SPEEDRUN!

100x

now this is not single-core anymore .. we run OpenSSL speed tests ..

10-100x

Back to Vincent from Pulp Fiction .. who now looks even more confused.

Because it’s possible that you get a 32 vCPU server, and your app that perfectly scales to multiple cores, runs slower than on a single core of another cloud server type.

So how to pick the right instance type from thousands of options?

And there’s quite a spectrum even when just looking at the 2 vCPU options … Any guesses how many?

[…]

611 instance types with 2 vCPUs … we monitor 8 vendors (logos on prior slide).

Click the chart to zoom out and reveal the answer.

SELECT
  vendor_id,
  COUNT(*) AS servers
FROM server
WHERE vcpus = 2
GROUP BY 1;

vendor_id	servers
alicloud	192
aws	146
azure	174
gcp	41
hcloud	8
ovh	13
upcloud	18
vultr	25

$ apropos sparecores.com

• Open-source tools, database schemas and documentation to inspect and inventory cloud vendors and their offerings, including pricing and measured performance.
• Managed infrastructure, databases, APIs, SDKs, and web applications to make this data publicly accessible.
• Open-source helpers to select, start and manage instances in your own environment.
• Open-source Python/R packages and workflow orchestration extensions (e.g. Metaflow) to track resource usage and cost of DS/ML/AI jobs. Open-source tooling to right-size instances.
• Add-on services to scale data science workflows, even without direct vendor engagement.

• so Spare Cores is an open-source ecosystem, including software, database schemas, guides,
• and actual databases if you don’t want to run the ETL tooling yourself .. also providing APIs, SDKs etc to make it easier to query data
• unified CLI to start machines
• and working on an an optional SaaS offering built on the top of the open-source tooling for folks who would rather avoid registering with all cloud providers etc: give us a Docker image, a command to run, and you credit card .. all set, we will run it wherever it’s cheapest.

$ curl -I github.com/sparecores

FTR this dataset was generated, collected, standardized, and published in public GHA if ou are interested in the details. running around 100k jobs over 10k hours…

$ ls -1 sparecores

As mentioned previously, this is made available via multiple components that you can find on GitHub. We don’t have time to go through all of them, but I’d be happy to answer any related questions on Slack, or ping me on LinkedIn :)

WE will focus on the inspector now!

$ curl -I sparecores.com/navigator

Source: sparecores.com

high level numbers about the data we collect and make available

A: wait, do I need to write SQL to query this data?

$ curl sparecores.com/servers

G: The easiest way to query this data is through our web component, as you can see on the screen …
G: What kind of server are we looking for?
A: min 2 vCPU, 16 GB RAM, AWS, Oregon
G: Anything else?
A: Single core performance is the most important
G: We support a bunch of benchmarks, e.g. PassMark CPU Single Threaded Test?
G: Let’s order by performance then.
A: That metal server looks promising
G: Let’s see the details …

Check server details page: - server metadata - lstopo - benchmarks - price diffs -> oops .. spot vs ondemand

G: Actually, it’s $5 ondemand.
A: That’s too much. The devops team picked r5a.large .. can we find something better around the same price?
G: ADVISOR!! …

A: is there a way to query this data in a programmatic way?

$ curl sparecores.com/server/gcp/g2-standard-16

Clicking on a server shows you the technical details of the instance – much more than what’s provided publicly by the vendor, even more than what ChatGPT knows … as we actually start each machine and inspect the hardware.

$ curl sparecores.com/server/gcp/g2-standard-16

$ curl sparecores.com/server/gcp/g2-standard-16

Live and historical pricing

$ curl sparecores.com/server/gcp/g2-standard-16

and we also run benchmark scenarios on the servers, e.g.:

• memory bandwidth of read, write and mixed operations using various block sizes and also including the related L1/L2/L3 cache amounts
• or benchmarking compression algos - having the compression ratio on the x axis, and the bandwidth on the y axis, it’s clear that zpaq is a beast when it comes to compressing text, but might be slow on this machine
• OpenSSL hash functions and block ciphers

$ curl sparecores.com/server/gcp/g2-standard-16

also running test suites like PassMark or …

$ curl sparecores.com/server/gcp/g2-standard-16

Geekbench 6, which has been a standard tool for some time including workloads for text and image processing, compiling software etc

$ curl sparecores.com/server/gcp/g2-standard-16

visualizations on how well the machine can scale tasks to multiple CPU cores – e.g. showing the diminishing return on this Intel Xeon due to hyperthreading

$ curl sparecores.com/server/gcp/g2-standard-16

Or looking at LLM Inference Speed both for prompt processing and text generation, using smaller models with 135M parameters up to 70B larger models.

>>> from navigator.www import server

and other application-specific benchmarks, like serving a static website or running a key-value store database

$ curl sparecores.com/servers/compare

and making all these data available in comparison tables, e.g. you can see “Best performance servers for Redis below ¢50/hour” here

$ curl sparecores.com/servers/compare

or plots as well for human inspection

$ curl sparecores.com/advisor

r5a.large “cpu_single_threaded_test” AWS Oregon ondemand price

1. look at cheaper options
2. order by cost efficiency
3. exclude more expensive options

TODO Gergo check on aws r8i-flex – burstable?

G: compare baseline w/ r6a.large

…

A: Can we query this in a programmatic way?
G: yes!

$ curl docs.sparecores.com/api

G: for computers, we provide APIs … is it something you could use?
A: Definitely! Let me show you what I came up with!

A: Back to GILty — pure Python app

• latency + throughput → single-core (and cache) speed must be high
  • more CPUs -> more throughput, but no help for latency
• ran GILty on a few AWS instance types → measured speed
• SC DB: which benchmarks correlate with those results?
  • proxy: PassMark CPU Single Threaded — rank machines without re-running GILty everywhere

A: GIL — for folks who don’t know it yet

• the GIL has been with us since the early 90s — one thread at a time
• several attemps to remove GIL, like G(H)uido v(F)an Rossum + Larry Hastings (Gilectomy) without clear success for a long time
• there’s a new hope (free-threaded Python)
• but we still usually need a fast single core → project name: “Core Alone”

$ strace -e environment gilty.py

Before: AWS · EKS · Cluster Autoscaler + fixed node groups
→ capacity, not single-core / latency

Then: Karpenter → EKS Auto mode
→ dynamic EC2 pick under load — easy

But Karpenter AWS filters: family · cpu count · gen · …
plus instance-cpu-sustained-clock-speed-mhz — vendor clock, not a benchmark
→ how do we pick the fastest core for GILty?

A: what have we tried so far?

• before: CAS + fixed node groups — bad fit for GILty
  
• moved to Karpenter, then EKS Auto mode — flexible instance lists
  
• closest to “speed”: karpenter.k8s.aws/instance-cpu-sustained-clock-speed-mhz (EC2 SustainedClockSpeedInGhz → MHz)
  
• that’s catalog GHz, not measured single-thread · cache/IPC matter · GILty ≠ GHz ranking
  
• still missing: PassMark / Geekbench-style pick

PassMark ST vs catalog GHz

A: My question was: does clock speed predict single-thread performance?

• According to our measurements: there’s association between the clock speed and the PassMark ST score, and
• trend goes up — but same clock speed → wildly different scores
  
• sometimes a lower clock even beats a higher one
  
• so Karpenter’s MHz filter ≠ fastest core for GILty

Same GHz, how wide is PassMark?

Within one clock band: up to about +200% PassMark ST spread

A: same clock — how far apart are the scores?

• each bar: one GHz · gap between slowest and fastest
  
• some clocks: 2× difference in PassMark
  
• red = bad news · GHz alone won’t find the fast core

Just set the filter to 5 GHz?

Simple idea: instance-cpu-sustained-clock-speed-mhz >= 5000

	r5a.large	c8a.large	m8azn.large
catalog GHz	2.5	4.5	5.0
PassMark ST	~1400	~3800	~4300
on-demand	~$0.11	~$0.11	~$0.21
regions	~16	~7	~6

• only ~16 types · 2 families · ~1% of AWS catalog
  
• ~2× the money for ~+10% PassMark vs c8a
  
• tiny fallback pool → capacity / spot pain

A: should we just require 5 GHz, the current maximum?

• tiny menu — handful of SKUs, mostly m8azn / x8aedz
  
• few regions · new / niche
  
• vs c8a: 2× the price · only ~10% more PassMark
  
• Karpenter wants diversity — 5 GHz pin = brittle
  
• need benchmark ranking, not a GHz floor
  

$ claude –plan “fix problem”

Pub talk → need benchmark-ranked types in Karpenter (incl. EKS Auto)

Plan (sc-karpenter-nodepool ADRs + sequenced v1 plans):

1. Own NodePools only — never patch user pools / no NodeOverlay
   → managed Karpenter + GitOps (Argo CD, …) — no fights over a shared object
2. SCNodePool → one NodePool — template verbatim + selected instance-type list
3. Emit universal keys only (instance-type · arch · capacity)
   → avoid RequirementsDrifted loops
4. Selectors + ranking from Keeper (e.g. PassMark ST · family band)
5. Churn control — 24h refresh · hysteresis · never empty list
6. Go / kubebuilder — reuse Karpenter validation + provider test fakes

A: we sat down with Claude and planned the fix — six decisions:

• own NodePools — don’t touch user / Auto Mode pools · GitOps-safe
  
• one SCNodePool → one Karpenter NodePool (template + type list)
  
• NodePool gets only instance-type / arch / capacity — no AWS-only labels → avoid RequirementsDrifted loops
  
• Keeper ranks by benchmark (e.g. PassMark ST)
  
• churn control — ranking moves a bit → don’t replace running nodes
  
• Go — reuse Karpenter’s own validators
  

sc-karpenter-nodepool demo

TODO Attila speaker notes

$ strace -e environment sparecores.com

G: this is cool .. we could definitely use something like this at Spare Cores, as today we use GitHub Actions with Pulumi to pick and start VMs, keeping status and logs in GitHub repos, sometimes in S3 as well, and it became quite compliacated over the years to make sure we are decommisioning unwanted resources etc.

Instead of this, using k8s and a pod asking for an exact instance type at a given vendor would be so much simpler and more flexible.

I’ve checked Gardener, Container API … did not find anything plug and play.

G: how to use your kubernetes controller for this?

ENTRYPOINT [“sc-karpenter-nodepool”]

Great for picking types · not for inspector-style multi-cloud JIT

Limitations (vs “pod annotation → any cloud / region / AZ / SKU”):

• not a provisioner — only writes a Karpenter NodePool (instance-type list)
• one cloud · one cluster — regions from NodeClass / node labels, not per-pod
• no cross-cloud / cross-region hop in one request (AWS≠GCP≠Hetzner from one Pod)
• needs Karpenter + cloud provider already there (EKS / Auto · AKS NAP · GKE · …)
• no VPC / image import / Flatcar join / mesh — that’s not this controller
• Pod can pin a type (node.kubernetes.io/instance-type) — but only if it’s already in the pool
  · no “any vendor · any region · this SKU” from one annotation (≠ start.yml)

A: great for picking types — not for inspector JIT

There are still several limitations as you can see, but due to time constraints, let’s rather focus on the ideal scenario instead.

• we only write a NodePool — Karpenter still does the launch
  
• one cloud, one cluster — no per-pod region
  
• can’t hop AWS → GCP → Hetzner from one Pod
  
• Karpenter + cloud provider must already be there
  
• no VPC / image / Flatcar / mesh — out of scope
  
• Pod can pin a type if it’s already in the pool — not “any SKU anywhere” like start.yml

$ cat sc-karpenter-nodepool/ROADMAP.md

Ideal stack for sc-inspector (replace GH Actions + Pulumi start.yml):

• Control plane: on-prem / cheap VMs — k3s (3 servers) · not EKS/GKE/AKS
• Mesh: Headscale tailnet — workers need outbound only (any cloud/region)
• Orchestration: Hatchet — plan runs · create inspector Pods · retries / timeouts
• Provisioner: scarpenter — multi-vendor Karpenter provider + sc-keeper catalog
• Nodes: Flatcar + Ignition · TLS bootstrap join · single-tenant · die when empty
• Flow: Pod nodeSelector (vendor/type) → pending → cheapest offering → boot → inspect → tear down

A: ideal stack — replace start.yml with Pod-driven JIT

• k3s on cheap/on-prem VMs — not EKS/GKE/AKS
  

• Headscale mesh — workers only need outbound (any cloud, NAT OK)
  

• Hatchet runs the plans · Pods · retries (not GH Actions spaghetti)
  

• scarpenter — one multi-cloud Karpenter · catalog from Keeper
  

• Flatcar · TLS join · single-tenant (one bench at a time; more jobs → more nodes) · die when empty

• flow: Pod asks for a type → boot → inspect → tear down

> /review plan

Would you find something like this* useful?

* any k8s cluster + cloud credential + capacity need =>
ephemeral instances from any region of any cloud

TODO Gergo speaker notes

$ ping sparecores.com

Spare Cores

Websparecores.com

Docsdocs.sparecores.com

GitHubgithub.com/SpareCores

LinkedInlinkedin.com/company/SpareCores

Emailhi@sparecores.com

Attila Nagy

LinkedInlinkedin.com/in/11bra

Emailattila@sparecores.com

Gergely Daroczi

LinkedInlinkedin.com/in/daroczig

Emaildaroczig@sparecores.com