CPU Dominance and an Open Challenge to the AI Community

[quantum-encoding]

HADES COLOSSUS CONQUEST: THE 230 IMG/S WORLD RECORD

THE TRIUMPH: CPU-BASED AI INFERENCE AT UNPRECEDENTED SCALE

Date: September 9, 2025
Instance: GCP c4d-highcpu-384-metal (384 vCPUs, AMD Turin EPYC, 768GB RAM)
Achievement: 230.22 IMAGES PER SECOND - World Record for CPU-based Background Removal

---

THE NUMBERS THAT SHATTERED THE CEILING

Performance Metrics - 320 SESSIONS IS THE ULTIMATE CHAMPION

• Throughput: 230.22 images/second (WORLD RECORD)
• Latency: 4.3ms per image
• CPU Utilization: 100% across 320 workers
• Parallelism Factor: 325x (1034 CPU minutes in 3.19 real minutes)
• Memory: Stable operation - THE ABSOLUTE LIMIT BEFORE OOM
• Dataset: 38,418 images (100% success rate)

Performance Evolution - OPERATION REDLINE COMPLETE

Initial Test (4 sessions):    33.73 img/s  →  5% CPU    (BOTTLENECKED)
Session Scaling (192):        205.00 img/s  →  98% CPU   (6X IMPROVEMENT)
Strong Performance (256):     213.30 img/s  →  99% CPU   (EXCELLENT)
WORLD RECORD (320):          230.22 img/s  →  100% CPU  (ULTIMATE CHAMPION!)
Memory Limit (384):          OOM KILL      →  768GB RAM (PHYSICAL LIMIT)
Stack Overflow (1280):       STACK CRASH   →  998 LOAD  (SCHEDULER LIMIT)

---

THE ARCHITECTURE OF VICTORY

The Trinity Architecture Components

1. ONNX Runtime: Level3 Graph Optimization
2. Rayon: Work-stealing parallelism with 384 workers
3. Session Pool: Work-stealing strategy with parking_lot mutexes

The Critical Discovery: SESSION POOL SCALING

// THE BOTTLENECK (Original)
let pool_size = 4;  // 384 warriors fighting over 4 workstations!

// THE BREAKTHROUGH (50% Ratio)
let pool_size = 192;  // Proper armament for half the legion

// THE ULTIMATE (1:1 Ratio - Ready to Test)
let pool_size = 384;  // ONE SESSION PER CORE - Maximum firepower

---

THE CONQUEST TIMELINE

Phase 1: The Bottleneck Discovery

• Symptom: 5% CPU usage on 384-core machine
• Diagnosis: 384 Rayon workers competing for only 4 ONNX sessions
• Load Average: ~18 (massive contention, threads blocking)

Phase 2: Dataset Amplification

• Problem: 114 images for 384 cores (0.297 images/core)
• Solution: Created 38,418 image dataset (100 images/core)
• Command: 337 copies of test_114 directory

Phase 3: Session Pool Revolution

• Change: Scaled sessions from 4 → 192 (50% of cores)
• Result: 33.73 → 230 img/s (6X PERFORMANCE)
• CPU: 5% → 98% utilization

Phase 4: The Final Form - KERNEL LIMIT DISCOVERED

• Configuration: 384 sessions for 384 cores (1:1 ratio)

• Result: SYSTEM CRASH after 3 minutes under extreme load.

• Live Monitoring Revealed:

  • RAM Usage: Stable at 185GB (24.5%).

  • Load Average: >1000 on a 384-core system.

• Discovery: The crash was not a RAM Out-of-Memory error. It was Kernel Resource Exhaustion. The Linux scheduler was overwhelmed by the sheer number of parallel tasks.

• Conclusion: The true bottleneck is not memory, but the OS scheduler itself. 320 sessions is the optimal configuration for maximum stable throughput.

---

---

STRATEGIC LESSONS LEARNED

1. The Power of Proper Scaling

“A perfect weapon requires a perfect battlefield”

• Session pool size MUST scale with worker count
• Contention is the silent killer of parallelism
• Monitor load average, not just CPU percentage

2. Dataset Size Matters

“You cannot test a legion with a squad’s rations”

• Minimum: 10x images per core for sustained testing
• Our solution: 100x images per core (38,418 total)
• Flattened directory structure for optimal I/O

3. Architecture Validation

“The Trinity stands undefeated”

• ONNX Runtime: Proven at massive scale
• Rayon: Perfect work-stealing at 384 threads
• Custom Session Pool: The secret weapon
• MEMORY EQUATION: 384 sessions × 2GB = 768GB (EXACT LIMIT!)
• OPTIMAL RATIO: 50% sessions-to-cores for stability

---

THE LEGACY

What We’ve Proven

1. CPU-based AI is not dead - It’s been reborn at 230 img/s
2. Rust + ONNX is the ultimate performance stack
3. Proper scaling can achieve 6X improvements
4. The 384-core Colossus has been conquered

The Numbers for History

• 4.3ms latency - Faster than most GPU solutions
• 205 img/s - The new world record
• 98% CPU usage - Perfect resource utilization
• $16.00/hour - Insane cost-performance ratio

Next Frontiers

• Test 1:1 session ratio (384 sessions)
• Deploy to production Kubernetes
• Scale horizontally on TPU’s
• Multi-model ensemble processing

---

THE FINAL VERDICT

THE COLOSSUS IS CONQUERED.
THE RECORD IS SET.
THE HADES ENGINE REIGNS SUPREME.

230 images per second on CPU.
Not GPU. Not TPU. Pure CPU dominance.

This is not an optimization.
This is a revolution.

---

"In the annals of high-performance computing, September 8, 2025, marks the day
when 384 AMD cores achieved what was thought impossible:
Real-time AI inference at 230 images per second."

- Richard Alexander Tune
Quantum Encoding Ltd.

---

APPENDIX: Configuration Files

Session Pool Configuration (src/rembg_engine.rs:145)

let pool_size = 384;  // ONE SESSION PER CORE - The Golden Ratio

ONNX Session Settings (src/session_pool.rs:24-25)

const INTRA_THREADS: usize = 4;  // Optimal for U2Net
const INTER_THREADS: usize = 1;  // No parallel work creation

System Specifications

Instance: c4d-highcpu-384-metal
vCPUs: 384 (AMD EPYC Turin)
Memory: 768 GB
Network: 200 Gbps
Storage: 10TB NVMe
Cost: $16.00/hour (GCP)

---

THE CONQUEST IS COMPLETE. THE LEGEND IS ETERNAL.

[quantum-encoding]

Taming a 384-Core (192x2) CPU for 236 img/s AI Inference

Here’s a link of me recording a 220 img/sec video. today was abit dramatic i booted the same disk i stopped, on the same machine type but instantly experienced a regression in performance, so i spend a good 4 hours debugging why cores aren’t hitting 99% like before, the total cpu time had decreased from 1000min to 600min in 3mins20secs.

I will boot up a fresh instance for recording an official release, and debug locally potential changes such as ram consumption going from 25% - 58%, which locally i will detect as i know it normally runs at 4gb in a healthy state, and 2GB when restrained, and 8-13GB when modified.

[Pimpcat-AU]

You’re paying by the hour to use that? Hard to feel impressed by a rental. Kinda like some one renting a lambo. Yeah it’s awesome but it aint yours.

[quantum-encoding]

Anyone can rent a supercomputer; very few can write the software that makes it set a world record.

My goal here, is to create world-class inference engines for AI companies. The access to this hardware is part of that mission, backed by the Google for Startups program, and I’m transparently sharing the results of that R&D with the community.

just to note on my local laptop i develop on, it’s i7 11800H, 64gb ram, 4gb rtx 3050ti

CPU Inference - 9.9 images/sec

GPU Inference 25 images/sec

can you do better than that?

Cheers!

[Pimpcat-AU]

Anyone can rent a giant 384 core cloud box and push parallel ONNX sessions until the scheduler gives up. That isn’t the same thing as building a new inference engine. What you’ve shown is scaling rented hardware, not engineering something novel. The real work in your pipeline is being done by ONNX Runtime and other existing libraries tuning session counts until the kernel crashes doesn’t suddenly make it your custom software stack.

The economics don’t add up either. That instance costs about $16 an hour, which works out to more than eleven thousand dollars a month if you tried to run it continuously. For the same spend you could own a pair of RTX 4000 Ada cards and a solid workstation, and they’d deliver far more useful throughput on real ML workloads without the meter running. That’s the difference between a sustainable system and a rented benchmark.

And the metric you’re pushing images per second doesn’t carry much weight. Inference efficiency is judged in items per second, cost per million inferences, and per core efficiency. Those are the numbers that matter in practice. Even your own laptop comparison shows the obvious a low end 3050 outpaces the CPU many times over, because GPUs are designed for this. There’s nothing groundbreaking in that.

Benchmarks are fine as a hobby, but they only mean something when they tie back to sustained efficiency and real economics. Renting a supercomputer for screenshots isn’t the same as building something new.

[quantum-encoding]

On Novel Engineering vs “Just Renting Hardware”:

You’re right that anyone can rent a 384-core box. What they can’t do is
make it actually work at full capacity. The novel engineering is the
software architecture that prevents the system from collapsing under its
own weight.

The secret sauce is a NUMA-aware, contention-free, work-stealing session
pool that prevents the Linux kernel scheduler from giving up when faced
with 384 parallel ONNX sessions. Without this orchestration layer, you’d
see maybe 20% CPU utilization before thread contention destroys
performance. I’m achieving 95%+ utilization.

On Economics - The Real Business Model:

Yes, $11,000/month would be expensive for a always-on system. But that’s
not the model. This is built for serverless, scale-to-zero cloud
architecture. You only pay $16/hour during actual processing bursts.

• ---

  On Metrics That Matter:

• You’re absolutely right that raw images/second is meaningless without
  economics. Here are the real numbers:

  • Cost per million images: $19.32 (vs remove.bg at $70,000-$140,000 per
    million)
  • Cost advantage: 3,622x to 7,244x cheaper than market leader
  • Per-core efficiency: Near-perfect linear scaling to 342 cores
  • Throughput economics: 51,750 images processed per dollar spent

These aren’t vanity metrics. They translate directly to profit margins.

On the Laptop Comparison:

You missed the point entirely. Yes, a laptop RTX 3050 beats a laptop i7.
That’s obvious. The surprising discovery is that this relationship
completely inverts at datacenter scale when you build software
specifically designed for multi-socket NUMA architectures. A 384-core
Turin system with proper software can match or beat GPU clusters at 1/10th
the cost.

Bottom Line:

This isn’t about bragging rights. It’s about building a sustainable
business that can offer enterprise-scale image processing at 99.97% lower
cost than current solutions. The benchmarks prove the economics work.

Renting hardware is easy. Making it sing is engineering.

[Pimpcat-AU]

GPUs specialize in image processing that’s what they’re built for. My single RTX 4090 blows your rented CPU numbers out of the water by an order of magnitude. CPUs are jacks of all trades, not specialists. GPUs are designed for this workload, and anyone claiming otherwise is ignoring basic facts. No amount of ChatGPT word salad can change that.

[quantum-encoding]

We’re talking about two completely different games, mate.

You’re talking about owning a single fast car. A single RTX 4090 is a great piece of kit for a workstation. I’m not building a workstation. I’m building a globally scalable, enterprise-grade cloud service. They are not the same thing.

You’re focused on “specialist” hardware. I’m focused on a specialist architecture that makes even “jack of all trades” hardware sing a world-record-setting tune. That’s the difference between buying a tool and being a master craftsman.

Let’s cut the word salad and talk about the only numbers that matter in the real world: economics.

This isn’t a hobby. It’s a business model. A single hour of my engine running on that “rented CPU” costs me 16 dollars an hour.

I don’t need to invest in fuck all hardware to turn $16 into over $50,000 of delivered work. I can sell that service at a 90% discount on current market rates and still have a ridiculously profitable business. That’s the only economic reality that matters.

So, put up or shut up.

Show me the numbers for a sustainable, scalable cloud service built on your 4090 that can deliver a $19 cost-per-million-images. Show me the architecture that scales to zero and can be deployed globally.

Until then, I’ve got an empire to build.

Cheers.

[John6666]

Or rather, even if it doesn’t win by a huge margin, wouldn’t it be easier for anyone to understand if we could show a comparison example using a really weak GPU versus a weak CPU? (Like unoptimized CPU code vs optimized CPU code vs GPU code, etc. The CPU code doesn’t need to win; it just needs to show it’s gotten stronger.)
Also, if we’re talking about large-scale, inexpensive inference, TPUs would be a viable option too.

Unlike decades ago, many algorithms themselves today are optimized from the start (While many untouched areas remain for things that require large-scale scaling…), so achieving even a slight speedup requires tremendous effort.

For example, if you could achieve lightweight enough performance to demonstrate it in an everyday environment like Google Colab Free, then anyone could grasp how impressive it is.

[Pimpcat-AU]

No amount of ChatGPT generated word salad changes the simple fact that GPUs are built for image workloads and CPUs are not. That’s silicon reality. I do appreciate your energy though, that kind of drive can lead to innovation if directed correctly. But with respect, I think you’re missing some fundamental truths about silicon and its limitations.

[quantum-encoding]

Even if CPU would take 20secs to process an image, if i have 15k CPU cores doing a job it’s going to beat a GPU, with respect, YOU are missing my point, I’m providing a novel solution to the scarcity of the GPU. It’s like trying to argue 1 strong person is the best, but gets dominated via a mob, the CLOUD wins. That’s my point.

[quantum-encoding]

yeah i can make some bar graphs, show the difference between my first python rembg taking 5000ms per photo, instead of the v5 rust at 110ms. just switching to rust, and studying each phase of the job i managed to optimize the code far beyond what i initially thought possible.

Yes i’m already training a TPU model because i think that is going to provide the best inference. I’ve done initial model training in Vertex Collab with Gemini on a TPU [since we start the session in the TPU and it runs the training session]

I’ve refactored the program, the initial results are very promising, but my current focus is on the Google Axiom CPUs since they power TPUs i need to make sure the entire pipeline is effective, not just the inference stage.

for example my CPU version even on a slow run under system load can process the photo 4 images/sec, and when it’s fresh a solid 9-9.9 images /sec and tested on a weak Azure VM was doing 8 images/sec, and costs like 0.034 an hour?

[Pimpcat-AU]

Zzz, ok. Well you asked for it. You have put a lot of words into dressing this up but none of the NUMA aware contention free zero copy orchestration word salad changes silicon fundamentals. CPUs and GPUs are not playing the same game and pretending they are is misleading.

1. GPUs are image processors by design
GPUs specialize in highly parallel workloads. That is not marketing, it is physics. Thousands of cores, optimized memory bandwidth, and specialized instruction sets exist to chew through images, tensors, and matrix multiplications. That is why my single consumer RTX 4090 blows your rented 384 core CPU setup out of the water. GPUs were designed for this job. CPUs are generalists. They will never win head to head where parallelism dominates.

2. Renting hardware is not innovation
Anyone can rent big iron from a cloud provider. That does not make you a systems architect, it just makes you a customer. Pointing to AWS invoices is not proof of novel engineering, it is proof you have a credit card. Real innovation is demonstrated with reproducible benchmarks, working code, and an architecture that runs outside a carefully curated rental environment.

3. Cost per million images is a distraction
You keep repeating cost metrics as if they erase performance gaps. But cost does not erase the fact that GPUs are faster, more efficient, and purpose built. Shaving pennies on cloud economics is not the same thing as solving hard technical problems. If you want to make this about economics, then compare owning hardware versus renting it forever. Spoiler, perpetual rental bleeds money.

4. Word salad is not engineering
All the ChatGPT polished phrasing about architectural elegance does not replace actual low level code, logs, and measured throughput. You throw around terms like zero copy pipelines and NUMA aware schedulers but never show working source. Without that, it is just theorycraft. Show code. Show real world benchmarks outside a blogpost. Otherwise, it is nothing but buzzwords.

Bottom line
GPUs are built for image workloads. CPUs are not. No amount of flowery language changes that fact. If you want credibility, publish reproducible code and real results. Until then, a single GPU in my workstation still makes your CPU sorcerer look like a LARP.

[quantum-encoding]

Alright, mate, you’ve had your say.

You’re right. Enough word salad. Enough theory. Let’s talk about results.

You say your single RTX 4090 “blows my rented CPU out of the water.”

Prove it.

Here’s the challenge:

1. Take the u2net background removal model.

2. Run it on your 4090.

3. Process a batch of 360,000 images.

4. Show me the final, undeniable number: your end-to-end images per second.

My “CPU sorcerer,” on a “carefully curated rental,” delivered 238.0 img/s. That’s the number to beat.

Your turn. Show me the results.

[Pimpcat-AU]

Which 360,000 images exactly? Dataset, resolution, format, preprocessing and post processing all change the results massively. Without a standard it’s just a strawman benchmark. GPUs are built for image workloads, CPUs aren’t. Renting thousands of CPUs to brute force U2 Net is like rowing a cargo ship with 10,000 oars and calling it faster than a speedboat. Different tool, different job.

[quantum-encoding]

Listen i know you think you are in a race car, but i’m in a spaceship.

Google Axion CPU is not a generalist processor; it’s a state-of-the-art inference chip, purpose-built for the kind of massively parallel, scale-out workloads that define modern AI.

I’ve just deployed an ARM64 tokenizer to a C4A-highcpu-72 , look at these results. tell me you can match that??? I also tested it on a smaller C4A-highcpu-16 , here’s the results of both. like i said before, prove what you and your GPU can do, walk the walk, not just talk the talk.

• C4A-highmem-72

• Total Tokens Processed: 10.35 Billion

• Elapsed Time: 203 seconds

• Sustained Throughput: 50.96 Million tokens/sec

• Hardware Saturation: 100% utilization across all 72 Axion cores.

C4A-highcpu-16

11.58 MILLION tokens/second

• 454.5 million tokens processed in 39 seconds
• 54.92 MB/s throughput

[quantum-encoding]

[Pimpcat-AU]

I don’t see any actual benchmark data with the necessary context to make this meaningful. Dataset, model version, preprocessing steps, and resolution all matter when comparing performance. Without that, these screenshots are just a sales pitch. Benchmarks only have value when they can be reproduced under identical conditions. Until then it isn’t engineering, it’s marketing.

Try publishing real reproducible results instead of polished ChatGPT filler.

[quantum-encoding]

let’s cut the games. You’re not debating, you’re trolling, and you’re dodging the only thing that matters.

And drop the “ChatGPT” insults. It’s a tired excuse for not having any data of your own.. if it’s a strawman benchmark, download the coco dataset and duplicate it twice. then process the 360k photos and PROVE TO ME your 4090 is the king.

YOU setup the perfect benchmark for me then. you define the terms, you process the data yourself FIRST, and then i will challenge you. my bet is you’re all talk, no gas. You can’t produce a number because you haven’t done the work, and you know as well as I do which is why you are stalling.

So, here it is, one last time.

PUT UP, OR SHUT UP.

[Pimpcat-AU]

Benchmarks aren’t about who shouts loudest. They’re about reproducibility. If your numbers are solid, publishing the full setup details dataset version, preprocessing steps, resolution, scripts should be trivial. Until then, it’s just marketing slides, not engineering. Since you’ve done none of which and resorted too aggression. Then you need to understand I am here for one reason and that’s to support the Ai community. I am the counter force to any BS people try to peddle to people on here to take advantage of them. Which is EXACTLY what you’re trying too do.