6
System Design
Vertical vs horizontal scaling for a CPU-bound order service
Simplicity ↔ Scalability
Context
Handle 4× peak traffic on an order service that is CPU-bound today.
Our order service runs on two c6i.2xlarge nodes and hits 80% CPU during the evening peak, which is about 4× the daily average.
Option A: move to c6i.8xlarge. One config change, no code changes.
Option B: go horizontal, more smaller nodes behind the load balancer with autoscaling. But each node keeps a session cache in memory today, so we'd have to move it out first.
Is scaling up a sensible bridge, or are we just postponing the inevitable?
Constraints
- Peak vs average
- 4× for ~2 hours a day
- Current
- 2 × c6i.2xlarge at 80% CPU at peak
- State
- In-memory session cache on each node
- Team
- 6 backend engineers
Scale up (bigger instances)vsScale out (more stateless nodes)
Community verdict
Scale up (bigger instances) 38%Scale out (more stateless nodes) 62%
8 engineers · 5 opinions
With these constraints, what would you choose?
One choice per engineer. You can change it any time.
Scale up (bigger instances)
Scale out (more stateless nodes)
Simplicity →
Trade-offs
Dimensions
Scale up (bigger instances)Scale out (more stateless nodes)out of 5
- Simplicity
- 5Scale up (bigger instances) scores 5 of 53Scale out (more stateless nodes) scores 3 of 5
- Scalability ceiling
- 2Scale up (bigger instances) scores 2 of 55Scale out (more stateless nodes) scores 5 of 5
- Fault tolerance
Community discussion
5 comments
Have you made this decision in production? Share your reasoning.
Sign in to commentVertical scaling is underrated. A single bigger box has no coordination overhead, and an 8xlarge gives you 4× the cores today. If growth is 30% a year, that is years of headroom for one config change.
Headroom yes, but two big nodes is still two failure domains. Losing one at peak halves capacity instantly.
Scale up to buy time, but start the scale-out work now. The blocker isn't the load balancer, it's the in-memory session cache: move it to Redis (or make sessions stateless with signed tokens) and every node becomes disposable.
Once nodes are disposable you also get rolling deploys and autoscaling for free.
The session cache is ~2 days of work. That reframes it: the "simple" option is only simple because we haven't paid that debt.
Check whether you are actually CPU-bound on business logic or on JSON serialisation and GC. We halved CPU on a Java service by fixing allocation churn before touching instance sizes.