Parag Chitnis

Every system has limits.

• CPU saturates.
• Memory fills.
• Connections exhaust.
• Disks run out of space.
• External services slow down.

Ask:

• What happens when traffic exceeds expected capacity?
• How does the system behave when resources are constrained?
• Does it fail gracefully or catastrophically?
• Does it protect critical functionality when under stress?

Example

Imagine an API that handles 5,000 requests per minute comfortably. You have never tested it beyond that.

Scenario A:

• CPU spikes to 100%
• Database connections exhaust
• Requests pile up
• Timeouts cascade
• Entire service becomes unresponsive

Scenario B:

• Rate limiting activates
• Non-critical endpoints are throttled
• Critical routes are protected
• Some requests are rejected quickly with clear errors
• Core functionality remains available

Real-world usage is rarely smooth. Traffic comes in waves.

• Product launches
• Seasonal sales
• Promotional events
• External integrations
• Automated clients retrying aggressively

Consider:

• Can the system absorb short-term spikes without failure?
• Are queues or buffers absorbing burst?
• Are throttling or rate-limits preventing overload?
• Can important traffic be prioritized?

Example

Imagine a SaaS platform where most users log in at 9:00 am. For 10 minutes each morning, authentication traffic triples.

Without protection:

• Database connection pools are exhausted.
• Login requests time out.
• Users repeatedly retry.
• Retry traffic multiplies the load.

With proper design:

• Authentication requests are queued.
• Rate limiting protects the database.
• Retry logic includes exponential backoff.
• Low-priority background tasks pause temporarily.

The spike is absorbed instead of amplified.

A slow dependency can:

• Exhaust thread pools.
• Consume database connections.
• Trigger retries.
• Amplify load.
• Collapse unrelated services.

This is how small failures become outages.

Design thoughtfully:

• Separate critical and non-critical workloads.
• Use independent resource pools.
• Limit retries.
• Apply circuit breakers.
• Prevent shared bottlenecks.

Failures should hurt locally, not globally.

Without backpressure:

• Queues grow unbounded.
• Memory usage spikes.
• Latency increases.
• The system slows for everyone.

With backpressure:

• Excess requests are rejected early.
• Clear error responses guide clients.
• Critical flows are prioritized.
• Non-essentials tasks are paused.
• The system remains responsive for core functionality.

When incidents occur, ask:

• How quickly is failure detected?
• How easily can we roll back?
• Are recovery steps automated or manual?
• Are playbooks clear and accessible?

Example

Two services experience the same production bug.

System A:

• No automated rollback.
• Engineers manually investigate logs.
• Configuration fixes are applied via SSH.
• Recovery takes 3 hours.

System B:

• Monitoring system detects anomaly within minutes.
• Canary release automatically rolls back.
• Feature flags disable the faulty module.
• Service stabilizes in 5 minutes.

Scaling is not simply adding more machines. It is managing multiple factors as the system grows:

• Cost
• Complexity
• Consistency
• Performance
• Overheads

Think about:

• Where are the bottlenecks?
• What limits the system throughput or latency?
• Will it scale horizontally, vertically, or architecturally?
• How does growth impact infrastructure cost and operational effort?

Example

You deploy a stateless web service behind a load balancer. Traffic increases. You add more application servers, but latency remains high. Why?

True scaling may require:

• Read replicas
• Caching layers
• Partitioning or sharding
• Redesigning query patterns

A system may:

• Handle 10x traffic.
• Maintain performance.
• Pass all stress tests.

But if infrastructure cost grows faster than revenue, the system is economically unstable. Resilience includes financial sustainability.

Ask:

• What is the marginal cost per additional user?
• Does load increase cost linearly, exponentially, or minimally?
• Can infrastructure scale down during low usage?
• Are we over-provisioning permanently for rare spikes?

Ask:

• How much CPU does a single request consume?
• How much memory grows per user or per dataset?
• What is the growth rate of storage?
• At what threshold does performance degrade?

Example

A logging service stores application logs indefinitely.

Initially:

• Disk usage grows slowly.
• Costs are manageable.

After one year:

• Storage doubles.
• Query performance degrades.
• Backup times increase.
• Infrastructure cost becomes significant.

Had you tracked:

• Log growth per day
• Storage trends over time
• Query latency against dataset size

You could have:

• Implemented log retention policies
• Archived old data
• Compressed logs
• Partitioned storage