Parag Chitnis

Assume things will go wrong, because they will.

• Dependencies will slow down.
• Databases will lock.
• Networks will congest.
• People will make mistakes.

Consider:

• What happens when a dependency goes down?
• Can the system retry, fall back, or degrade gracefully?
• Will failure in one part cascade into others?
• Can parts of the system fail without bringing everything down?

Example

Your checkout system depends on a third-party payment provider. What if the provider becomes unavailable?

In a fragile system:

• Checkout blocks indefinitely.
• Threads pile up waiting for timeouts.
• Users repeatedly retry.
• Impact spreads to other components.

In a reliable system:

• Circuit breaker activates after threshold failures.
• System fails fast with clear error messages.
• Retry intervals increase.
• Checkout may temporarily disable non-essential features.
• Incident is contained to payment processing.

The difference between fragile and reliable systems is not whether they fail. It is:

• How quickly failures are detected
• How clearly they are understood
• How efficiently they are resolved

When things go wrong, what matters is:

• Who Knows?
• Who Acts?
• How Fast?

Example

Imagine a background job that processes financial transactions. A subtle logic bug causes incorrect rounding in rare edge cases.

Contrast that with:

• Automated reconciliation checks
• Alerts triggered on threshold deviation
• Clear dashboards showing anomaly patterns

Early detection reduces impact dramatically.

Ask:

• Do we have clear signals when things go wrong?
• Are alerts based on actual user impact
• Are they actionable or just noisy?
• Can we distinguish minor degradation from major outage?

Example

Over time:

• Alerts are muted.
• Noise is ignored.
• And so are real incidents.

In contrast:

• Alerts are tied to user-visible impact.
• Thresholds reflect meaningful degradation.
• Severity levels are clearly defined.

When a critical alert fires, it is trusted and acted upon immediately.

Consider:

• What do users see when the system degrades?
• What do support teams see when handling tickets?
• What do engineers see when diagnosing failures?
• What does leadership see when assessing risk and impact?

Example

Suppose response time increases due to database load.

Users should see:

• Clear error messages or temporary retry prompts
• Not cryptic stack traces

Support teams should see:

• A simple status summary
• Known incident reference
• Clear explanation to communicate

Engineers should see:

• Detailed logs
• Query performance metrics
• Database health indicators

Leadership should see:

• Impact scope
• Duration
• Business risk exposure

Ask:

• Is there a clear incident owner?
• Are there clear response playbooks?
• Can recurring issues be resolved without escalation?
• Are post-incident learnings documented?

Example

An incident occurs at 2 am.

In a fragile system:

• Only one senior engineer understands recovery steps.
• Documentation is outdated.
• Diagnosis requires trial and error.
• Recovery takes hours.

In a reliable system:

• The playbook clearly defines diagnostic steps.
• Dashboards guide investigation.
• Recovery commands are standardized.
• On-call engineer resolves issue confidently.

Strong systems provide:

• Logs that explain what happened and why
• Metrics that show trends and thresholds
• Traces that reveal flow across services

Example

Imagine an inventory system distributed across different services. Users report intermittent slowness.

Without tracing:

• Each service appears healthy individually.
• Logs are fragmented.
• Root cause is unclear.

With distributed tracing:

• End-to-end request path is visible.
• Latency spike traced to one dependency.
• Bottleneck isolated quickly.