Browse cheat sheets · APIs

Technology: APIsCategory: TroubleshootingDifficulty: IntermediateUpdated

Troubleshoot API Requests

Troubleshoot failing or slow API requests by isolating DNS, TLS, authentication, routing, rate limits, payloads, gateways, application errors, retries, and downstream dependencies.

On this page
  1. Problem
  2. Quick answer
  3. Code / commands
  4. 1. Reproduce outside the application
  5. 2. Capture status, body, and headers separately
  6. 3. Add a correlation ID
  7. 4. Test the exact HTTP method
  8. 5. Send the exact request body
  9. 6. Confirm the request Content-Type
  10. 7. Confirm Accept separately
  11. 8. Diagnose DNS timing
  12. 9. See which IP was reached
  13. 10. Override DNS for one controlled test
  14. 11. Check TCP connection timing
  15. 12. Check TLS handshake timing
  16. 13. Inspect TLS details
  17. 14. Do not use --insecure as the fix
  18. 15. Use the correct internal CA
  19. 16. Test mTLS where required
  20. 17. Detect proxy influence
  21. 18. Bypass a proxy for one host
  22. 19. Force proxy use for comparison
  23. 20. Check redirect behavior
  24. 21. Do not send credentials blindly across redirects
  25. 22. Debug authentication separately
  26. 23. Distinguish 401 and 403
  27. 24. Verify tenant/resource authorization
  28. 25. Diagnose 404
  29. 26. Diagnose 405
  30. 27. Diagnose 409
  31. 28. Diagnose 412
  32. 29. Diagnose 415
  33. 30. Diagnose 422
  34. 31. Diagnose 429
  35. 32. Avoid retry storms
  36. 33. Retry only when operation semantics are safe
  37. 34. Use idempotency keys for retryable create operations when supported
  38. 35. Diagnose 500
  39. 36. Diagnose 502
  40. 37. Diagnose 503
  41. 38. Diagnose 504
  42. 39. Identify which layer returned the error
  43. 40. Measure time to first byte
  44. 41. Measure response size
  45. 42. Check compression
  46. 43. Check pagination before blaming latency
  47. 44. Reproduce a specific page
  48. 45. Keep filters identical while comparing pages
  49. 46. Validate JSON response shape
  50. 47. Inspect response headers separately from JSON parsing
  51. 48. Confirm character encoding if text is corrupted
  52. 49. Check request size for 413
  53. 50. Check header size for 431
  54. 51. Test CORS separately from API reachability
  55. 52. If cURL works but browser fails
  56. 53. If local works but CI fails
  57. 54. If local works but container fails
  58. 55. If one region fails
  59. 56. If only one client fails
  60. 57. If all clients fail at once
  61. 58. If failures begin after deployment
  62. 59. If failures are intermittent
  63. 60. Watch for token-expiry patterns
  64. 61. Prevent token-refresh stampedes
  65. 62. Check clock synchronization
  66. 63. Check connection reuse behavior
  67. 64. Test repeated requests
  68. 65. Compare HTTP versions when relevant
  69. 66. Inspect negotiated HTTP version
  70. 67. Do not confuse client timeout with server timeout
  71. 68. Align timeout budgets intentionally
  72. 69. Avoid retrying beyond the caller’s deadline
  73. 70. Check downstream dependencies for slow requests
  74. 71. Track dependency timings separately
  75. 72. Check database query behavior
  76. 73. Check connection pool exhaustion
  77. 74. Check thread/event-loop starvation
  78. 75. Check serialization cost for huge responses
  79. 76. Check gateway transformations
  80. 77. Check WAF/security policy
  81. 78. Check caching
  82. 79. Bypass cache only if the API supports a safe diagnostic path
  83. 80. Compare ETag behavior
  84. 81. Check idempotency on write failures
  85. 82. Distinguish transport failure from HTTP failure
  86. 83. Check cURL exit code
  87. 84. Do not hide the error body in automation
  88. 85. Capture a minimal incident record
  89. 86. Compare good vs bad requests
  90. 87. Reproduce with the smallest request
  91. 88. Change one variable at a time
  92. 89. Do not disable security controls to make the test pass
  93. 90. Stop when evidence points outside the API
  94. What the result means
  95. What to look for
  96. Failures cluster by one client
  97. Failures cluster by one backend IP
  98. Failures begin at token expiry
  99. 5xx body is HTML
  100. 429 grows after the first throttling response
  101. High TTFB but healthy API CPU
  102. Fast API locally, slow through gateway
  103. POST timeout followed by duplicate records
  104. Production notes
  105. Reproduction quality matters
  106. Protect secrets in diagnostics
  107. Timeouts and retries interact
  108. Gateways change the troubleshooting surface
  109. Monitoring should preserve correlation
  110. One successful request does not prove health
  111. Common mistakes
  112. 1. Starting in application code before checking DNS/TLS/status
  113. 2. Testing a different request than the one that fails
  114. 3. Treating 401 and 403 as interchangeable
  115. 4. Increasing timeouts before finding the slow dependency
  116. 5. Retrying every failure automatically
  117. 6. Using --insecure as a permanent TLS fix
  118. 7. Sharing verbose traces with secrets
  119. 8. Assuming every 5xx came from application code
  120. 9. Looking only at average latency
  121. 10. Changing several things at once
  122. Next step
  123. Related deep dives

Problem

An API request is failing, timing out, returning the wrong status, or behaving differently between environments.

The challenge is to determine which layer is failing:

client
→ DNS
→ network
→ TLS
→ proxy / gateway / load balancer
→ authentication
→ routing
→ application
→ database / downstream service

The goal is to reproduce the smallest failing request, capture enough evidence to identify the failing layer, and avoid random changes that hide the real problem.

Quick answer

Start with a reproducible request:

curl \
  --silent \
  --show-error \
  --location \
  --dump-header headers.txt \
  --output body.json \
  --write-out \
  "status=%{http_code} ip=%{remote_ip} dns=%{time_namelookup}s connect=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s\n" \
  "https://api.example.com/v1/reports"

Then classify the failure:

No connection
→ DNS, routing, firewall, proxy, port

TLS failure
→ hostname, certificate chain, trust, mTLS

401
→ authentication

403
→ authorization / tenant / policy

404
→ route, version, resource, gateway mapping

409 / 412
→ state or concurrency conflict

415 / 422
→ content type or validation

429
→ rate limiting / retry behavior

5xx
→ application, gateway, or downstream failure

Slow TTFB
→ server or dependency processing

Fast TTFB, slow total
→ payload size / transfer path

Always capture:

timestamp
request ID / correlation ID
method
URL
status
response headers
safe response body
client identity
timing
environment

Never capture credentials in shared logs.

Code / commands

1. Reproduce outside the application

Use cURL to separate application-client behavior from API behavior:

curl \
  --verbose \
  "https://api.example.com/v1/health"

If cURL succeeds but the application fails, compare:

  • URL
  • method
  • headers
  • authentication
  • proxy settings
  • TLS trust
  • timeout
  • request body
  • redirect handling

Do not assume the server is at fault until the request shapes are equivalent.


2. Capture status, body, and headers separately

curl \
  --silent \
  --show-error \
  --dump-header headers.txt \
  --output body.json \
  --write-out "%{http_code}\n" \
  "https://api.example.com/v1/reports"

This is better than mixing everything into terminal output during incident analysis.


3. Add a correlation ID

REQUEST_ID="debug-$(date +%s)"

curl \
  --header "X-Correlation-ID: $REQUEST_ID" \
  "https://api.example.com/v1/reports"

Use the API’s documented correlation header if one already exists.

The same ID should appear in gateway/application telemetry when possible.


4. Test the exact HTTP method

curl \
  --request POST \
  "https://api.example.com/v1/reports"

Do not test a failing POST endpoint with GET and conclude the route is healthy.

Different methods can use different:

  • authorization
  • routing
  • body parsing
  • backend logic

5. Send the exact request body

curl \
  --request POST \
  --header "Content-Type: application/json" \
  --data @payload.json \
  "https://api.example.com/v1/reports"

Use a saved payload when reproducing a production issue.

Sanitize sensitive fields before sharing it.


6. Confirm the request Content-Type

For JSON:

curl \
  --header "Content-Type: application/json" \
  --data @payload.json \
  "https://api.example.com/v1/reports"

If the server returns 415, verify the actual header reaching the service.

A proxy or client library may alter it.


7. Confirm Accept separately

curl \
  --header "Accept: application/json" \
  "https://api.example.com/v1/reports"

Remember:

Content-Type
→ format of request body

Accept
→ preferred response format

8. Diagnose DNS timing

curl \
  --silent \
  --output /dev/null \
  --write-out "dns=%{time_namelookup}s\n" \
  "https://api.example.com/health"

If DNS time is unusually high relative to normal for the same environment, inspect:

  • resolver
  • VPN
  • corporate DNS
  • split DNS
  • container DNS
  • VNet/VPC DNS

Do not diagnose from one sample only.


9. See which IP was reached

curl \
  --silent \
  --output /dev/null \
  --write-out "remote_ip=%{remote_ip}\n" \
  "https://api.example.com/health"

Useful when:

  • DNS returns several addresses
  • traffic is regionally routed
  • one load-balancer/backend path is unhealthy

10. Override DNS for one controlled test

curl \
  --resolve api.example.com:443:203.0.113.10 \
  "https://api.example.com/health"

This preserves the hostname for TLS/SNI while connecting to the specified IP.

Use only when you know the target IP is valid.


11. Check TCP connection timing

curl \
  --silent \
  --output /dev/null \
  --write-out "connect=%{time_connect}s\n" \
  "https://api.example.com/health"

Long connection time can point toward:

  • network path
  • firewall
  • overloaded load balancer
  • packet loss
  • unreachable backend path

12. Check TLS handshake timing

curl \
  --silent \
  --output /dev/null \
  --write-out "tls=%{time_appconnect}s\n" \
  "https://api.example.com/health"

For HTTPS, compare:

time_connect
vs
time_appconnect

A large gap may indicate TLS negotiation delay.


13. Inspect TLS details

curl \
  --verbose \
  "https://api.example.com/health"

Look for:

  • certificate subject/SAN
  • issuer
  • certificate verification
  • TLS negotiation
  • SNI/hostname mismatch
  • client certificate requirements

Verbose output can expose headers and tokens. Redact before sharing.


14. Do not use --insecure as the fix

Diagnostic-only example:

curl \
  --insecure \
  "https://api.example.com/health"

If this works while normal verification fails, investigate:

  • trust chain
  • hostname
  • internal CA
  • certificate expiry
  • system trust store

Do not leave certificate validation disabled in production clients.


15. Use the correct internal CA

curl \
  --cacert company-root-ca.pem \
  "https://internal-api.example.com/health"

This preserves TLS verification.


16. Test mTLS where required

curl \
  --cert client-cert.pem \
  --key client-key.pem \
  "https://api.example.com/v1/data"

If mTLS fails, inspect both:

  • server certificate validation
  • client certificate acceptance

17. Detect proxy influence

Check environment:

env | grep -i proxy

Common variables:

HTTP_PROXY
HTTPS_PROXY
NO_PROXY

A request may behave differently inside:

  • laptop
  • CI
  • container
  • VM
  • Kubernetes pod

because proxy configuration differs.


18. Bypass a proxy for one host

curl \
  --noproxy "api.example.com" \
  "https://api.example.com/health"

If direct works but proxied fails, investigate proxy policy/routing.


19. Force proxy use for comparison

curl \
  --proxy "http://proxy.example.com:8080" \
  "https://api.example.com/health"

Use only if that proxy is approved for the environment.


20. Check redirect behavior

curl \
  --include \
  "https://api.example.com/v1/reports"

Then follow:

curl \
  --location \
  --include \
  "https://api.example.com/v1/reports"

Redirects can hide:

  • HTTP→HTTPS transitions
  • login pages
  • wrong base paths
  • gateway rewrites
  • region changes

21. Do not send credentials blindly across redirects

Inspect the redirect target first.

Credentials should not be forwarded to an unrelated host.


22. Debug authentication separately

Bearer token request:

curl \
  --header "Authorization: Bearer $ACCESS_TOKEN" \
  "https://api.example.com/v1/reports"

If response is 401, check:

  • token present
  • expiry
  • issuer
  • audience
  • signature
  • token type
  • client clock

Do not log the token itself.


23. Distinguish 401 and 403

Typical interpretation:

401
→ missing or invalid authentication

403
→ identity accepted but action not allowed

For 403, inspect:

  • scope
  • role
  • tenant
  • resource ownership
  • network/access policy

24. Verify tenant/resource authorization

For multi-tenant APIs:

authenticated tenant
must match
resource tenant

A valid token alone does not prove access to the requested resource.


25. Diagnose 404

Check:

base URL
route
HTTP method
API version
resource ID
gateway route
tenant mapping

A 404 can be generated by:

  • API gateway
  • application router
  • application resource lookup

Identify which layer responded.


26. Diagnose 405

If you get:

405 Method Not Allowed

inspect the Allow header when present.

The route may exist but reject the method.


27. Diagnose 409

Typical causes:

  • optimistic concurrency conflict
  • duplicate unique key
  • invalid state transition
  • already-completed operation

Do not retry automatically without understanding the conflict semantics.


28. Diagnose 412

A 412 Precondition Failed often means a conditional request failed.

Example:

If-Match: "old-etag"

The resource changed after the client read it.

Reload the resource before deciding whether to retry.


29. Diagnose 415

Check:

Content-Type
body encoding
multipart boundary
client serializer
gateway transformation

Do not assume the JSON body is valid just because the client object serialized locally.


30. Diagnose 422

Inspect semantic validation:

  • required combinations
  • date relationships
  • domain rules
  • enum values
  • resource state

The request may be syntactically valid but logically unacceptable.


31. Diagnose 429

Inspect response headers:

curl \
  --include \
  "https://api.example.com/v1/data"

Look for:

Retry-After
RateLimit-*
X-RateLimit-*

Then inspect:

  • requests per client
  • concurrency
  • retries
  • batch/page size
  • one noisy caller

32. Avoid retry storms

Bad pattern:

429
→ immediate retry
→ 429
→ immediate retry

Prefer bounded backoff and respect Retry-After.


33. Retry only when operation semantics are safe

Potentially transient statuses:

429
502
503
504

may be retryable.

But safety depends on:

  • idempotency
  • method
  • whether server may already have committed work
  • provider guidance

Do not blindly retry POST/PATCH/DELETE.


34. Use idempotency keys for retryable create operations when supported

curl \
  --request POST \
  --header "Idempotency-Key: $REQUEST_ID" \
  --header "Content-Type: application/json" \
  --data @payload.json \
  "https://api.example.com/v1/payments"

This works only if the server implements idempotency semantics.


35. Diagnose 500

A 500 indicates unexpected server-side failure.

Capture:

request ID
timestamp
route
safe request metadata
caller identity

Then correlate with server logs.

Do not expose stack traces to the caller.


36. Diagnose 502

Typical path:

client
→ gateway
→ backend

A 502 often means the gateway received an invalid/failing upstream response.

Check:

  • backend health
  • connection reset
  • upstream TLS
  • response protocol
  • proxy/gateway logs

37. Diagnose 503

Check:

  • deployment state
  • healthy backend count
  • overload
  • maintenance
  • dependency availability
  • load shedding

If Retry-After is present, honor it.


38. Diagnose 504

A 504 usually means a gateway/proxy timed out waiting for an upstream service.

Check:

gateway timeout
application duration
database duration
downstream API duration
queueing

Increasing timeout is not automatically the fix.


39. Identify which layer returned the error

Inspect:

curl \
  --include \
  "https://api.example.com/v1/reports"

Useful clues:

Server
Via
X-Request-ID
X-Correlation-ID
gateway-specific headers
Content-Type
body format

An HTML error page from a JSON API often indicates an intermediary.


40. Measure time to first byte

curl \
  --silent \
  --output /dev/null \
  --write-out "ttfb=%{time_starttransfer}s total=%{time_total}s\n" \
  "https://api.example.com/v1/reports"

Interpretation:

high TTFB
→ server/gateway/dependency processing delay

low TTFB but high total
→ large response or slow transfer

41. Measure response size

curl \
  --silent \
  --output /dev/null \
  --write-out "bytes=%{size_download}\n" \
  "https://api.example.com/v1/reports"

Large payloads can explain long total time even when server response begins quickly.


42. Check compression

curl \
  --compressed \
  "https://api.example.com/v1/reports"

For large JSON responses, compression can materially reduce transfer size.


43. Check pagination before blaming latency

If the endpoint returns a huge dataset, inspect whether the caller is:

  • requesting excessive page size
  • walking deep offsets
  • retrieving unnecessary fields
  • ignoring pagination

The problem may be API usage rather than server execution alone.


44. Reproduce a specific page

Example:

curl \
  "https://api.example.com/v1/orders?limit=100&offset=10000"

or:

curl \
  "https://api.example.com/v1/orders?limit=100&after=$CURSOR"

Compare early vs later pages.

Deep-offset slowness can be query-related.


45. Keep filters identical while comparing pages

Changing filters or sort invalidates the comparison.

Use the exact same:

  • filters
  • order
  • tenant
  • auth identity

46. Validate JSON response shape

With jq:

curl \
  --silent \
  "https://api.example.com/v1/reports" \
  | jq .

If parsing fails, inspect raw response.

You may have received:

  • HTML
  • empty body
  • truncated JSON
  • proxy error

47. Inspect response headers separately from JSON parsing

curl \
  --dump-header headers.txt \
  --output body.json \
  "https://api.example.com/v1/reports"

Do not let a JSON parser hide the actual HTTP response.


48. Confirm character encoding if text is corrupted

Inspect:

Content-Type: application/json; charset=utf-8

and the actual bytes.

Encoding problems are different from API business errors.


49. Check request size for 413

If the response is:

413 Content Too Large

inspect limits at:

  • CDN
  • WAF
  • load balancer
  • gateway
  • web server
  • application

The application may never receive the request.


50. Check header size for 431

Common causes:

  • oversized cookies
  • large JWTs
  • repeated tracing headers
  • proxy header accumulation

Again, an intermediary may reject the request before the application.


51. Test CORS separately from API reachability

Manual preflight:

curl \
  --request OPTIONS \
  --header "Origin: https://app.example.com" \
  --header "Access-Control-Request-Method: POST" \
  --header "Access-Control-Request-Headers: authorization,content-type" \
  --include \
  "https://api.example.com/v1/reports"

Remember:

browser blocks due to CORS
!=
API is unreachable

cURL does not enforce browser CORS policy.


52. If cURL works but browser fails

Inspect:

  • CORS headers
  • preflight
  • cookie SameSite rules
  • mixed content
  • browser credential policy

Do not debug the backend connection first if raw API calls succeed.


53. If local works but CI fails

Compare:

DNS
proxy
CA certificates
secret availability
service identity
firewall
egress rules
environment variables

CI often has a different network and trust environment.


54. If local works but container fails

Check from inside the container:

curl \
  --verbose \
  "https://api.example.com/health"

Inspect:

  • DNS
  • CA bundle
  • proxy env vars
  • network namespace
  • service discovery
  • mounted secrets

55. If one region fails

Compare:

  • remote IP
  • DNS answer
  • gateway region
  • backend pool
  • deployment version
  • regional dependency health

Do not assume a global API has one identical network path.


56. If only one client fails

Compare its exact request with a known-good client.

Look for:

  • different token
  • wrong base URL
  • stale API version
  • custom proxy
  • different timeout
  • request-body differences

57. If all clients fail at once

Prioritize shared components:

DNS
gateway
identity provider
backend deployment
database
shared downstream dependency

A synchronized client-side bug is less likely unless a shared deployment/config changed.


58. If failures begin after deployment

Compare:

  • deployed version
  • route configuration
  • environment variables
  • auth settings
  • schema changes
  • dependency versions
  • gateway policy

Rollback only when the evidence supports it.


59. If failures are intermittent

Capture per-request:

request ID
timestamp
remote IP
status
latency
backend instance
retry count

Intermittent failures often correlate with one:

  • instance
  • region
  • dependency
  • connection pool
  • credential refresh cycle

60. Watch for token-expiry patterns

If failures happen near a regular interval:

every 60 minutes
every 24 hours

inspect:

  • access-token expiry
  • refresh logic
  • cache invalidation
  • certificate/token rotation

61. Prevent token-refresh stampedes

In high-concurrency clients, multiple workers may all refresh an expiring token.

Use single-flight/locking or a shared token cache.

A burst of auth requests can create intermittent 401/429 failures.


62. Check clock synchronization

Token validation and TLS can fail if system time is wrong.

Inspect host/container clock when seeing:

  • token not yet valid
  • token expired unexpectedly
  • certificate validity anomalies

63. Check connection reuse behavior

A client using pooled HTTP connections may behave differently from cURL’s one-off request.

Investigate:

  • stale pooled connections
  • DNS changes
  • idle timeout
  • proxy keep-alive
  • load-balancer connection lifetime

64. Test repeated requests

for i in 1 2 3 4 5; do
  curl \
    --silent \
    --output /dev/null \
    --write-out \
    "status=%{http_code} ip=%{remote_ip} total=%{time_total}s\n" \
    "https://api.example.com/health"
done

This can reveal intermittent backend/IP behavior.

Do not turn production diagnostics into load testing.


65. Compare HTTP versions when relevant

curl \
  --http1.1 \
  "https://api.example.com/health"

and:

curl \
  --http2 \
  "https://api.example.com/health"

Use only when evidence suggests protocol-level differences.


66. Inspect negotiated HTTP version

curl \
  --silent \
  --output /dev/null \
  --write-out "%{http_version}\n" \
  "https://api.example.com/health"

67. Do not confuse client timeout with server timeout

Possible failure points:

client timeout
gateway timeout
application timeout
database timeout
downstream API timeout

Capture where the timeout is generated.


68. Align timeout budgets intentionally

Conceptually:

client timeout
>
gateway timeout
>
application downstream timeout

may be useful so downstream failures surface before outer callers give up, but actual values depend on architecture.

Do not copy universal timeout numbers.


69. Avoid retrying beyond the caller’s deadline

Retries should fit within the overall request budget.

A client that retries three long timeouts can exceed user-facing latency expectations badly.


70. Check downstream dependencies for slow requests

If TTFB is high, inspect:

  • database
  • cache
  • message broker
  • downstream APIs
  • identity calls
  • filesystem/object storage

The endpoint may only be the visible edge of a longer dependency chain.


71. Track dependency timings separately

Good server-side telemetry records:

request total
database duration
downstream API duration
queue duration
serialization duration

Without breakdown, “API took 8 seconds” is not actionable.


72. Check database query behavior

If API latency maps to database calls, inspect:

  • blocking
  • long-running queries
  • high CPU
  • high reads
  • missing/inefficient indexes
  • parameter sensitivity
  • connection pool exhaustion

Do not scale API instances to solve a database bottleneck blindly.


73. Check connection pool exhaustion

Symptoms can include:

  • intermittent timeouts
  • increased queueing
  • low backend CPU despite slow responses
  • failures under concurrency

Inspect pool metrics rather than only endpoint logs.


74. Check thread/event-loop starvation

Application symptoms can include:

  • slow responses despite healthy downstreams
  • request queue growth
  • rising concurrency
  • blocked synchronous work

Use runtime-specific telemetry.

Do not diagnose this from HTTP status alone.


75. Check serialization cost for huge responses

Large nested JSON responses can spend meaningful time in:

  • object materialization
  • serialization
  • compression
  • network transfer

Reduce payload shape before simply adding compute.


76. Check gateway transformations

Gateways may:

  • rewrite paths
  • add/remove headers
  • validate tokens
  • throttle
  • transform bodies
  • cache
  • terminate TLS

Compare request/response behavior before and after the gateway where architecture permits.


77. Check WAF/security policy

Unexpected 403 or blocked requests may come from:

  • WAF rule
  • IP restriction
  • geo restriction
  • bot protection
  • request-size policy
  • suspicious-pattern rule

Identify the responding layer before changing application authorization.


78. Check caching

A stale or inconsistent response may come from:

  • CDN
  • reverse proxy
  • API gateway
  • application cache

Inspect:

Cache-Control
Age
ETag
Last-Modified
Via
X-Cache

where available.


79. Bypass cache only if the API supports a safe diagnostic path

Do not invent headers such as:

X-Bypass-Cache: true

unless the platform explicitly supports them.

Use documented cache controls.


80. Compare ETag behavior

Request:

curl \
  --include \
  "https://api.example.com/v1/config"

Then:

curl \
  --header 'If-None-Match: "abc123"' \
  --include \
  "https://api.example.com/v1/config"

Useful for debugging stale-client behavior.


81. Check idempotency on write failures

Scenario:

POST times out

Unknown state:

Did server commit the operation before connection failed?

Do not blindly retry unless the endpoint supports idempotency or you can query operation state safely.


82. Distinguish transport failure from HTTP failure

Examples:

DNS failure
TLS failure
connection reset
timeout before HTTP response

may produce no HTTP status at all.

Your client should record both:

transport error
HTTP status if present

83. Check cURL exit code

curl \
  --fail-with-body \
  --silent \
  --show-error \
  "https://api.example.com/v1/data"

echo $?

A non-zero cURL exit code is not the same thing as an HTTP status code.


84. Do not hide the error body in automation

Prefer:

curl \
  --fail-with-body \
  --silent \
  --show-error \
  "https://api.example.com/v1/data"

over plain --fail when the body contains useful diagnostics.


85. Capture a minimal incident record

Useful template:

UTC timestamp:
Environment:
Method:
URL/path:
Caller/client:
Request ID:
HTTP status:
Transport error:
Remote IP:
DNS time:
Connect time:
TLS time:
TTFB:
Total time:
Response Content-Type:
Safe error code/message:
Retry attempted:

Do not include secrets.


86. Compare good vs bad requests

Create a side-by-side comparison of:

method
URL
query string
headers
auth identity
payload
status
timing
remote IP
environment

Differences often reveal the issue faster than reading logs in isolation.


87. Reproduce with the smallest request

Strip away optional:

  • headers
  • query params
  • payload fields
  • client wrappers

until the failure disappears.

Then add them back one at a time.

This isolates the trigger.


88. Change one variable at a time

Bad troubleshooting:

change timeout
change proxy
change token
change route
redeploy

all at once.

You lose causality.

Prefer one controlled change with before/after evidence.


89. Do not disable security controls to make the test pass

Avoid using as permanent fixes:

--insecure
disabled auth
open firewall
disabled WAF
unlimited rate limit

If one of these changes isolates the failing layer, restore the control and fix the actual configuration.


90. Stop when evidence points outside the API

If:

DNS failing

the next step is not application code.

If:

gateway 504 with slow database dependency

the next step is not changing the HTTP status handler.

Follow the evidence to the responsible layer.

What the result means

Signal Likely layer
DNS failure / high DNS time DNS / network configuration
Connection refused Endpoint, port, firewall, listener
TLS verification failure Certificate / trust / hostname
401 Authentication
403 Authorization, tenant, WAF/policy
404 Route, version, resource, gateway mapping
409 / 412 State or concurrency
415 / 422 Request format or semantic validation
429 Rate limiting / client request pattern
500 Application failure
502 Gateway-to-upstream failure
503 Availability / overload / maintenance
504 Upstream timeout
High TTFB Server/gateway/dependency processing
High total, low TTFB Transfer size / network throughput
cURL works, app fails Client behavior/configuration difference
Only one region/instance fails Regional/backend-instance path

The best troubleshooting model is:

transport
→ security
→ routing
→ application
→ dependencies

What to look for

Failures cluster by one client

Inspect client-specific:

  • credentials
  • proxy
  • SDK version
  • request shape
  • retry loop
  • tenant

Failures cluster by one backend IP

Suspect one unhealthy instance or regional path.

Failures begin at token expiry

Inspect refresh and token caching.

5xx body is HTML

Likely gateway/proxy/intermediary response rather than application JSON.

429 grows after the first throttling response

Client retries may be amplifying load.

High TTFB but healthy API CPU

Inspect downstream dependencies, queueing, connection pools, and blocking.

Fast API locally, slow through gateway

Inspect gateway/WAF/proxy path and TLS/network differences.

POST timeout followed by duplicate records

The client likely retried a non-idempotent operation without a safe idempotency mechanism.

Production notes

Reproduction quality matters

A good reproduction includes the same:

  • environment
  • method
  • headers
  • identity
  • payload
  • route

as the failing request.

A simplified request that changes those dimensions may test something else.

Protect secrets in diagnostics

Never put raw:

  • access tokens
  • API keys
  • cookies
  • passwords
  • client secrets
  • private keys

into shared tickets or logs.

Timeouts and retries interact

Long timeout + multiple retries can produce very long user-visible failures and duplicate side effects.

Design them together.

Gateways change the troubleshooting surface

In production, the externally visible API often includes several layers before the application.

Capture telemetry at each relevant boundary.

Monitoring should preserve correlation

A request ID that crosses:

client
→ gateway
→ API
→ downstream service

is far more useful than unrelated log lines.

One successful request does not prove health

Intermittent issues require multiple timestamped samples with backend/region identity where possible.

Common mistakes

1. Starting in application code before checking DNS/TLS/status

You may debug the wrong layer.

2. Testing a different request than the one that fails

Changing method, auth, payload, or environment invalidates the comparison.

3. Treating 401 and 403 as interchangeable

They point to different areas.

4. Increasing timeouts before finding the slow dependency

This can hide the symptom and worsen capacity.

5. Retrying every failure automatically

Retries can duplicate writes or create load storms.

6. Using --insecure as a permanent TLS fix

It disables certificate verification.

7. Sharing verbose traces with secrets

cURL traces can expose authentication material.

8. Assuming every 5xx came from application code

Gateways and proxies generate errors too.

9. Looking only at average latency

Intermittent failures may affect one region, instance, or dependency.

10. Changing several things at once

You lose the evidence needed to identify root cause.

Next step

Use the first failing layer to decide what to inspect next.

  • No DNS resolution → inspect resolver, VNet/VPC, container DNS, VPN, proxy.
  • Connection failure → inspect endpoint, port, firewall, listener, routing.
  • TLS failure → inspect hostname, certificate chain, trust, mTLS, system clock.
  • 401 → inspect token/key validity, issuer, audience, expiry.
  • 403 → inspect scope, role, tenant, WAF, IP/security policy.
  • 404/405 → inspect route, method, version, gateway mapping.
  • 409/412 → inspect state/version/concurrency rules.
  • 415/422 → inspect Content-Type, serializer, payload, domain validation.
  • 429 → inspect rate limits, concurrency, retries, caller identity.
  • 500 → correlate request ID to application logs.
  • 502/503/504 → inspect gateway plus upstream/backend dependencies.
  • High TTFB → inspect server execution and downstream timings.
  • Low TTFB, high total → inspect payload size, compression, network transfer.
  • cURL succeeds but application fails → compare exact headers, proxy, auth, TLS, redirect, and timeout behavior.
  • Nothing obvious appears → compare good vs bad requests and change one variable at a time.

← All cheat sheets