Browse cheat sheets · APIs
Troubleshoot API Requests
Troubleshoot failing or slow API requests by isolating DNS, TLS, authentication, routing, rate limits, payloads, gateways, application errors, retries, and downstream dependencies.
On this page
- Problem
- Quick answer
- Code / commands
- 1. Reproduce outside the application
- 2. Capture status, body, and headers separately
- 3. Add a correlation ID
- 4. Test the exact HTTP method
- 5. Send the exact request body
- 6. Confirm the request Content-Type
- 7. Confirm Accept separately
- 8. Diagnose DNS timing
- 9. See which IP was reached
- 10. Override DNS for one controlled test
- 11. Check TCP connection timing
- 12. Check TLS handshake timing
- 13. Inspect TLS details
- 14. Do not use --insecure as the fix
- 15. Use the correct internal CA
- 16. Test mTLS where required
- 17. Detect proxy influence
- 18. Bypass a proxy for one host
- 19. Force proxy use for comparison
- 20. Check redirect behavior
- 21. Do not send credentials blindly across redirects
- 22. Debug authentication separately
- 23. Distinguish 401 and 403
- 24. Verify tenant/resource authorization
- 25. Diagnose 404
- 26. Diagnose 405
- 27. Diagnose 409
- 28. Diagnose 412
- 29. Diagnose 415
- 30. Diagnose 422
- 31. Diagnose 429
- 32. Avoid retry storms
- 33. Retry only when operation semantics are safe
- 34. Use idempotency keys for retryable create operations when supported
- 35. Diagnose 500
- 36. Diagnose 502
- 37. Diagnose 503
- 38. Diagnose 504
- 39. Identify which layer returned the error
- 40. Measure time to first byte
- 41. Measure response size
- 42. Check compression
- 43. Check pagination before blaming latency
- 44. Reproduce a specific page
- 45. Keep filters identical while comparing pages
- 46. Validate JSON response shape
- 47. Inspect response headers separately from JSON parsing
- 48. Confirm character encoding if text is corrupted
- 49. Check request size for 413
- 50. Check header size for 431
- 51. Test CORS separately from API reachability
- 52. If cURL works but browser fails
- 53. If local works but CI fails
- 54. If local works but container fails
- 55. If one region fails
- 56. If only one client fails
- 57. If all clients fail at once
- 58. If failures begin after deployment
- 59. If failures are intermittent
- 60. Watch for token-expiry patterns
- 61. Prevent token-refresh stampedes
- 62. Check clock synchronization
- 63. Check connection reuse behavior
- 64. Test repeated requests
- 65. Compare HTTP versions when relevant
- 66. Inspect negotiated HTTP version
- 67. Do not confuse client timeout with server timeout
- 68. Align timeout budgets intentionally
- 69. Avoid retrying beyond the caller’s deadline
- 70. Check downstream dependencies for slow requests
- 71. Track dependency timings separately
- 72. Check database query behavior
- 73. Check connection pool exhaustion
- 74. Check thread/event-loop starvation
- 75. Check serialization cost for huge responses
- 76. Check gateway transformations
- 77. Check WAF/security policy
- 78. Check caching
- 79. Bypass cache only if the API supports a safe diagnostic path
- 80. Compare ETag behavior
- 81. Check idempotency on write failures
- 82. Distinguish transport failure from HTTP failure
- 83. Check cURL exit code
- 84. Do not hide the error body in automation
- 85. Capture a minimal incident record
- 86. Compare good vs bad requests
- 87. Reproduce with the smallest request
- 88. Change one variable at a time
- 89. Do not disable security controls to make the test pass
- 90. Stop when evidence points outside the API
- What the result means
- What to look for
- Failures cluster by one client
- Failures cluster by one backend IP
- Failures begin at token expiry
- 5xx body is HTML
- 429 grows after the first throttling response
- High TTFB but healthy API CPU
- Fast API locally, slow through gateway
- POST timeout followed by duplicate records
- Production notes
- Reproduction quality matters
- Protect secrets in diagnostics
- Timeouts and retries interact
- Gateways change the troubleshooting surface
- Monitoring should preserve correlation
- One successful request does not prove health
- Common mistakes
- 1. Starting in application code before checking DNS/TLS/status
- 2. Testing a different request than the one that fails
- 3. Treating 401 and 403 as interchangeable
- 4. Increasing timeouts before finding the slow dependency
- 5. Retrying every failure automatically
- 6. Using --insecure as a permanent TLS fix
- 7. Sharing verbose traces with secrets
- 8. Assuming every 5xx came from application code
- 9. Looking only at average latency
- 10. Changing several things at once
- Next step
- Related deep dives
Problem
An API request is failing, timing out, returning the wrong status, or behaving differently between environments.
The challenge is to determine which layer is failing:
client
→ DNS
→ network
→ TLS
→ proxy / gateway / load balancer
→ authentication
→ routing
→ application
→ database / downstream service
The goal is to reproduce the smallest failing request, capture enough evidence to identify the failing layer, and avoid random changes that hide the real problem.
Quick answer
Start with a reproducible request:
curl \
--silent \
--show-error \
--location \
--dump-header headers.txt \
--output body.json \
--write-out \
"status=%{http_code} ip=%{remote_ip} dns=%{time_namelookup}s connect=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s\n" \
"https://api.example.com/v1/reports"
Then classify the failure:
No connection
→ DNS, routing, firewall, proxy, port
TLS failure
→ hostname, certificate chain, trust, mTLS
401
→ authentication
403
→ authorization / tenant / policy
404
→ route, version, resource, gateway mapping
409 / 412
→ state or concurrency conflict
415 / 422
→ content type or validation
429
→ rate limiting / retry behavior
5xx
→ application, gateway, or downstream failure
Slow TTFB
→ server or dependency processing
Fast TTFB, slow total
→ payload size / transfer path
Always capture:
timestamp
request ID / correlation ID
method
URL
status
response headers
safe response body
client identity
timing
environment
Never capture credentials in shared logs.
Code / commands
1. Reproduce outside the application
Use cURL to separate application-client behavior from API behavior:
curl \
--verbose \
"https://api.example.com/v1/health"
If cURL succeeds but the application fails, compare:
- URL
- method
- headers
- authentication
- proxy settings
- TLS trust
- timeout
- request body
- redirect handling
Do not assume the server is at fault until the request shapes are equivalent.
2. Capture status, body, and headers separately
curl \
--silent \
--show-error \
--dump-header headers.txt \
--output body.json \
--write-out "%{http_code}\n" \
"https://api.example.com/v1/reports"
This is better than mixing everything into terminal output during incident analysis.
3. Add a correlation ID
REQUEST_ID="debug-$(date +%s)"
curl \
--header "X-Correlation-ID: $REQUEST_ID" \
"https://api.example.com/v1/reports"
Use the API’s documented correlation header if one already exists.
The same ID should appear in gateway/application telemetry when possible.
4. Test the exact HTTP method
curl \
--request POST \
"https://api.example.com/v1/reports"
Do not test a failing POST endpoint with GET and conclude the route is healthy.
Different methods can use different:
- authorization
- routing
- body parsing
- backend logic
5. Send the exact request body
curl \
--request POST \
--header "Content-Type: application/json" \
--data @payload.json \
"https://api.example.com/v1/reports"
Use a saved payload when reproducing a production issue.
Sanitize sensitive fields before sharing it.
6. Confirm the request Content-Type
For JSON:
curl \
--header "Content-Type: application/json" \
--data @payload.json \
"https://api.example.com/v1/reports"
If the server returns 415, verify the actual header reaching the service.
A proxy or client library may alter it.
7. Confirm Accept separately
curl \
--header "Accept: application/json" \
"https://api.example.com/v1/reports"
Remember:
Content-Type
→ format of request body
Accept
→ preferred response format
8. Diagnose DNS timing
curl \
--silent \
--output /dev/null \
--write-out "dns=%{time_namelookup}s\n" \
"https://api.example.com/health"
If DNS time is unusually high relative to normal for the same environment, inspect:
- resolver
- VPN
- corporate DNS
- split DNS
- container DNS
- VNet/VPC DNS
Do not diagnose from one sample only.
9. See which IP was reached
curl \
--silent \
--output /dev/null \
--write-out "remote_ip=%{remote_ip}\n" \
"https://api.example.com/health"
Useful when:
- DNS returns several addresses
- traffic is regionally routed
- one load-balancer/backend path is unhealthy
10. Override DNS for one controlled test
curl \
--resolve api.example.com:443:203.0.113.10 \
"https://api.example.com/health"
This preserves the hostname for TLS/SNI while connecting to the specified IP.
Use only when you know the target IP is valid.
11. Check TCP connection timing
curl \
--silent \
--output /dev/null \
--write-out "connect=%{time_connect}s\n" \
"https://api.example.com/health"
Long connection time can point toward:
- network path
- firewall
- overloaded load balancer
- packet loss
- unreachable backend path
12. Check TLS handshake timing
curl \
--silent \
--output /dev/null \
--write-out "tls=%{time_appconnect}s\n" \
"https://api.example.com/health"
For HTTPS, compare:
time_connect
vs
time_appconnect
A large gap may indicate TLS negotiation delay.
13. Inspect TLS details
curl \
--verbose \
"https://api.example.com/health"
Look for:
- certificate subject/SAN
- issuer
- certificate verification
- TLS negotiation
- SNI/hostname mismatch
- client certificate requirements
Verbose output can expose headers and tokens. Redact before sharing.
14. Do not use --insecure as the fix
Diagnostic-only example:
curl \
--insecure \
"https://api.example.com/health"
If this works while normal verification fails, investigate:
- trust chain
- hostname
- internal CA
- certificate expiry
- system trust store
Do not leave certificate validation disabled in production clients.
15. Use the correct internal CA
curl \
--cacert company-root-ca.pem \
"https://internal-api.example.com/health"
This preserves TLS verification.
16. Test mTLS where required
curl \
--cert client-cert.pem \
--key client-key.pem \
"https://api.example.com/v1/data"
If mTLS fails, inspect both:
- server certificate validation
- client certificate acceptance
17. Detect proxy influence
Check environment:
env | grep -i proxy
Common variables:
HTTP_PROXY
HTTPS_PROXY
NO_PROXY
A request may behave differently inside:
- laptop
- CI
- container
- VM
- Kubernetes pod
because proxy configuration differs.
18. Bypass a proxy for one host
curl \
--noproxy "api.example.com" \
"https://api.example.com/health"
If direct works but proxied fails, investigate proxy policy/routing.
19. Force proxy use for comparison
curl \
--proxy "http://proxy.example.com:8080" \
"https://api.example.com/health"
Use only if that proxy is approved for the environment.
20. Check redirect behavior
curl \
--include \
"https://api.example.com/v1/reports"
Then follow:
curl \
--location \
--include \
"https://api.example.com/v1/reports"
Redirects can hide:
- HTTP→HTTPS transitions
- login pages
- wrong base paths
- gateway rewrites
- region changes
21. Do not send credentials blindly across redirects
Inspect the redirect target first.
Credentials should not be forwarded to an unrelated host.
22. Debug authentication separately
Bearer token request:
curl \
--header "Authorization: Bearer $ACCESS_TOKEN" \
"https://api.example.com/v1/reports"
If response is 401, check:
- token present
- expiry
- issuer
- audience
- signature
- token type
- client clock
Do not log the token itself.
23. Distinguish 401 and 403
Typical interpretation:
401
→ missing or invalid authentication
403
→ identity accepted but action not allowed
For 403, inspect:
- scope
- role
- tenant
- resource ownership
- network/access policy
24. Verify tenant/resource authorization
For multi-tenant APIs:
authenticated tenant
must match
resource tenant
A valid token alone does not prove access to the requested resource.
25. Diagnose 404
Check:
base URL
route
HTTP method
API version
resource ID
gateway route
tenant mapping
A 404 can be generated by:
- API gateway
- application router
- application resource lookup
Identify which layer responded.
26. Diagnose 405
If you get:
405 Method Not Allowed
inspect the Allow header when present.
The route may exist but reject the method.
27. Diagnose 409
Typical causes:
- optimistic concurrency conflict
- duplicate unique key
- invalid state transition
- already-completed operation
Do not retry automatically without understanding the conflict semantics.
28. Diagnose 412
A 412 Precondition Failed often means a conditional request failed.
Example:
If-Match: "old-etag"
The resource changed after the client read it.
Reload the resource before deciding whether to retry.
29. Diagnose 415
Check:
Content-Type
body encoding
multipart boundary
client serializer
gateway transformation
Do not assume the JSON body is valid just because the client object serialized locally.
30. Diagnose 422
Inspect semantic validation:
- required combinations
- date relationships
- domain rules
- enum values
- resource state
The request may be syntactically valid but logically unacceptable.
31. Diagnose 429
Inspect response headers:
curl \
--include \
"https://api.example.com/v1/data"
Look for:
Retry-After
RateLimit-*
X-RateLimit-*
Then inspect:
- requests per client
- concurrency
- retries
- batch/page size
- one noisy caller
32. Avoid retry storms
Bad pattern:
429
→ immediate retry
→ 429
→ immediate retry
Prefer bounded backoff and respect Retry-After.
33. Retry only when operation semantics are safe
Potentially transient statuses:
429
502
503
504
may be retryable.
But safety depends on:
- idempotency
- method
- whether server may already have committed work
- provider guidance
Do not blindly retry POST/PATCH/DELETE.
34. Use idempotency keys for retryable create operations when supported
curl \
--request POST \
--header "Idempotency-Key: $REQUEST_ID" \
--header "Content-Type: application/json" \
--data @payload.json \
"https://api.example.com/v1/payments"
This works only if the server implements idempotency semantics.
35. Diagnose 500
A 500 indicates unexpected server-side failure.
Capture:
request ID
timestamp
route
safe request metadata
caller identity
Then correlate with server logs.
Do not expose stack traces to the caller.
36. Diagnose 502
Typical path:
client
→ gateway
→ backend
A 502 often means the gateway received an invalid/failing upstream response.
Check:
- backend health
- connection reset
- upstream TLS
- response protocol
- proxy/gateway logs
37. Diagnose 503
Check:
- deployment state
- healthy backend count
- overload
- maintenance
- dependency availability
- load shedding
If Retry-After is present, honor it.
38. Diagnose 504
A 504 usually means a gateway/proxy timed out waiting for an upstream service.
Check:
gateway timeout
application duration
database duration
downstream API duration
queueing
Increasing timeout is not automatically the fix.
39. Identify which layer returned the error
Inspect:
curl \
--include \
"https://api.example.com/v1/reports"
Useful clues:
Server
Via
X-Request-ID
X-Correlation-ID
gateway-specific headers
Content-Type
body format
An HTML error page from a JSON API often indicates an intermediary.
40. Measure time to first byte
curl \
--silent \
--output /dev/null \
--write-out "ttfb=%{time_starttransfer}s total=%{time_total}s\n" \
"https://api.example.com/v1/reports"
Interpretation:
high TTFB
→ server/gateway/dependency processing delay
low TTFB but high total
→ large response or slow transfer
41. Measure response size
curl \
--silent \
--output /dev/null \
--write-out "bytes=%{size_download}\n" \
"https://api.example.com/v1/reports"
Large payloads can explain long total time even when server response begins quickly.
42. Check compression
curl \
--compressed \
"https://api.example.com/v1/reports"
For large JSON responses, compression can materially reduce transfer size.
43. Check pagination before blaming latency
If the endpoint returns a huge dataset, inspect whether the caller is:
- requesting excessive page size
- walking deep offsets
- retrieving unnecessary fields
- ignoring pagination
The problem may be API usage rather than server execution alone.
44. Reproduce a specific page
Example:
curl \
"https://api.example.com/v1/orders?limit=100&offset=10000"
or:
curl \
"https://api.example.com/v1/orders?limit=100&after=$CURSOR"
Compare early vs later pages.
Deep-offset slowness can be query-related.
45. Keep filters identical while comparing pages
Changing filters or sort invalidates the comparison.
Use the exact same:
- filters
- order
- tenant
- auth identity
46. Validate JSON response shape
With jq:
curl \
--silent \
"https://api.example.com/v1/reports" \
| jq .
If parsing fails, inspect raw response.
You may have received:
- HTML
- empty body
- truncated JSON
- proxy error
47. Inspect response headers separately from JSON parsing
curl \
--dump-header headers.txt \
--output body.json \
"https://api.example.com/v1/reports"
Do not let a JSON parser hide the actual HTTP response.
48. Confirm character encoding if text is corrupted
Inspect:
Content-Type: application/json; charset=utf-8
and the actual bytes.
Encoding problems are different from API business errors.
49. Check request size for 413
If the response is:
413 Content Too Large
inspect limits at:
- CDN
- WAF
- load balancer
- gateway
- web server
- application
The application may never receive the request.
50. Check header size for 431
Common causes:
- oversized cookies
- large JWTs
- repeated tracing headers
- proxy header accumulation
Again, an intermediary may reject the request before the application.
51. Test CORS separately from API reachability
Manual preflight:
curl \
--request OPTIONS \
--header "Origin: https://app.example.com" \
--header "Access-Control-Request-Method: POST" \
--header "Access-Control-Request-Headers: authorization,content-type" \
--include \
"https://api.example.com/v1/reports"
Remember:
browser blocks due to CORS
!=
API is unreachable
cURL does not enforce browser CORS policy.
52. If cURL works but browser fails
Inspect:
- CORS headers
- preflight
- cookie SameSite rules
- mixed content
- browser credential policy
Do not debug the backend connection first if raw API calls succeed.
53. If local works but CI fails
Compare:
DNS
proxy
CA certificates
secret availability
service identity
firewall
egress rules
environment variables
CI often has a different network and trust environment.
54. If local works but container fails
Check from inside the container:
curl \
--verbose \
"https://api.example.com/health"
Inspect:
- DNS
- CA bundle
- proxy env vars
- network namespace
- service discovery
- mounted secrets
55. If one region fails
Compare:
- remote IP
- DNS answer
- gateway region
- backend pool
- deployment version
- regional dependency health
Do not assume a global API has one identical network path.
56. If only one client fails
Compare its exact request with a known-good client.
Look for:
- different token
- wrong base URL
- stale API version
- custom proxy
- different timeout
- request-body differences
57. If all clients fail at once
Prioritize shared components:
DNS
gateway
identity provider
backend deployment
database
shared downstream dependency
A synchronized client-side bug is less likely unless a shared deployment/config changed.
58. If failures begin after deployment
Compare:
- deployed version
- route configuration
- environment variables
- auth settings
- schema changes
- dependency versions
- gateway policy
Rollback only when the evidence supports it.
59. If failures are intermittent
Capture per-request:
request ID
timestamp
remote IP
status
latency
backend instance
retry count
Intermittent failures often correlate with one:
- instance
- region
- dependency
- connection pool
- credential refresh cycle
60. Watch for token-expiry patterns
If failures happen near a regular interval:
every 60 minutes
every 24 hours
inspect:
- access-token expiry
- refresh logic
- cache invalidation
- certificate/token rotation
61. Prevent token-refresh stampedes
In high-concurrency clients, multiple workers may all refresh an expiring token.
Use single-flight/locking or a shared token cache.
A burst of auth requests can create intermittent 401/429 failures.
62. Check clock synchronization
Token validation and TLS can fail if system time is wrong.
Inspect host/container clock when seeing:
- token not yet valid
- token expired unexpectedly
- certificate validity anomalies
63. Check connection reuse behavior
A client using pooled HTTP connections may behave differently from cURL’s one-off request.
Investigate:
- stale pooled connections
- DNS changes
- idle timeout
- proxy keep-alive
- load-balancer connection lifetime
64. Test repeated requests
for i in 1 2 3 4 5; do
curl \
--silent \
--output /dev/null \
--write-out \
"status=%{http_code} ip=%{remote_ip} total=%{time_total}s\n" \
"https://api.example.com/health"
done
This can reveal intermittent backend/IP behavior.
Do not turn production diagnostics into load testing.
65. Compare HTTP versions when relevant
curl \
--http1.1 \
"https://api.example.com/health"
and:
curl \
--http2 \
"https://api.example.com/health"
Use only when evidence suggests protocol-level differences.
66. Inspect negotiated HTTP version
curl \
--silent \
--output /dev/null \
--write-out "%{http_version}\n" \
"https://api.example.com/health"
67. Do not confuse client timeout with server timeout
Possible failure points:
client timeout
gateway timeout
application timeout
database timeout
downstream API timeout
Capture where the timeout is generated.
68. Align timeout budgets intentionally
Conceptually:
client timeout
>
gateway timeout
>
application downstream timeout
may be useful so downstream failures surface before outer callers give up, but actual values depend on architecture.
Do not copy universal timeout numbers.
69. Avoid retrying beyond the caller’s deadline
Retries should fit within the overall request budget.
A client that retries three long timeouts can exceed user-facing latency expectations badly.
70. Check downstream dependencies for slow requests
If TTFB is high, inspect:
- database
- cache
- message broker
- downstream APIs
- identity calls
- filesystem/object storage
The endpoint may only be the visible edge of a longer dependency chain.
71. Track dependency timings separately
Good server-side telemetry records:
request total
database duration
downstream API duration
queue duration
serialization duration
Without breakdown, “API took 8 seconds” is not actionable.
72. Check database query behavior
If API latency maps to database calls, inspect:
- blocking
- long-running queries
- high CPU
- high reads
- missing/inefficient indexes
- parameter sensitivity
- connection pool exhaustion
Do not scale API instances to solve a database bottleneck blindly.
73. Check connection pool exhaustion
Symptoms can include:
- intermittent timeouts
- increased queueing
- low backend CPU despite slow responses
- failures under concurrency
Inspect pool metrics rather than only endpoint logs.
74. Check thread/event-loop starvation
Application symptoms can include:
- slow responses despite healthy downstreams
- request queue growth
- rising concurrency
- blocked synchronous work
Use runtime-specific telemetry.
Do not diagnose this from HTTP status alone.
75. Check serialization cost for huge responses
Large nested JSON responses can spend meaningful time in:
- object materialization
- serialization
- compression
- network transfer
Reduce payload shape before simply adding compute.
76. Check gateway transformations
Gateways may:
- rewrite paths
- add/remove headers
- validate tokens
- throttle
- transform bodies
- cache
- terminate TLS
Compare request/response behavior before and after the gateway where architecture permits.
77. Check WAF/security policy
Unexpected 403 or blocked requests may come from:
- WAF rule
- IP restriction
- geo restriction
- bot protection
- request-size policy
- suspicious-pattern rule
Identify the responding layer before changing application authorization.
78. Check caching
A stale or inconsistent response may come from:
- CDN
- reverse proxy
- API gateway
- application cache
Inspect:
Cache-Control
Age
ETag
Last-Modified
Via
X-Cache
where available.
79. Bypass cache only if the API supports a safe diagnostic path
Do not invent headers such as:
X-Bypass-Cache: true
unless the platform explicitly supports them.
Use documented cache controls.
80. Compare ETag behavior
Request:
curl \
--include \
"https://api.example.com/v1/config"
Then:
curl \
--header 'If-None-Match: "abc123"' \
--include \
"https://api.example.com/v1/config"
Useful for debugging stale-client behavior.
81. Check idempotency on write failures
Scenario:
POST times out
Unknown state:
Did server commit the operation before connection failed?
Do not blindly retry unless the endpoint supports idempotency or you can query operation state safely.
82. Distinguish transport failure from HTTP failure
Examples:
DNS failure
TLS failure
connection reset
timeout before HTTP response
may produce no HTTP status at all.
Your client should record both:
transport error
HTTP status if present
83. Check cURL exit code
curl \
--fail-with-body \
--silent \
--show-error \
"https://api.example.com/v1/data"
echo $?
A non-zero cURL exit code is not the same thing as an HTTP status code.
84. Do not hide the error body in automation
Prefer:
curl \
--fail-with-body \
--silent \
--show-error \
"https://api.example.com/v1/data"
over plain --fail when the body contains useful diagnostics.
85. Capture a minimal incident record
Useful template:
UTC timestamp:
Environment:
Method:
URL/path:
Caller/client:
Request ID:
HTTP status:
Transport error:
Remote IP:
DNS time:
Connect time:
TLS time:
TTFB:
Total time:
Response Content-Type:
Safe error code/message:
Retry attempted:
Do not include secrets.
86. Compare good vs bad requests
Create a side-by-side comparison of:
method
URL
query string
headers
auth identity
payload
status
timing
remote IP
environment
Differences often reveal the issue faster than reading logs in isolation.
87. Reproduce with the smallest request
Strip away optional:
- headers
- query params
- payload fields
- client wrappers
until the failure disappears.
Then add them back one at a time.
This isolates the trigger.
88. Change one variable at a time
Bad troubleshooting:
change timeout
change proxy
change token
change route
redeploy
all at once.
You lose causality.
Prefer one controlled change with before/after evidence.
89. Do not disable security controls to make the test pass
Avoid using as permanent fixes:
--insecure
disabled auth
open firewall
disabled WAF
unlimited rate limit
If one of these changes isolates the failing layer, restore the control and fix the actual configuration.
90. Stop when evidence points outside the API
If:
DNS failing
the next step is not application code.
If:
gateway 504 with slow database dependency
the next step is not changing the HTTP status handler.
Follow the evidence to the responsible layer.
What the result means
| Signal | Likely layer |
|---|---|
| DNS failure / high DNS time | DNS / network configuration |
| Connection refused | Endpoint, port, firewall, listener |
| TLS verification failure | Certificate / trust / hostname |
401 |
Authentication |
403 |
Authorization, tenant, WAF/policy |
404 |
Route, version, resource, gateway mapping |
409 / 412 |
State or concurrency |
415 / 422 |
Request format or semantic validation |
429 |
Rate limiting / client request pattern |
500 |
Application failure |
502 |
Gateway-to-upstream failure |
503 |
Availability / overload / maintenance |
504 |
Upstream timeout |
| High TTFB | Server/gateway/dependency processing |
| High total, low TTFB | Transfer size / network throughput |
| cURL works, app fails | Client behavior/configuration difference |
| Only one region/instance fails | Regional/backend-instance path |
The best troubleshooting model is:
transport
→ security
→ routing
→ application
→ dependencies
What to look for
Failures cluster by one client
Inspect client-specific:
- credentials
- proxy
- SDK version
- request shape
- retry loop
- tenant
Failures cluster by one backend IP
Suspect one unhealthy instance or regional path.
Failures begin at token expiry
Inspect refresh and token caching.
5xx body is HTML
Likely gateway/proxy/intermediary response rather than application JSON.
429 grows after the first throttling response
Client retries may be amplifying load.
High TTFB but healthy API CPU
Inspect downstream dependencies, queueing, connection pools, and blocking.
Fast API locally, slow through gateway
Inspect gateway/WAF/proxy path and TLS/network differences.
POST timeout followed by duplicate records
The client likely retried a non-idempotent operation without a safe idempotency mechanism.
Production notes
Reproduction quality matters
A good reproduction includes the same:
- environment
- method
- headers
- identity
- payload
- route
as the failing request.
A simplified request that changes those dimensions may test something else.
Protect secrets in diagnostics
Never put raw:
- access tokens
- API keys
- cookies
- passwords
- client secrets
- private keys
into shared tickets or logs.
Timeouts and retries interact
Long timeout + multiple retries can produce very long user-visible failures and duplicate side effects.
Design them together.
Gateways change the troubleshooting surface
In production, the externally visible API often includes several layers before the application.
Capture telemetry at each relevant boundary.
Monitoring should preserve correlation
A request ID that crosses:
client
→ gateway
→ API
→ downstream service
is far more useful than unrelated log lines.
One successful request does not prove health
Intermittent issues require multiple timestamped samples with backend/region identity where possible.
Common mistakes
1. Starting in application code before checking DNS/TLS/status
You may debug the wrong layer.
2. Testing a different request than the one that fails
Changing method, auth, payload, or environment invalidates the comparison.
3. Treating 401 and 403 as interchangeable
They point to different areas.
4. Increasing timeouts before finding the slow dependency
This can hide the symptom and worsen capacity.
5. Retrying every failure automatically
Retries can duplicate writes or create load storms.
6. Using --insecure as a permanent TLS fix
It disables certificate verification.
7. Sharing verbose traces with secrets
cURL traces can expose authentication material.
8. Assuming every 5xx came from application code
Gateways and proxies generate errors too.
9. Looking only at average latency
Intermittent failures may affect one region, instance, or dependency.
10. Changing several things at once
You lose the evidence needed to identify root cause.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.