Article

The Silent Killer: When Your Logs Tell a Different Story Than Your Monitoring

5 min read
Originally published on Medium. Read, clap or comment on Medium ↗

The realtime socket dashboard was green. All systems operational. No alerts firing. Everything looked perfect.

Except our users couldn’t do anything.

Every API request to our upstream service was returning 504 Gateway Timeout. The kind of error that makes your stomach drop because it could be anything. Network issues? Database deadlocks? A rogue deployment? The realtime WebSocket connections were flowing smoothly, mocking us with their perfect health while the REST API burned.

This is the story of how a seemingly healthy system was silently suffocating.

When Green Doesn’t Mean Go

Our monitoring told us a beautiful lie. The Socket.IO service? Humming along perfectly. Connection counts stable. Memory usage normal. CPU barely breaking a sweat.

But the upstream service — our main API that the socket service proxied to — was lit up like a Christmas tree. Red. Everywhere red. 504 errors cascading across every endpoint.

The disconnect was maddening. How could the proxy be fine while everything it touched was failing?

The Logs Don’t Lie

When dashboards fail you, grep saves you.

I SSH’d into the server and did what every engineer does when nothing makes sense: I checked the nginx access logs.

tail -f /var/log/nginx/access.log

Normal traffic. 200s, some 304s, the usual stuff. Then I checked the error logs:

tail -f /var/log/nginx/error.log

And there it was. Not buried. Not subtle. Glaringly, almost mockingly obvious:

2025/09/24 11:21:19 [alert] 761#761: *896774 socket() failed (24: Too many open files) while connecting to upstream, client: 172.*.*.*, server: realtime-socket.example.com, request: "GET /socket.io/?EIO=4&transport=websocket HTTP/1.1", upstream: "http://127.0.0.1:3000/socket.io/?EIO=4&transport=websocket", host: "realtime-socket.example.com"

Too many open files.

The most frustratingly simple error message in all of systems engineering. Four words that explain everything and nothing simultaneously.

The Invisible Wall

Here’s what was actually happening:

The WebSocket connections were established and stable. They were just sitting there, happily maintaining their persistent connections. Each one holding open a file descriptor.

But when the upstream API tried to handle actual work — processing requests, querying databases, opening new connections — nginx couldn’t give it the file descriptors it needed. The limit had been reached.

The WebSocket service looked healthy because those connections were already established. They weren’t asking for anything new. They were the problem precisely because they were working so well — camping out on file descriptors like tourists who refuse to leave the hotel room.

Meanwhile, every new upstream request was being rejected at the socket level. nginx literally couldn’t open the socket to connect to the upstream service.

Result? 504 Gateway Timeout. The gateway couldn’t complete the request because it couldn’t even establish the connection.

The Iceberg Principle

This is what makes “too many open files” so insidious. The symptoms show up nowhere near the cause.

Your monitoring shows:

The natural assumption? Something’s wrong with the upstream service. You restart it. Check the database. Review recent deployments. Scan for memory leaks.

All while the real problem is sitting one layer above, in the proxy, quietly exhausting a resource you probably weren’t even monitoring.

The Investigation That Should Have Been Faster

If I’m being honest, I wasted two hours looking in the wrong places:

Hour 1: Checked the upstream service

Hour 2: Checked network and infrastructure

Minute 121: Finally checked nginx error logs

The answer was waiting there the whole time, shouting at me in plaintext.

The Fix: A Three-Step Recovery

Step 1: Emergency Triage

First, I needed to verify the limit:

cat /proc/$(cat /var/run/nginx.pid)/limits | grep "open files"

Output:

Max open files    1024    1024    files

There it was. 1024 file descriptors max. With 800+ persistent WebSocket connections and all the other files nginx needed to operate, we were constantly hitting the ceiling.

Step 2: Raise the System Limits

Edit /etc/security/limits.conf:

* soft nofile 65535
* hard nofile 65535

Then create /etc/systemd/system/nginx.service.d/override.conf:

[Service]
LimitNOFILE=65535

Reload systemd:

sudo systemctl daemon-reload
sudo systemctl restart nginx

Step 3: Configure nginx Properly

Updated /etc/nginx/nginx.conf:

worker_rlimit_nofile 65535;
events {
    worker_connections 10000;
    use epoll;
}

And configured proper WebSocket timeouts in the server block:

location /socket.io/ {
    proxy_pass http://127.0.0.1:3000;
    
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    
    # Long timeouts for persistent connections
    proxy_connect_timeout 7d;
    proxy_send_timeout 7d;
    proxy_read_timeout 7d;
    
    proxy_buffering off;
}

The Moment of Truth

sudo nginx -t
sudo systemctl restart nginx

I held my breath and refreshed the monitoring dashboard.

Green. Actually green this time. The 504 errors stopped instantly. Upstream requests flowing normally again.

The WebSocket connections stayed healthy — but now the rest of the system could breathe too.

What I Learned (The Hard Way)

1. Your monitoring is lying to you

Not intentionally, but by omission. We were monitoring everything except file descriptors. Now we track them religiously:

watch -n 5 'lsof -u nginx | wc -l'

2. Check the logs first, not last

I lost several hours checking everything except the error logs. Now, my incident response checklist starts with:

  1. Check error logs
  2. Check access logs
  3. Then check everything else

3. Persistent connections are resource hogs

WebSockets, Server-Sent Events, long-polling — they’re all file descriptor vampires. Plan for 10x the file descriptors you think you need.

4. The symptom location ≠ the problem location

The upstream service was showing 504s, but the problem was in the proxy. The WebSocket service looked healthy, but it was consuming all the resources. Always look at the layer between the broken thing and the user.

5. Default limits were set in 1995

The default 1024 file descriptor limit made sense when servers handled hundreds of short-lived HTTP requests. It makes zero sense in the era of persistent connections and real-time systems.

The Monitoring To Add

We now track:

# Current open files by nginx
lsof -u nginx | wc -l
# Current limit
cat /proc/$(cat /var/run/nginx.pid)/limits | grep "open files"# Connection state distribution
ss -s# Per-worker file descriptor usage
for pid in $(pgrep nginx); do 
    echo "PID $pid: $(ls /proc/$pid/fd | wc -l) files"
done

We alert when open files exceed 70% of the limit. Never again will we hit the ceiling without warning.

The Questions That Haunt Me

How many other times had this happened at smaller scales? How many random 504s that “fixed themselves” were actually this problem partially manifesting?

How much revenue did we lose before I finally checked the error logs?

Why didn’t our monitoring coverage include such a fundamental resource?

Your Action Checklist

Don’t make my mistakes. Do this today:

Check your current limits right now:

Set up monitoring for open file descriptors

If you’re running WebSockets, increase limits preemptively:

Make error logs your first stop in investigations

Load test for persistent connections, not just requests/second

The Irony

The system that looked perfectly healthy was the one killing everything else. The green dashboard was the red flag.

Sometimes the scariest production issues are the ones that don’t trigger alerts.

Have you been burned by missing log entries while chasing ghosts in your monitoring? What’s your “should have checked the logs first” story? Share in the comments.

If this saves someone hours of investigation time, it’ll have been worth sharing. Consider sending it to your ops team before they learn this lesson at 3 AM.

← All articles Discuss on Medium ↗