The morning started like any other — until our monitoring systems lit up red.
Our Elasticsearch cluster, which had been humming along reliably for months, suddenly began rejecting write requests with a cryptic 400 error. The API was throwing illegal_argument_exception messages left and right. It took us a few minutes to parse the error, but when we did, the message was clear: we'd hit the shard limit.
1,996 out of 2,000 maximum shards. Open. Consumed. Full.
The Hunt Begins
In the world of distributed systems, when production goes down, the first rule is simple: stay calm and dig into the logs. Our team sprang into action, tracing the request path that had triggered the error. The failing operation was straightforward — just trying to write a chat session document to our platform. Nothing that should have required 10 additional shards.
But that’s what the error said: the operation would add [10] total shards, which would push us over the limit.
As we investigated further, a troubling pattern emerged. Our Elasticsearch cluster wasn’t supposed to have thousands of indices. We should have had a handful of well-organized indices, each with a manageable number of shards. Instead, when we ran GET _cat/shards?v, we found something that looked less like a database and more like a filesystem that had exploded.
There were so many indices.
The Culprit Reveals Itself
The smoking gun came when we started tracing back to where these indices were being created. A microservice that handled caching had been given the green light to store temporary data. The developer who wrote the script had chosen Elasticsearch as the storage backend, which made sense — it was available, it was scalable, and it could handle high throughput.
But here’s where things went sideways: instead of using a single index and storing different cache entries as documents, the script was creating a new index for each cache key, almost as if they were treating Elasticsearch like Redis, where you could just create new “buckets” on the fly.
It was a fundamental misunderstanding of how Elasticsearch works. Redis is a key-value store where you can create keys without penalty. Elasticsearch is a search engine where indices are heavyweight — each one has its own configuration, mappings, and shard allocation. Creating thousands of indices is like creating thousands of folders when you should have just had one well-organized directory with thousands of files inside.
The script had been running for weeks, dutifully creating one new index per cache operation. By the time we discovered it, we’d accumulated so many indices that we’d hit the cluster-wide shard limit — a safety mechanism designed to prevent exactly this kind of runaway resource consumption.
The Cleanup
The fix was decisive: delete the offending indices. All of them. It was the kind of cleanup that would make a disk defragmentation feel gentle by comparison. Within minutes of removing the bloated indices, our shard count plummeted back to reasonable levels, and our cluster was breathing again.
But the real work started after the emergency was resolved.
Lessons Learned
This incident taught us several critical lessons that resonated through our entire engineering culture:
First: understand your database’s model before using it. Elasticsearch is not a cache. It’s a search and analytics engine optimized for a specific access pattern. Treating it like Redis was bound to fail eventually.
Second: set guardrails before they’re needed. We added limits on index creation, set up better monitoring on index count, and established policies about which services could create indices and how.
Third: document architectural decisions. The developer who wrote the caching script wasn’t being reckless — they were solving a problem with the tools available to them. But there had been no documentation, no examples, no architectural guidelines that said, “This is the wrong approach.”
Fourth: make it easy to do the right thing. We ended up creating a dedicated caching layer with proper patterns, and made it the default choice for services that needed temporary storage. A good abstraction prevents bad decisions downstream.
The Broader Picture
In retrospect, this was a relatively benign failure. We caught it, fixed it, and nothing important was lost. But it reminded us of something every infrastructure team learns the hard way: the gap between understanding a system and using it correctly can be vast.
A developer can read the Elasticsearch documentation, understand what shards are, and still make a design decision that leads to a production incident. Not because they’re careless, but because under pressure, it’s easy to choose the familiar pattern even in an unfamiliar context.
The real win wasn’t in the debugging — it was in the conversation that followed. It gave us an opportunity to share knowledge across teams, to standardize on better patterns, and to make it easier for future developers to make the right choice without having to learn it the hard way.
So if you’re reading this and you’re building on Elasticsearch, Redis, or any sophisticated system: take a moment to understand not just how to use it, but why it was designed the way it was. The pain of learning it right is far less than the pain of explaining to your team why production is down because you treated a search engine like a cache.