TL;DR: Sometimes the simplest solutions hide behind the most frustrating bugs. A missing cleanTarget toggle in an Azure DevOps pipeline nearly drove me mad — until it didn’t.
The Perfect Deploy Gone Wrong
It was supposed to be a routine Tuesday morning. Our team had just completed a significant feature release, and the Azure DevOps pipeline ran flawlessly. All green checkmarks. All stages completed successfully. The deployment logs showed zero errors. Everything looked perfect.
But something felt off.
Our latest API changes weren’t live. The endpoints that should have been updated were still returning old responses. The database schema modifications we’d carefully tested weren’t reflecting in production. It was as if we’d deployed to yesterday instead of today.
“The build succeeded,” I said, staring at the screen.
“But the release didn’t update,” my teammate replied, equally confused.
And so began my descent into the Azure DevOps rabbit hole.
The Investigation Begins
As a DevOps engineer, my first instinct was to trust the logs. After all, if the pipeline says it succeeded, it succeeded — right? Wrong. I learned this lesson the hard way.
Step One: Chasing Release Logs
I pulled up the release definition and started digging through the deployment logs. Every single step showed green. The artifact was downloaded. The files were copied to the web server. According to the telemetry, everything happened exactly as intended.
Yet somehow, the old code was still running.
I spent an hour combing through those logs, searching for hidden errors or warnings that might explain the discrepancy. I even checked the raw output for subtle failures masked by successful exit codes. Nothing. The logs were clean — perhaps too clean.
Step Two: Checking the Release Version on the Server
Next, I SSHed into the production server and started investigating manually. This is where things got interesting.
I checked the file timestamps. The deployment supposedly completed at 2:47 PM, but the files on disk showed a modification time from our previous release at 11:30 AM. How was that possible?
I navigated to the application directory and examined the version information embedded in the DLLs. Still the old version. I checked the config file — untouched since yesterday. Even the application logs were pulling from stale sessions.
The server was running the old code, but our pipeline insisted it had deployed the new code. This was no longer just confusing; it was alarming.
Step Three: Testing the Endpoints
I decided to test the actual API endpoints to confirm my suspicions. Using Postman, I hit one of the new endpoints we’d added. The response? An HTTP 404. The endpoint didn’t exist — because the old code was still running.
I tested another endpoint that should have been modified. Same result. The changes we’d made were nowhere to be found in the running application.
By this point, I’d been investigating for nearly two hours, and the mystery had only deepened. The deployment was “successful,” but the application wasn’t actually updated. This shouldn’t be possible.
The Breakthrough Moment
Frustration was setting in. I decided to step back and think about what could cause a deployment to appear successful while leaving old code untouched.
The possibilities raced through my mind:
- A file locking issue? But the logs said files were copied.
- A caching problem? But this was production, not a browser.
- A deployment timeout that got masked? Unlikely, but possible.
Then I opened the Azure DevOps pipeline definition itself — the YAML configuration that orchestrates everything. I started examining each task, scrutinizing every parameter.
And there it was.
In the Copy Files task, there was a checkbox labeled cleanTarget. It was unchecked.
My heart sank. All this investigation, and the answer was sitting there the whole time — waiting for me to notice it.
Understanding the Problem
For those unfamiliar, the cleanTarget option in Azure DevOps determines whether the deployment process should remove existing files before deploying new ones.
Without cleanTarget enabled, the deployment works on a merge basis. New files are added, and modified files are overwritten, but old files are never deleted. This means:
- If you remove a file from your codebase and deploy, the old version still exists on the server
- Outdated DLLs remain in the bin directory
- Old configuration files linger in the application folder
- Stale web service registrations don’t get cleared
In our case, something about how the files were being deployed wasn’t triggering the proper replacements. The application pool recycle wasn’t enough to clear the in-memory cache of old assemblies. We needed a clean slate.
The Solution: Turning On cleanTarget
The fix was simple: enable the cleanTarget toggle.
I made the change, triggered a new release, and watched as the deployment proceeded.
This time was different. The deployment task didn’t just copy files — it replaced everything. The old files were removed. The bin directory was cleared of stale assemblies. The application pool recycled with a clean slate.
When I tested the endpoints five minutes later, they worked. The new API responses came through. The version information in the DLLs matched our latest build.
The deployment was finally complete.
Why This Happened: The Lessons Learned
Reflecting on the investigation, I realized several things:
First, successful deployment logs don’t always tell the whole story. A task can complete without error while still failing to achieve its intended outcome. In this case, the files were “deployed,” but the deployment was incomplete.
Second, in web applications with in-memory caching and compiled assemblies, simply replacing files might not be enough. You need to ensure a complete refresh — hence the need for cleanTarget.
Third, I should have checked the pipeline definition earlier. Instead of spending hours investigating symptoms on the server, I could have solved this in minutes by examining the configuration.
The Takeaway
Sometimes the most frustrating bugs have the simplest fixes. A single checkbox consumed my entire day — but that day taught me valuable lessons about deployments, caching, and the importance of understanding your CI/CD configuration at a granular level.
For DevOps engineers, the moral is clear: when something doesn’t match expectations, don’t just read the logs. Read the configuration. Verify the actual state of your infrastructure. Test the symptoms directly. And never underestimate the power of a checkbox you didn’t even know was there.
The next time a deployment looks successful but isn’t, I’ll remember the cleanTarget toggle. And I’ll probably check the configuration settings before spending two hours investigating the server.
Lessons learned. Investigation closed. Moving on.
Have you experienced a similar situation? Share your DevOps horror stories in the comments — I’d love to know I’m not the only one who’s been tripped up by a simple setting.