After a full year investigation with Micro$oft and another impacted vendor, Micro$oft has informed us that they will not be fixing the bug below, and will also not release any official documentation. As such, I will provide what technical information I can here to save some poor soul a year of pain.

I will only be referring to the vendor as such. They will be spared a direct name-and-shame (this time) given that they were also not aware of this issue when they made the decisions they did, and have been provided a technical breakdown of this impact as well.

This issue has been observed in our environment on server 2008 through server 2019.

The Setup:

Our Antivirus software began leveraging Volume Shadow Copy (VSS) to take a snapshot of all drives (usually 2) on all servers every 4 hours. The vendor's intent with these snapshots was to provide a rollback feature in the event of a cryptolocker event. I have not been provided any disaster recovery literature utilizing this feature for our environment, but that does not mean it doesn't exist outside my scope.

The Problem:

My team responds to automated alerts for disk space exhaustion. These can also result in an on-call being notified as a drive filling can result in a larger cascade failure across our environment. We noticed an uptick in calls, and after investigating one of the impacted machines, we noticed a discrepancy: while the drive was reported by Windows as full, Spacemonger and wintree showed the space as available. A quick file copy test showed that the space was indeed unavailable to write into.

The first machine was recovered with a reboot. An investigation ticket was raised after the second machine was found with this behavior and placed in my queue, and I tapped a coworker to tag along for the ticket as a second set of eyes and because they were also interested in it.

The Investigation:

My teammate was investigating an impacted machine with me, and found that running chkdsk [drive letter] /v and waiting 10 minutes caused all the space to return. This confused both of us as this command shouldn't change anything, only display information. This quickly became our triage path moving forward: run the check disk command, wait 10 minutes, reboot if it didn't recover.

Running Spacemonger as system displayed accurate Volume System Information file sizes and drive state, allowing us to quickly identify the footprint moving forward.

One of our impacted machines did next to nothing, acting as a relay for some web traffic. It has ~1GB of actual data on a 60GB F: drive, and would fill every 3 weeks. This box quickly became our main investigation machine. Being a virtual machine, snapshots, and even full dumps to convert to windows debug files were taken.

I traced the activity of this box down to a hidden system file in the Volume System Information folder, but it was only identified as a GUID. I would later identify this as a system Index file. Further investigation with Windbg showed these as being Volume Shadow Copy files. The only 'service' on our investigation machine that used Volume Shadow Copy was our Antivirus, in order to take snapshots every 4 hours. It wasn't long before I had the vendor engaged.

This same week, this failure occurred on a database server. Rather than running the check disk, the tech attempted to extend the drive. This resulted in a corrupted drive that had to be restored from backup, and suddenly there was great interest in our investigation. This quickly resulted in both Vendor and Micro$oft being on investigation calls. There was much arguing and passing the blame: Microsoft claimed Vendor was not using Volume Shadow Copy properly and that was resulting in the failure. Vendor pushed back that there was no literature or behavior to indicate they were causing this issue. Eventually I managed to get both entities to recreate the failure in their respective labs.

The Failure Chain:

• As snapshots are created and removed, VSS tracks the changes in an ‘index’ file.
• This index file is a hidden system file located in the System Volume Information folder, and does not have a proper file name, only a GUID (system identifier). This file is usually ~3KB under normal operation.
• Other file system operations are also tracked in the index file.
• Per Microsoft, the maximum number of snapshots that can be tracked in this index file is 512 (since last reboot).
• Once this 512 count has been exceeded in the index, null data begins to write to the index file at a rate of ~10KB/s.
• This write will continue until all available drive space is consumed by the index file.

Some of our Volume Shadow Copies are configured to route both drive C:/ and F:/ to F:/ (Such as Databases). This cuts the time to failure down as 2 drives worth of snapshots, in addition to any other application using Volume Shadow copy quickly exhausting this 512 figure.

Kick in the teeth:

Micro$oft confirmed they had internal documentation of this issue, but both declined to fix this issue or release any official documentation concerning it. Micro$oft confirmed many times during the investigation and during the resolution that we are not in any way misconfiguring Volume Shadow Copy, and that there is no expectation for our configuration to not work as intended.

Vendor has also taken our finding back to their internal teams, and I hope will be adjusting their practices and internal literature.

Resolution:

Our internal team, given the above information, has elected to disable the snapshot feature. I am providing this post in hopes to save someone else out there the headache this all has been.