The Incident

A few days ago, we upgraded our server from one type of storage to another. Suddenly, most of our data disappeared from view. It still existed on disk, but we could not see it in the search dialog. What happened?

What Actually Happened

Here’s what was going on under the hood.

We store our data in the file system and treat folder names as case-insensitive. At startup, the system scans folders on disk one by one and inserts them into a case-insensitive dictionary keyed by folder name. If two folders differ only by case, first wins. E.g., if we first see Foo and then foo, the latter is ignored, as if it didn’t exist.

So the behavior depends on directory iteration order.

We ended up having both Foo and foo on disk. Foo had most of the data. foo was created by accident and had no useful data at all.

Old storage reading order: Apple, Foo, foo, Hickory.
New storage reading order: Apple, foo, Foo, Hickory.

On the old storage, Foo came before foo, so everything worked. On the new storage, foo came first and shadowed Foo, which made most of the data effectively disappear.

Why We Had Both Foo and foo

Now the obvious question: how did we end up with both in the first place?

Internal folder IDs are case-insensitive, but some write-backs fail to normalize IDs to the canonical form. As a result, they update foo instead of Foo, creating a second directory.

This went unnoticed on developers’ machines, which are Macs with a case-insensitive file system. There, foo and Foo resolve to the same folder, so writes still land in the right place.

It also went unnoticed on the old storage. There, foo happened to appear after Foo in directory listings, so those write-backs were effectively ignored.

It only surfaced after we switched storage and the ordering flipped, allowing foo to shadow Foo.

How to Avoid This Kind of Bug

A few takeaways:

1. Consider something other than the file system (e.g., a database) for storing persistent data.
2. If you do use the file system, always sort directory listings. Do not rely on the order returned by the file system.
3. Match your development environment to production. If production is case-sensitive, use a case-sensitive file system locally.

How to Create a Case-Sensitive Volume on MacOS

1. Create a case-sensitive disk image.
hdiutil create -type SPARSEBUNDLE -fs "APFSX" -size 20g -volname "CaseSensitiveVolume" ~/dev/case-sensitive.disk

2. Mount the image.
hdiutil attach ~/dev/images/case-sensitive.disk

After that, it appears as /Volumes/CaseSensitiveVolume.

3. Use folders in that volume directly, or create symbolic links:
mkdir /Volumes/CaseSensitiveVolume/data
ln -s /Volumes/CaseSensitiveVolume/data ~/dev/my-program/data

Conclusion

Storing data on a file system can look attractive, but not all file systems behave the same. They expose a similar interface, but the details vary.

Two details that matter here:
– whether names are case-sensitive
– directory iteration order

Moving between file systems with different behavior can introduce subtle bugs.

Don’t let these things bite you. If you use an AI agent, make it audit your code for assumptions about case sensitivity and directory ordering.