A few days ago, we upgraded our server from one type of storage to another. Suddenly, most of our data disappeared from view. It still existed on disk, but we could not see it in the search dialog. What happened?<\/p>\n
Here\u2019s what was going on under the hood.<\/p>\n
We store our data in the file system and treat folder names as case-insensitive. At startup, the system scans folders on disk one by one and inserts them into a case-insensitive dictionary keyed by folder name. If two folders differ only by case, first wins. E.g., if we first see So the behavior depends on directory iteration order.<\/p>\n We ended up having both Old storage reading order: On the old storage, Now the obvious question: how did we end up with both in the first place?<\/p>\n Internal folder IDs are case-insensitive, but some write-backs fail to normalize IDs to the canonical form. As a result, they update This went unnoticed on developers’ machines, which are Macs with a case-insensitive file system. There, It also went unnoticed on the old storage. There, It only surfaced after we switched storage and the ordering flipped, allowing A few takeaways:<\/p>\n 1. Consider something other than the file system (e.g., a database) for storing persistent data. 1. Create a case-sensitive disk image. 2. Mount the image. After that, it appears as 3. Use folders in that volume directly, or create symbolic links: Storing data on a file system can look attractive, but not all file systems behave the same. They expose a similar interface, but the details vary.<\/p>\n Two details that matter here: Moving between file systems with different behavior can introduce subtle bugs.<\/p>\n Don\u2019t let these things bite you. If you use an AI agent, make it audit your code for assumptions about case sensitivity and directory ordering.<\/p>\n","protected":false},"excerpt":{"rendered":"Foo<\/code> and then foo<\/code>, the latter is ignored, as if it didn\u2019t exist.<\/p>\nFoo<\/code> and foo<\/code> on disk. Foo<\/code> had most of the data. foo<\/code> was created by accident and had no useful data at all.<\/p>\nApple, Foo, foo, Hickory<\/code>.
\nNew storage reading order: Apple, foo, Foo, Hickory<\/code>.<\/p>\nFoo<\/code> came before foo<\/code>, so everything worked. On the new storage, foo<\/code> came first and shadowed Foo<\/code>, which made most of the data effectively disappear.<\/p>\nWhy We Had Both Foo and foo<\/h2>\n
foo<\/code> instead of Foo<\/code>, creating a second directory.<\/p>\nfoo<\/code> and Foo<\/code> resolve to the same folder, so writes still land in the right place.<\/p>\nfoo<\/code> happened to appear after Foo<\/code> in directory listings, so those write-backs were effectively ignored.<\/p>\nfoo<\/code> to shadow Foo<\/code>.<\/p>\nHow to Avoid This Kind of Bug<\/h2>\n
\n2. If you do use the file system, always sort directory listings. Do not rely on the order returned by the file system.
\n3. Match your development environment to production. If production is case-sensitive, use a case-sensitive file system locally.<\/p>\nHow to Create a Case-Sensitive Volume on MacOS<\/h2>\n
\nhdiutil create -type SPARSEBUNDLE -fs \"APFSX\" -size 20g -volname \"CaseSensitiveVolume\" ~\/dev\/case-sensitive.disk<\/code><\/p>\n
\nhdiutil attach ~\/dev\/images\/case-sensitive.disk<\/code><\/p>\n\/Volumes\/CaseSensitiveVolume<\/code>.<\/p>\n
\nmkdir \/Volumes\/CaseSensitiveVolume\/data<\/code>
\nln -s \/Volumes\/CaseSensitiveVolume\/data ~\/dev\/my-program\/data<\/code><\/p>\nConclusion<\/h2>\n
\n– whether names are case-sensitive
\n– directory iteration order<\/p>\n