Lessons from Updating a 211 GB Offline StackOverflow Archive

How an effort to bring an old ZIM archive up to date turned into an infrastructure struggle against rate limits, silent sync bugs, and a 13-year CDN migration mismatc

By John Moses

While the StackExchange network hosts its raw data dumps on the Internet Archive, they are distributed as giant, raw XML files. You can’t easily query a 103 GB XML file on a laptop to debug an environment or look up a forgotten regex pattern when you’re completely offline. To make this data actually usable, it needs to be compiled into a ZIM archive—a highly compressed, fully indexed format browseable via tools like Kiwix.

Rather than building an archive completely from scratch, my goal was to take an existing local snapshot and run an incremental update to bring it in line with the latest site content. What seemed like a straightforward delta-update task quickly devolved into an infrastructure headache involving an 850 GB data dump, multi-interface WireGuard routing, and a 13-year-old CDN migration mismatch.

What I got in the end is two production-ready ZIM files totaling 211 GB, containing 24.15 million questions, tens of millions of answers, and over 4.3 million fully recovered images. Here is the post-mortem of how the update pipeline was built, the bottlenecks that stalled it, and the engineering workarounds required to get it across the finish line.

The Pipeline Architecture

The baseline data flow spans raw XML parsing, a distributed asset recovery framework, and a multi-threaded compiler:

Pipeline Stage	Inputs	Core Process / Operations	Outputs
1. Delta Scraping	Raw StackExchange XML Dumps	sotoki parses updates, skips unchanged content, and tracks rankings in Redis.	Updated Staging Directory & 55M Redis Keys
2. Multi-Track Recovery	521,489 failed image hashes	Parallel execution across IA ZIP dumps, WireGuard IP rotation, raw XML byte scanning, and edge scraping.	410,856 Recovered Visual Assets (78.8% success)
3. ZIM Assembly	755 GB Staging Directory (HTML + Images)	Optimized libzim compilation using strict Unicode title sanitization and --assemble-only flags.	stackoverflow-final.zim (142 GB) stackoverflow-nopic.zim (69 GB)

Phase 1: The Incremental Update and the 500,000-Image Wall

The foundation of the project relies on sotoki, the OpenZIM project’s StackExchange scraper. Because I was updating an existing stage directory (/home/jmoses/sotoki-stage/), the scraper could skip unchanged posts and only process the delta updates.

Running on a local workstation featuring a 32-thread Ryzen CPU and 128 GB of RAM, the update parse still processed a massive modern footprint. It maintained:

• 24,152,540 staged HTML pages organized in a directory structure by post ID.
• 55 million Redis keys to handle voting scores, answer counts, and internal page ranking.
• 4,375,716 staged images stored by their MD5 hash.

But when the update phase pulled in the latest batch of posts, the logs revealed that 521,489 images failed to download. Cloudflare rate-limiting on StackOverflow’s live CDN (i.sstatic.net) caps incoming traffic at roughly 500 requests per 5 minutes per origin IP. Relying purely on standard live network fetching left roughly 12% of the archive’s visual assets replaced by a generic 1,852-byte WebP placeholder file.

Leaving half a million broken image links on popular technical questions wasn’t an option. The project had to pivot into a multi-stage image recovery operation.

Phase 2: Extracting Assets from the 850 GB Internet Archive Dump

To fix the missing images, I had to account for historical infrastructure changes. Historically, StackOverflow hosted user images on Imgur (i.stack.imgur.com) before migrating to their own i.sstatic.net domain in 2024. The Internet Archive preserved a complete historical snapshot of those Imgur-hosted assets split into 62 ZIP files totaling 859 GB.

Mapping the missing images to the ZIP archive presented a technical hurdle. Sotoki hashes files based on their live rewritten URLs:

$$\text{Hash} = \text{MD5}(\text{“[https://i.sstatic.net/FILENAME.ext](https://i.sstatic.net/FILENAME.ext)”})$$

To match the new delta files inside the backup without unzipping nearly a terabyte of data directly onto a fragile filesystem, we reversed the process:

1. I scanned the staged HTML pages for <a href="..."> wrappers surrounding the placeholder images to extract their intended filenames.
2. I deployed a custom Docker container (so-ia-sstatic-basename) directly onto my Unraid NAS. By targeting the raw disk share to avoid FUSE filesystem overhead, disk I/O performance jumped from a crawl of 1.3 MB/s to 34 MB/s.
3. The container matched the filenames inside the IA ZIP files on the fly, extracted them, converted static PNGs/JPEGs to WebP, and preserved animated GIFs.

This phase successfully recovered 233,884 images from the historical backup.

Phase 3: Bypassing Cloudflare via WireGuard IP Rotation

For images uploaded after the 2023 Internet Archive snapshot, the only source was the live i.sstatic.net CDN. To bypass the aggressive 500-request limit, I configured a fleet of Docker workers on my NAS, each bound to a distinct WireGuard VPN interface (wg0, wg3, etc.).

By setting up an internal SQLite state machine to queue and track the downloads, each container ran its traffic through a completely separate public IP address, multiplying the rate-limit ceiling. This setup pulled down an additional 147,903 post-2023 images without triggering IP bans.

Phase 4: Reversing One-Way Hashes via Raw XML Scanning

Even after the Internet Archive extraction and the WireGuard live scrape, 121,298 images remained completely unmapped. In these cases, the HTML contained raw image tags (<img src="../../images/HASH">) with no anchor tag wrapper. I had the target MD5 hash, but reversing an MD5 hash to find the original source filename is mathematically impossible.

I initially tried searching StackOverflow’s Posts.xml (a massive 103 GB uncompressed file containing 59.8 million rows) for occurrences of i.sstatic.net to map the missing hashes. I got zero matches across all 60 million rows.

The issue turned out to be how the data was historically stored. The raw database dump preserves history as it was written, meaning it contains old i.stack.imgur.com URLs, whereas the scraper generated hashes based on the modern, rewritten live site. We were looking for a domain string that literally didn’t exist in the file.

To solve this, I wrote a high-performance raw byte scanner (recover_unmapped.py). It streamed Posts.xml directly out of a compressed 7z archive, fast-rejected any lines that lacked the words imgur or sstatic to preserve CPU cycles, and used a regex to pull the base filename out of the old Imgur URLs. It then calculated candidate hashes for multiple variations:

Python

# The scanner tested several common permutations to find a match
hash_variants = [
    md5("https://i.sstatic.net/" + filename),
    md5("http://i.sstatic.net/" + filename),
    md5("https://i.stack.imgur.com/" + filename)
]

This raw byte scanner successfully unmasked and recovered 43,565 hashes from Posts.xml and Comments.xml that would have otherwise been lost forever.

Phase 5: Handling the Long Tail with an Edge Case Resolver

The remaining missing assets were scattered across 116 different external hosts like GitHub, Postimg, Gyazo, Dropbox, and YouTube thumbnails. These weren’t hosted on StackOverflow’s CDN at all. I deployed an enhanced edge resolver (rescue_edge_cases_enhanced.py) packed with web-scraping fallbacks to clean up the data:

• GitHub Camo Decoding: It reversed GitHub’s image proxy hex encodings back to their true origin URLs.
• Extended Page Scraping: If a URL pointed to an HTML page instead of a raw image, the script scraped OpenGraph tags (og:image), Twitter cards, and HTML5 <picture> source tags to extract the underlying media.
• MIME Sniffing & FFmpeg Transcoding: For servers returning incorrect content-type headers, the script analyzed the file’s magic bytes directly and used FFmpeg to transcode problematic formats into lightweight WebP images.

This resolved an additional 2,765 images, while confirming that another ~1,200 URLs were genuinely dead (HTTP 404/403).

Phase 6: The Silent Sync Bug

With over 410,000 missing images successfully recovered on the NAS, I copied them over to the main staging directory on my workstation using what I thought was a safe command:

Bash

rsync -av --ignore-existing /nas/recovered/ /workstation/sotoki-stage/images/

I kicked off the final ZIM build, waited 14 hours, and opened up a test page in Kiwix. 11 out of 14 images were still placeholders.

The culprit was a classic silent logic error. The staging directory already contained the 1,852-byte placeholder WebP files generated during the initial failed scrape. Because files with those exact hash names already existed, --ignore-existing skipped every single recovered image.

To fix it, I switched to an explicit manifest-based sync pipeline. The script identified the exact placeholder hashes, verified their precise 1,852-byte footprint, intersected them with the recovery directory, and forced an explicit overwrite using targeted tar streams.

Phase 7: Overcoming Title Crashes and Final Assembly

Compiling tens of millions of files into a single index causes immense strain on serialization libraries. During early test runs, libzim repeatedly threw a hard crash around the 21.7 millionth question:

Unknown asynchronous exception: Too much loss of data during title indexing

A deep dive into the text formatting revealed that a handful of StackOverflow titles contained invisible, malformed Unicode control and formatting characters (such as zero-width joiners and bidirectional text markers). When libzim normalized the strings, it threw a fatal exception.

I patched the core sotoki library with a custom cleanup function (sanitize_zim_title()) to strip out Cc, Cf, and Cs Unicode character categories and collapse erratic whitespace before handing data over to the compiler.

Because I was compiling from a pre-existing stage directory, I was able to leverage optimization flags (--assemble-only --keep --keep-redis). This bypassed re-parsing entirely, reducing the final ZIM assembly time from 48 hours down to roughly 14 hours per run.

The Final Output & The Future LLM Vision

The completed update pipeline successfully generated two distinct archive flavors:

1. stackoverflow-final.zim (142 GB)

The complete offline copy. Contains all 24.15 million questions, answers, comments, tags, and 4,375,716 fully rendered images (representing a 78.8% total recovery rate of all previously broken assets). Any remaining unrecoverable images drop back to a clean, custom “Visual asset unavailable” graphic rather than displaying a broken browser icon.

2. stackoverflow-nopic.zim (69 GB)

While the text-only ZIM acts as a great low-capacity option for mobile devices, its primary purpose is strategic: creating a clean, highly structured local data source for Large Language Models (LLMs).

By rewriting image blocks cleanly to read “Image omitted in this no-pic archive,” we’ve removed the noise of dead layout links while preserving pure code, prose, and metadata hierarchy. In a future post, I’ll be breaking down exactly how I am pipe-lining this text-only ZIM directly into a local RAG (Retrieval-Augmented Generation) framework to act as a highly optimized, hallucination-free knowledge base for a local LLM setup.

Key Takeaways for Processing Data at Scale

1. The Database Dump Preserves History, Not Modern Views: Don’t trust live site transformations when working with historical updates. StackOverflow’s modern frontend mask hides a decade of domain migrations. Always check the raw XML data dumps before writing your parsing algorithms.
2. Size Thresholds are Fragile: Assuming a file is a placeholder based on a “less than 2KB” threshold will eventually corrupt your real data. Real images can occasionally compress to the exact same footprint as your error placeholders. Use exact size matching or explicit filename catalogs.
3. Isolate I/O Bottlenecks on Shared Storage: If you are processing data out of massive compressed archives on a network array, bypass virtualized filesystem layers (like FUSE). Reading from direct disk shares can unlock massive performance gains.
4. Sanitize Strings Early: When writing data into highly optimized C++ indexing engines like libzim, never trust raw user input. Run strict character normalization and strip invisible formatting bytes before initiating a long assembly run.

This might seem like a relative waste of energy, considering the value of most of the questions and answers coming from people who may or may not have the best solutions to an issue. This is really just a smaller part of a larger RAG that should allow any LLM to reach out and fill gaps that would otherwise slow down a development session. I’ll be discussing how i sanitize a majority of this data in future posts.

This was an interesting exercise in data processing, though i’m sure i could have chosen a smaller goal. Moving forward, I should have the foundation for ingesting many varied sources of data to end up with my own little library to utilize as needed!