Every developer knows the feeling: a weird error, a half-remembered answer, and the exact StackOverflow page that would save the day — except you’re offline. StackOverflow deserves to be readable anywhere, with no network required.
Thankfully, Zim’s exist for many important information sources. After much futzing, stackoverflow is up to date now.
Two complete StackOverflow ZIM archives, live on archive.org right now with every question, answer, comment, and tag page, fully searchable in Kiwix or any ZIM reader:
The rebuild kit, now public at github.com/johnsmoses/stackoverflow-zim-rebuild — lets anyone rebuild or update the archive from a fresh StackExchange data dump, with all six CI jobs green as of publication.
What’s available
Archive
Size
Contents
stackoverflow-final.zim
142 GB (152,103,236,002 bytes)
Full build with images — all 4,375,716 staged images
stackoverflow-nopic.zim
69 GB (73,421,904,273 bytes)
Text-only build (--without-images), same pages
Both archives contain 85,441,337 ZIM entries: 24,152,540 question pages, 65,900 tag pages, and 101 list pages, built in July 2026 from the official StackExchange data dump (SNAPSHOT_ID=2026-07-06). Fully searchable — titles, tags, and full text — in Kiwix or any ZIM reader.
A note on images. 521,489 images failed to stage during the original build. A recovery pipeline brought back 410,856 of them (78.8%) using four sources: the archive.org stack-exchange-images corpus (62 ZIPs, 859 GB), the live CDN, XML dump scans, and an edge resolver. The ones that couldn’t be recovered render as a graceful “External visual asset unavailable” placeholder rather than a broken link.
How it was built
Three weeks, one data dump, and a lot of patience. The build runs on sotoki, the OpenZIM project’s StackOverflow-to-ZIM converter, fed by the official data dump. The pipeline looks like this:
Stage
What happens
Dump & stage
Parse the official XML dump, render every question once to static HTML, catalog image failures
Recover images
Match the 521,489 failed hashes against the IA corpus, live CDN, XML scans, and an edge resolver
Assemble
Build the ZIM from the stage (--assemble-only) with sanitized titles
Verify
Re-derive checksums, compare against the July 2026 baseline
Differences from base sotoki
The build depends on nine local modifications to sotoki, captured as a patch series (base 157ca9a, v3.0.2):
Patch
What it does
0001
Staging/assemble/resume/content-threads CLI options and context fields
0002
Staged rendering with manifest checkpointing (atomic index.html + manifest.json)
0003
Assemble-only ZIM builder, Unicode title sanitization (fixes the libzim crash at ~21.7M), staged-HTML fixes
Fresh Redis metadata rebuild (FLUSHDB, never FLUSHALL), incremental-mode metadata passes, opt-in stale-page pruning with audit journals
0009
Local dump archives via --archive-dir (per-snapshot build dir, forced re-extraction, SHA-256 stamping)
The rebuild kit
The rebuild-kit repository, github.com/johnsmoses/stackoverflow-zim-rebuild, is now public. It pins the exact sotoki source state (upstream base + the 9-patch series) and provides everything needed to reproduce or update the archive:
patch series and lockfile (sotoki.lock), applied and verified by CI;
baseline restore tooling (the July 2026 stage is restorable from a verified bundle);
the image recovery pipeline (recovery/ modules), dry-run by default;
37+ offline tests that need no redis, libzim, or network.
The core flow is:
make bootstrap → make restore-baseline → make update →
make recover-images → make finalize-placeholders → make assemble → make verify
Point make update at a fresh dump: incremental mode re-renders only what changed, recovery tops up images, and make verify checks against the July 2026 baseline.
Licensing & takedown
Stack Exchange content (questions, answers, comments, profile text) is CC BY-SA 4.0 — the share-alike obligation applies to the covered content, not to the tooling.
The sotoki patches are GPL-3.0-only — a derivative of GPL-3.0 sotoki, shipped as patches rather than a fork bundle.
The rebuild-kit tooling is CC0-1.0, dedicated to the public domain. Third-party image assets are not blanket-licensed; they are included on an availability basis and their rights belong to their actual rightsholders. Being findable on the Internet Archive is not the same as being licensed — availability is not a copyright license.
Try it
Download either ZIM from archive.org and open it in Kiwix — desktop, Android, iOS, or kiwix-serve on a Pi. Search titles, tags, and full text — no network needed.
Where this is going
Updates flow whenever Stack Exchange publishes a new dump; the kit handles the rest. The text-only stackoverflow-nopic.zim is promising material for offline search, RAG, and responsible LLM experimentation — suitability, not a claim of license-free training data; the licensing above still applies.
If you’ve ever wanted a personal, offline, fully searchable copy of StackOverflow — it’s here. And if you want to build your own, the rebuild kit is open.
No affiliation: this is an unofficial project. It is not affiliated with, endorsed by, or sponsored by Stack Overflow / Stack Exchange or the OpenZIM project.
How an effort to bring an old ZIM archive up to date turned into an infrastructure struggle against rate limits, silent sync bugs, and a 13-year CDN migration mismatch
By John Moses
While the StackExchange network hosts its raw data dumps on the Internet Archive, they are distributed as giant, raw XML files. You can’t easily query a 103 GB XML file on a laptop to debug an environment or look up a forgotten regex pattern when you’re completely offline. To make this data actually usable, it needs to be compiled into a ZIM archive—a highly compressed, fully indexed format browseable via tools like Kiwix.
Rather than building an archive completely from scratch, my goal was to take an existing local snapshot and run an incremental update to bring it in line with the latest site content. What seemed like a straightforward delta-update task quickly devolved into an infrastructure headache involving an 850 GB data dump, multi-interface WireGuard routing, and a 13-year-old CDN migration mismatch.
What I got in the end is two production-ready ZIM files totaling 211 GB, containing 24.15 million questions, tens of millions of answers, and over 4.3 million fully recovered images. Here is the post-mortem of how the update pipeline was built, the bottlenecks that stalled it, and the engineering workarounds required to get it across the finish line.
The Pipeline Architecture
The baseline data flow spans raw XML parsing, a distributed asset recovery framework, and a multi-threaded compiler:
Pipeline Stage
Inputs
Core Process / Operations
Outputs
1. Delta Scraping
Raw StackExchange XML Dumps
sotoki parses updates, skips unchanged content, and tracks rankings in Redis.
Updated Staging Directory & 55M Redis Keys
2. Multi-Track Recovery
521,489 failed image hashes
Parallel execution across IA ZIP dumps, WireGuard IP rotation, raw XML byte scanning, and edge scraping.
410,856 Recovered Visual Assets (78.8% success)
3. ZIM Assembly
755 GB Staging Directory (HTML + Images)
Optimized libzim compilation using strict Unicode title sanitization and --assemble-only flags.
Phase 1: The Incremental Update and the 500,000-Image Wall
The foundation of the project relies on sotoki, the OpenZIM project’s StackExchange scraper. Because I was updating an existing stage directory (/home/jmoses/sotoki-stage/), the scraper could skip unchanged posts and only process the delta updates.
Running on a local workstation featuring a 32-thread Ryzen CPU and 128 GB of RAM, the update parse still processed a massive modern footprint. It maintained:
24,152,540 staged HTML pages organized in a directory structure by post ID.
55 million Redis keys to handle voting scores, answer counts, and internal page ranking.
4,375,716 staged images stored by their MD5 hash.
But when the update phase pulled in the latest batch of posts, the logs revealed that 521,489 images failed to download. Cloudflare rate-limiting on StackOverflow’s live CDN (i.sstatic.net) caps incoming traffic at roughly 500 requests per 5 minutes per origin IP. Relying purely on standard live network fetching left roughly 12% of the archive’s visual assets replaced by a generic 1,852-byte WebP placeholder file.
Leaving half a million broken image links on popular technical questions wasn’t an option. The project had to pivot into a multi-stage image recovery operation.
Phase 2: Extracting Assets from the 850 GB Internet Archive Dump
To fix the missing images, I had to account for historical infrastructure changes. Historically, StackOverflow hosted user images on Imgur (i.stack.imgur.com) before migrating to their own i.sstatic.net domain in 2024. The Internet Archive preserved a complete historical snapshot of those Imgur-hosted assets split into 62 ZIP files totaling 859 GB.
Mapping the missing images to the ZIP archive presented a technical hurdle. Sotoki hashes files based on their live rewritten URLs:
To match the new delta files inside the backup without unzipping nearly a terabyte of data directly onto a fragile filesystem, we reversed the process:
I scanned the staged HTML pages for <a href="..."> wrappers surrounding the placeholder images to extract their intended filenames.
I deployed a custom Docker container (so-ia-sstatic-basename) directly onto my Unraid NAS. By targeting the raw disk share to avoid FUSE filesystem overhead, disk I/O performance jumped from a crawl of 1.3 MB/s to 34 MB/s.
The container matched the filenames inside the IA ZIP files on the fly, extracted them, converted static PNGs/JPEGs to WebP, and preserved animated GIFs.
This phase successfully recovered 233,884 images from the historical backup.
Phase 3: Bypassing Cloudflare via WireGuard IP Rotation
For images uploaded after the 2023 Internet Archive snapshot, the only source was the live i.sstatic.net CDN. To bypass the aggressive 500-request limit, I configured a fleet of Docker workers on my NAS, each bound to a distinct WireGuard VPN interface (wg0, wg3, etc.).
By setting up an internal SQLite state machine to queue and track the downloads, each container ran its traffic through a completely separate public IP address, multiplying the rate-limit ceiling. This setup pulled down an additional 147,903 post-2023 images without triggering IP bans.
Phase 4: Reversing One-Way Hashes via Raw XML Scanning
Even after the Internet Archive extraction and the WireGuard live scrape, 121,298 images remained completely unmapped. In these cases, the HTML contained raw image tags (<img src="../../images/HASH">) with no anchor tag wrapper. I had the target MD5 hash, but reversing an MD5 hash to find the original source filename is mathematically impossible.
I initially tried searching StackOverflow’s Posts.xml (a massive 103 GB uncompressed file containing 59.8 million rows) for occurrences of i.sstatic.net to map the missing hashes. I got zero matches across all 60 million rows.
The issue turned out to be how the data was historically stored. The raw database dump preserves history as it was written, meaning it contains old i.stack.imgur.com URLs, whereas the scraper generated hashes based on the modern, rewritten live site. We were looking for a domain string that literally didn’t exist in the file.
To solve this, I wrote a high-performance raw byte scanner (recover_unmapped.py). It streamed Posts.xml directly out of a compressed 7z archive, fast-rejected any lines that lacked the words imgur or sstatic to preserve CPU cycles, and used a regex to pull the base filename out of the old Imgur URLs. It then calculated candidate hashes for multiple variations:
Python
# The scanner tested several common permutations to find a match
hash_variants = [
md5("https://i.sstatic.net/" + filename),
md5("http://i.sstatic.net/" + filename),
md5("https://i.stack.imgur.com/" + filename)
]
This raw byte scanner successfully unmasked and recovered 43,565 hashes from Posts.xml and Comments.xml that would have otherwise been lost forever.
Phase 5: Handling the Long Tail with an Edge Case Resolver
The remaining missing assets were scattered across 116 different external hosts like GitHub, Postimg, Gyazo, Dropbox, and YouTube thumbnails. These weren’t hosted on StackOverflow’s CDN at all. I deployed an enhanced edge resolver (rescue_edge_cases_enhanced.py) packed with web-scraping fallbacks to clean up the data:
GitHub Camo Decoding: It reversed GitHub’s image proxy hex encodings back to their true origin URLs.
Extended Page Scraping: If a URL pointed to an HTML page instead of a raw image, the script scraped OpenGraph tags (og:image), Twitter cards, and HTML5 <picture> source tags to extract the underlying media.
MIME Sniffing & FFmpeg Transcoding: For servers returning incorrect content-type headers, the script analyzed the file’s magic bytes directly and used FFmpeg to transcode problematic formats into lightweight WebP images.
This resolved an additional 2,765 images, while confirming that another ~1,200 URLs were genuinely dead (HTTP 404/403).
Phase 6: The Silent Sync Bug
With over 410,000 missing images successfully recovered on the NAS, I copied them over to the main staging directory on my workstation using what I thought was a safe command:
I kicked off the final ZIM build, waited 14 hours, and opened up a test page in Kiwix. 11 out of 14 images were still placeholders.
The culprit was a classic silent logic error. The staging directory already contained the 1,852-byte placeholder WebP files generated during the initial failed scrape. Because files with those exact hash names already existed, --ignore-existing skipped every single recovered image.
To fix it, I switched to an explicit manifest-based sync pipeline. The script identified the exact placeholder hashes, verified their precise 1,852-byte footprint, intersected them with the recovery directory, and forced an explicit overwrite using targeted tar streams.
Phase 7: Overcoming Title Crashes and Final Assembly
Compiling tens of millions of files into a single index causes immense strain on serialization libraries. During early test runs, libzim repeatedly threw a hard crash around the 21.7 millionth question:
Unknown asynchronous exception: Too much loss of data during title indexing
A deep dive into the text formatting revealed that a handful of StackOverflow titles contained invisible, malformed Unicode control and formatting characters (such as zero-width joiners and bidirectional text markers). When libzim normalized the strings, it threw a fatal exception.
I patched the core sotoki library with a custom cleanup function (sanitize_zim_title()) to strip out Cc, Cf, and Cs Unicode character categories and collapse erratic whitespace before handing data over to the compiler.
Because I was compiling from a pre-existing stage directory, I was able to leverage optimization flags (--assemble-only --keep --keep-redis). This bypassed re-parsing entirely, reducing the final ZIM assembly time from 48 hours down to roughly 14 hours per run.
The Final Output & The Future LLM Vision
The completed update pipeline successfully generated two distinct archive flavors:
1. stackoverflow-final.zim (142 GB)
The complete offline copy. Contains all 24.15 million questions, answers, comments, tags, and 4,375,716 fully rendered images (representing a 78.8% total recovery rate of all previously broken assets). Any remaining unrecoverable images drop back to a clean, custom “Visual asset unavailable” graphic rather than displaying a broken browser icon.
2. stackoverflow-nopic.zim (69 GB)
While the text-only ZIM acts as a great low-capacity option for mobile devices, its primary purpose is strategic: creating a clean, highly structured local data source for Large Language Models (LLMs).
By rewriting image blocks cleanly to read “Image omitted in this no-pic archive,” we’ve removed the noise of dead layout links while preserving pure code, prose, and metadata hierarchy. In a future post, I’ll be breaking down exactly how I am pipe-lining this text-only ZIM directly into a local RAG (Retrieval-Augmented Generation) framework to act as a highly optimized, hallucination-free knowledge base for a local LLM setup.
Key Takeaways for Processing Data at Scale
The Database Dump Preserves History, Not Modern Views: Don’t trust live site transformations when working with historical updates. StackOverflow’s modern frontend mask hides a decade of domain migrations. Always check the raw XML data dumps before writing your parsing algorithms.
Size Thresholds are Fragile: Assuming a file is a placeholder based on a “less than 2KB” threshold will eventually corrupt your real data. Real images can occasionally compress to the exact same footprint as your error placeholders. Use exact size matching or explicit filename catalogs.
Isolate I/O Bottlenecks on Shared Storage: If you are processing data out of massive compressed archives on a network array, bypass virtualized filesystem layers (like FUSE). Reading from direct disk shares can unlock massive performance gains.
Sanitize Strings Early: When writing data into highly optimized C++ indexing engines like libzim, never trust raw user input. Run strict character normalization and strip invisible formatting bytes before initiating a long assembly run.
This might seem like a relative waste of energy, considering the value of most of the questions and answers coming from people who may or may not have the best solutions to an issue. This is really just a smaller part of a larger RAG that should allow any LLM to reach out and fill gaps that would otherwise slow down a development session. I’ll be discussing how i sanitize a majority of this data in future posts.
This was an interesting exercise in data processing, though i’m sure i could have chosen a smaller goal. Moving forward, I should have the foundation for ingesting many varied sources of data to end up with my own little library to utilize as needed!