CodeKits.shopNew developer offer

Stream ZIP downloads without buffering whole folders

Design a streaming ZIP export with an authorized immutable manifest, bounded resources, backpressure and explicit handling of interrupted archives.

By Jonathan · 4 min read

Published

Authorized list, ZIP stream, and Backpressure
In this article

A ZIP export does not have to read every file into a JavaScript array before responding. A streaming archive can read one source incrementally and send compressed output as it becomes available. The harder parts are authorization, resource limits and what happens when a file disappears halfway through.

This is a Node-runtime integration walkthrough for a Next.js App Router service using Archiver. The example is a bounded adapter sketch, not a complete downloadable server. Archiver's current quickstart was consulted; no installed Archiver/Next.js integration was exercised here. Verify the exact package version in your own lockfile before shipping.

Freeze an authorized manifest

Resolve the requested folder to a list of immutable file IDs under the current account. The browser supplies a folder identifier, not arbitrary storage paths or archive entry names.

Before starting the response, enforce maximum file count, total uncompressed bytes and archive concurrency. Check each entry's canonical relative path for traversal, duplicate names and file-directory conflicts. Fetch metadata from the authorized records and resolve private storage objects server-side.

Freezing the manifest prevents newly uploaded files from unpredictably appearing mid-export. It does not freeze mutable file contents: use immutable object versions or a snapshot mechanism. If cleanup can remove sources, acquire an export lease that the cleanup worker respects.

Connect the streams before finalizing

Archiver produces a Node stream. This excerpt shows the response boundary after the application has created and authorized the archive. The omitted setup must attach source streams, archive error and warning handlers, cleanup hooks and resource limits.

zip-response.mjs — adapter boundary
import { PassThrough, Readable } from "node:stream";
 
export function zipResponse(archive, requestSignal, stopSources) {
  const output = new PassThrough();
  const stop = () => {
    archive.abort();
    stopSources(); // Must destroy every opened source stream.
    output.destroy();
  };
  const fail = error => {
    stopSources();
    archive.abort();
    output.destroy(error);
  };
  archive.on("error", fail);
  archive.on("warning", fail); // Missing private files fail this export.
  requestSignal.addEventListener("abort", stop, { once: true });
  output.once("close", () => {
    requestSignal.removeEventListener("abort", stop);
    stopSources();
    archive.abort();
  });
  archive.pipe(output);
  if (requestSignal.aborted) stop();
  else archive.finalize().catch(fail);
  return new Response(Readable.toWeb(output), {
    headers: {
      "Content-Type": "application/zip",
      "Content-Disposition": 'attachment; filename="export.zip"',
      "Cache-Control": "private, no-store"
    }
  });
}

Register event handlers before calling finalize, because completion events can occur promptly. Archiver distinguishes warnings from errors; this private export deliberately treats missing sources as failure instead of delivering an unexpectedly incomplete archive. Archiver quickstart.

This adapter does not replace source cleanup: aborting an archive is not permission to leave network downloads or file descriptors open. The caller's stopSources must be idempotent and cover already-open and queued work.

Understand what streaming saves

Backpressure limits how far a producer runs ahead of a slow recipient. It does not eliminate all memory use. The archive retains metadata, streams use buffers and compression consumes CPU. A folder of millions of tiny files can be expensive even if total bytes are modest.

Use modest compression for mixed documents. Already compressed video and JPEG files may gain little from expensive recompression. Cap simultaneous exports and prefer a queued background export for huge collections. A prepared object can then be downloaded with conventional range support, at the cost of temporary storage and another cleanup lifecycle.

Do not set a guessed content length. The final compressed size is generally unknown before generation. Check proxy response buffering and timeouts on the deployed path.

Verify failure behaviour

Create a synthetic folder with nested paths, empty files, Unicode names and a moderately large incompressible file. Export it, run a ZIP integrity check and compare hashes of extracted files against the source manifest.

Then delete a source in a controlled fixture, force a read error, cancel the browser download and throttle the receiver. Inspect open descriptors, memory and active source requests afterward. Once response headers or body bytes have been sent, an archive error cannot become a neat JSON error response; the client receives a failed or corrupt transfer. The UI should explain how to retry.

No integration result is claimed here. These acceptance tests are essential, especially the disconnect case. Combine this with safe folder manifests, or ask about a Next.js document portal.