Preservation · Jun 25, 2026 · 5 min read

Web archiving for small institutions

How small archives and libraries can collect the web: WARC files, Archive-It and Webrecorder tools, scoping a collection, QA checks and providing access.

An archivist at a workstation reviewing a website snapshot timeline

Websites disappear quietly. A department redesigns its pages, a grant ends, a hosting bill goes unpaid, and a decade of public record is gone. Large national libraries crawl the web at scale, but they rarely capture the local: the county historical society, the campus student newspaper, the small press that only ever published online.

That gap is where small institutions matter most. You do not need a data center to archive the web well. You need a clear scope, a standard file format, a capture tool that fits your content, and a habit of checking your work.

Why the web needs collecting

Web pages behave differently from paper. They change without notice, they depend on scripts and outside services, and they leave no carbon copy behind. A link that worked when a researcher cited it may now point to a parking page. Studies of link rot in scholarly and legal citations have documented this for years, and anyone who maintains a reading list has seen it firsthand.

Collecting the web is also a continuation of older library work. In the 2000s, the MetaScholar Initiative at Emory's Woodruff Library ran MetaCombine, a project that tested how harvested metadata and ordinary web content could be searched together. The premise was simple. Scholarly evidence lives on the open web as well as in catalogs, and a library that ignores the web leaves a hole in the record. Web archiving makes that premise durable: it keeps a copy of the web content itself, not only a pointer to it.

WARC and the tools around it

The core standard is WARC, the Web ARChive file format. It is published as ISO 28500, first in 2009 and revised in 2017. A WARC file stores the raw HTTP requests and responses from a crawl, along with metadata records about how the capture was made. That is why an archived page can be replayed later with its images, stylesheets and scripts intact.

Around WARC sits a small ecosystem of tools. Most small institutions use some combination of these:

  • Heritrix, the open source crawler developed by the Internet Archive. It is built for large, scheduled crawls of many sites.
  • Archive-It, a subscription service from the Internet Archive. It wraps crawling, storage, quality review and public access into one hosted service.
  • Webrecorder tools, such as ArchiveWeb.page and ReplayWeb.page. They capture pages through a real browser as you interact with them, which suits complex, script-heavy sites. They can package captures in WACZ, a zipped bundle that contains WARC data plus indexes.
  • Wayback-style replay software, which serves archived captures back to readers by URL and date.

The choice is less about features and more about staffing. A hosted service trades money for time. Open tools trade time for control.

ApproachGood fit forMain trade-off
Hosted service (Archive-It)Ongoing collecting programs with modest technical staffRecurring cost and dependence on a vendor
Browser-based capture (Webrecorder tools)Small, high-value sites with interactive contentManual effort per page or session
Self-run crawler (Heritrix)Institutions with servers and a systems personSetup, tuning and storage are all on you
quality metrics · sample collection
findability
82%
completeness
67%
consistency
74%
rights clarity
58%
link health
91%
Figure 1 · interactive illustration

Scope before you crawl

The most common mistake is starting with the crawler. Start with a collecting policy instead. Web archiving is a selection activity, exactly like accepting a box of papers. The same questions apply. What do we collect, why, and for whom?

A workable scope document answers a few concrete points:

  • Seeds. The starting URLs, such as a campus news site or a list of local organizations.
  • Boundaries. Which domains and paths are in, and which are out. Calendars, search result pages and shopping carts can trap a crawler in endless URLs.
  • Frequency. A news site may need weekly capture. A static project site may need one capture before it goes offline.
  • Permissions. Whether you notify site owners, seek permission, or rely on your institution's legal advice. Policies differ, and you should write yours down.
  • Exclusions. Personal data, login areas and content the owner has asked you not to keep.

Write the scope as if a successor will read it in ten years. They will want to know why a site is in the collection and why another is not.

Quality assurance is the real work

A crawl that finishes without errors is not the same as a good capture. Modern pages load content with JavaScript, pull fonts and video from third-party hosts, and hide navigation behind menus. The crawler may record the HTML and miss everything that makes the page usable.

Quality assurance means opening captures in a replay tool and looking. Check that images render, that internal links resolve inside the archive rather than escaping to the live web, and that embedded media plays. When a page fails, you have a few options. Add missing URLs as new seeds, patch the capture with a browser-based tool, or note the gap in the collection record. Documenting a known gap is honest and useful. Pretending it is complete is neither.

Keep QA proportional. Spot-check every new seed in depth, then sample later crawls. A short checklist helps different staff apply the same standard.

Audience at a small academic symposium
Photo illustration

Description, storage and access

Archived websites need description like any other collection. At minimum, record the seed URL, the capture dates, the scope decisions and any known problems. Many institutions describe web collections at the collection level in their catalog or finding aid system, then rely on the replay interface for item-level browsing. If you already manage archival description, our piece on finding aids, EAD and ArchivesSpace covers how that layer fits together.

Storage follows the same rules as other digital collections. WARC and WACZ files are large but stable. Package them, record checksums, keep more than one copy in more than one place, and check fixity on a schedule. Our guide to BagIt and fixity walks through a simple routine. For shared, geographically distributed copies, the approach behind LOCKSS and distributed preservation applies to web archives as well as to scanned books.

Access is the final step and often the forgotten one. Decide whether captures are public, on-site only, or dark until a set date. Then make the collection findable. A web archive no one can locate is only a backup.

A realistic first project

Pick one thing that is at risk and that clearly belongs to your mission. A retiring faculty member's project site. A local election season. Your own institution's previous website before the next redesign. Write a one-page scope, capture it with the simplest tool that works, review every page, describe it, and store the files with checksums.

That single project will teach you more than a survey of tools. It will also give you a policy, a workflow and a small collection to build on. Web archiving rewards steady, modest effort. The sites you save this year may be the only surviving copies by the end of the decade.

Questions

What is a WARC file?
WARC is the standard file format for web archives, published as ISO 28500. It stores the raw HTTP requests and responses of a capture plus metadata about how the capture was made.
Should a small institution use Archive-It or open source tools?
It depends on staffing. A hosted service like Archive-It saves technical time for a recurring fee. Open tools such as Webrecorder and Heritrix give more control but need staff time to run and maintain.
How do I know a web capture is good enough?
Open it in a replay tool and check it. Confirm that images, links and embedded media work inside the archive, patch gaps where possible, and record any known problems in the collection description.