MetaCombine: combining metadata, the web and services
MetaCombine asked how to organise records harvested from many institutions, search them alongside the open web and connect library services that change at different speeds.

Harvesting left three gaps
By 2003 the first wave of harvesting projects had shown that OAI-PMH could move metadata reliably between systems. It had also exposed three problems, and the MetaCombine proposal set out to address each.
- Browsing. Records harvested from many institutions used Dublin Core without shared vocabularies or authority control. Nobody could browse the combined set by subject or author in a meaningful way. Archival collections made this worse, because they are narrow and deep and often use local subject terms that general schemes like LCSH cannot match.
- Two realms. Much of what scholars needed sat on websites that would never expose OAI metadata. Searching harvested records alone missed it.
- Coordination. Library services were multiplying, but there was no good model for joining them into one working system.
The proposal noted that the browsing problem was not unique to Emory. Projects such as the UC San Diego Union Catalog of Art Images had met the same wall.
Pulled the files for “MetaCombine: combining metadata, the web and ser”. Folder names are inconsistent, I will map them before ingest.
Use the same crosswalk as last time. Dates go to ISO 8601, places to the gazetteer.
Once the record is clean we link it from the archive page and the journal piece.
Letting software find the categories
The first experiment applied semantic clustering to aggregated records. The idea was simple: if institutions could not agree on subject headings in advance, perhaps software could group records by meaning after the fact.
Two techniques were chosen. Support vector machines, a family of classification algorithms, were then considered the strongest available method for sorting information into categories automatically. Multidimensional scaling, an older statistical method, places items on a map so that similar items sit close together. The two complemented each other. One assigned categories; the other showed relationships visually.
The plan was to evaluate the available open source SVM tools, pick one or more, and run them over collections from the AmericanSouth and MetaArchive portals. Those collections were typical in lacking any overall classification, so results would be easy to judge.
Today, machine classification of catalogue records is routine. MetaCombine was an early, careful test of whether it could help scholars browse at all.
Searching OAI and the web together
The second line of work tried two ways to give one search across harvested metadata and web content.
The first approach turned web pages into metadata. Semantic clustering would generate records for web resources, and those records would then be exposed through OAI-PMH like any other repository. The second approach went the other way: index web content and harvested metadata together in one search engine.
Both approaches were meant to be built, demonstrated and assessed, not just described. The proposal tied each deliverable to a working system on one or both of the existing scholarly portals, an assessment using several methods, and reports to the profession through conferences and journals.
The question still matters. Discovery services today blend catalogue records, repository content and web indexes. The design choices MetaCombine weighed, between converting everything to records or indexing everything as text, are the same choices a discovery team faces now.

Loosely coupled by design
The third experiment looked at architecture. MetaCombine argued that digital libraries would stay loose assemblies of distributed services for years to come. Tightly integrated systems were not realistic, given how little coordination existed between institutions and how quickly the technology changed.
The answer, the proposal argued, was more lightweight protocols like OAI-PMH that hide each system's internals behind a simple interface. MetaCombine planned to develop and evaluate a framework for coordinating such components, using OAI-PMH and web services as the glue.
Two commitments ran through the whole project. It would use and produce only open source software that other institutions could reuse freely. And it would build on the relationships with scholars that AmericanSouth and MetaArchive had established. Its tools later fed into the Cyberinfrastructure for Humanities work. For background on the founding projects and the portals MetaCombine was meant to improve, see the 2003 proceedings briefing.