Project · 2003–2005

MetaCombine: combining metadata, the web and services

MetaCombine asked how to organise records harvested from many institutions, search them alongside the open web and connect library services that change at different speeds.

ProposalJuly 2003
FunderAndrew W. Mellon Foundation
ProtocolOAI-PMH
MethodsSVM and MDS
MetaScholar — Project · 2003–2005
record.xmlnotes.mdsources.txt
All collectionsSymposia06Projects07Photographs02Documents06Journal20
Project · 2003–2005
oai_dcindexedneeds review
1<record status="active">
2 <dc:relation>/metacombine.html</dc:relation>
3</record>
Record
ProposalJuly 2003
FunderAndrew W. Mellon Foundation
ProtocolOAI-PMH
MethodsSVM and MDS
Publish

Harvesting left three gaps

By 2003 the first wave of harvesting projects had shown that OAI-PMH could move metadata reliably between systems. It had also exposed three problems, and the MetaCombine proposal set out to address each.

  • Browsing. Records harvested from many institutions used Dublin Core without shared vocabularies or authority control. Nobody could browse the combined set by subject or author in a meaningful way. Archival collections made this worse, because they are narrow and deep and often use local subject terms that general schemes like LCSH cannot match.
  • Two realms. Much of what scholars needed sat on websites that would never expose OAI metadata. Searching harvested records alone missed it.
  • Coordination. Library services were multiplying, but there was no good model for joining them into one working system.

The proposal noted that the browsing problem was not unique to Emory. Projects such as the UC San Diego Union Catalog of Art Images had met the same wall.

#collections — thread
AR
Archivist

Pulled the files for “MetaCombine: combining metadata, the web and ser”. Folder names are inconsistent, I will map them before ingest.

ML
Metadata librarian

Use the same crosswalk as last time. Dates go to ISO 8601, places to the gazetteer.

crosswalk-v3.xsl · 42 rules
ED
MetaScholar editor

Once the record is clean we link it from the archive page and the journal piece.

Letting software find the categories

The first experiment applied semantic clustering to aggregated records. The idea was simple: if institutions could not agree on subject headings in advance, perhaps software could group records by meaning after the fact.

Two techniques were chosen. Support vector machines, a family of classification algorithms, were then considered the strongest available method for sorting information into categories automatically. Multidimensional scaling, an older statistical method, places items on a map so that similar items sit close together. The two complemented each other. One assigned categories; the other showed relationships visually.

The plan was to evaluate the available open source SVM tools, pick one or more, and run them over collections from the AmericanSouth and MetaArchive portals. Those collections were typical in lacking any overall classification, so results would be easy to judge.

Today, machine classification of catalogue records is routine. MetaCombine was an early, careful test of whether it could help scholars browse at all.

MetaScholar — Archive
All collectionsSymposia06Projects07Photographs02Documents06Journal20
Project · 2003–2005
oai_dcindexedneeds review
1<record status="active">
2 <dc:relation>/metacombine.html</dc:relation>
3</record>
Record
ProposalJuly 2003
FunderAndrew W. Mellon Foundation
ProtocolOAI-PMH
MethodsSVM and MDS
Publish

Loosely coupled by design

The third experiment looked at architecture. MetaCombine argued that digital libraries would stay loose assemblies of distributed services for years to come. Tightly integrated systems were not realistic, given how little coordination existed between institutions and how quickly the technology changed.

The answer, the proposal argued, was more lightweight protocols like OAI-PMH that hide each system's internals behind a simple interface. MetaCombine planned to develop and evaluate a framework for coordinating such components, using OAI-PMH and web services as the glue.

Two commitments ran through the whole project. It would use and produce only open source software that other institutions could reuse freely. And it would build on the relationships with scholars that AmericanSouth and MetaArchive had established. Its tools later fed into the Cyberinfrastructure for Humanities work. For background on the founding projects and the portals MetaCombine was meant to improve, see the 2003 proceedings briefing.

preservation nodeverified copyreplication path