Project · 2004–2006 · IMLS

What makes a search result good

Emory and Virginia Tech set out to measure what different groups of users actually value when a metasearch system ranks results, and to turn those preferences into a model.

Years2004–2006
PartnersEmory, Virginia Tech
FunderIMLS
FocusMetasearch ranking
MetaScholar — Project · 2004–2006 · IMLS
record.xmlnotes.mdsources.txt
All collectionsSymposia06Projects07Photographs02Documents06Journal20
Project · 2004–2006 · IMLS
oai_dcindexedneeds review
1<record status="active">
2 <dc:relation>/quality_metrics/</dc:relation>
3</record>
Record
Years2004–2006
PartnersEmory, Virginia Tech
FunderIMLS
FocusMetasearch ranking
Publish

The question behind the study

Metasearch systems pull results from many sources and present them in one list. Someone has to decide the order. In most systems of the time, that decision was made by developers, using ranking formulas built for general web search or for a single catalogue.

The Study of User Quality Metrics asked whether those formulas matched what users wanted. A historian searching archival collections might care most about the authority of the holding institution. A student might care about whether full text is available. A scientist might weight recency above everything else. Different users bring different assumptions about what a good result is, and they rarely state them.

The project was a collaboration between the MetaScholar Initiative at Emory and Virginia Tech, with support from the Institute of Museum and Library Services. It used production digital library services rather than laboratory setups, so that the findings would reflect real searching by real users.

The aim was to replace guesswork about relevance with evidence.

#collections — thread
AR
Archivist

Pulled the files for “What makes a search result good”. Folder names are inconsistent, I will map them before ingest.

ML
Metadata librarian

Use the same crosswalk as last time. Dates go to ISO 8601, places to the gazetteer.

crosswalk-v3.xsl · 42 rules
ED
MetaScholar editor

Once the record is clean we link it from the archive page and the journal piece.

A testbed with adjustable ranking

To study preferences, the team needed a search system whose ranking could be changed on purpose. They took existing open source search engines and extended them with new algorithms and interfaces. The additions allowed custom ranking metrics based on the attributes of individual resources and of whole collections.

With that testbed in place, the researchers could show users results ranked in different ways, using different retrieval algorithms and different weightings of quality factors. They could then record how users reacted. Did they prefer one ordering over another? Did they click deeper into the list? Did certain groups respond differently?

The users' responses were to be analysed statistically. From that analysis the team planned to build a theoretical model of user quality metrics for metasearch systems of various kinds.

This approach, varying one ranking factor at a time and measuring behaviour, is close to what search teams now call online experimentation. In a library context in the mid-2000s, it was a fairly unusual way to work.

quality metrics · sample collection
findability
82%
completeness
67%
consistency
74%
rights clarity
58%
link health
91%

Three planned outcomes

The project set out three outcomes, aimed at both researchers and practitioners:

  1. Empirical data. Evidence about user preferences, expectations and other quality metrics for metasearch systems, published openly as a dataset.
  2. A theoretical model. A model of user quality metrics that fits the empirical data, so that the findings could be applied beyond the systems tested.
  3. Open source software. A digital library search engine that accepts explicit ranking algorithms for metasearch, based on the model and released for other institutions to use and study.

Results were to be shared through project reports, articles and presentations. The project's website was intended as the central place for those documents.

The combination is worth noting. Data, theory and software were planned together, so that each could check the others. Many research projects produce only one of the three, which makes their results harder to reuse.

record.xml — before / after enrichment
Harvested
titleWhat makes a search result good
date2004–2006
creator—
subjectmisc
rightsempty
Enriched
dc:titleWhat makes a search result good
dc:date2004–2006
dc:typeProject
dc:subjectDigital libraries
dc:rightsRights statement attached
premisfixity sha256 ✓

Why it still matters

Relevance ranking in library discovery systems is still often opaque. Vendors tune their algorithms, and librarians and users have limited insight into why results appear in a given order. The question this study asked, what different users count as quality, has not been settled.

The study's main lesson for current practice is methodological. Ranking choices should be tested with the people who use the system, and different communities may need different weightings. A single formula for everyone is a design decision, not a neutral default.

The project also connects to earlier MetaScholar work. MetaCombine explored how to combine harvested metadata and web content into one search. Once results come from many sources, ranking becomes the next hard problem. This study took that problem on directly.

Measuring before deciding is the habit it tried to build. For the programme's other outputs, see the documents library.

A graduate student working at a library carrel
Photo illustration