Authors:
Published:
In February, the Library Innovation Lab announced its archive of the federal data clearinghouse Data.gov. Today, we’re pleased to share Data.gov Archive Search, an interface for exploring this important collection of government datasets. Our work builds on recent advancements in lightweight, browser-based querying to enable discovery of more than 311,000 datasets comprising some 17.9 terabytes of data on topics ranging from automotive recalls to chronic disease indicators.
Traditionally, supporting search across massive collections has required investment in dedicated computing infrastructure, such as a server running a database or search index. In recent years, innovative tools and methods for client-side querying have opened a new path. With these technologies, users can execute fast queries over large volumes of static data using only a web browser.
This interface joins a host of recent efforts not only to preserve government data, but also to make it accessible in independent interfaces. The recently released Data Rescue Project Portal offers metadata-level search of the more than 1,000 datasets it has preserved. Most of these datasets live in DataLumos, the archive for valuable government data resources maintained by the University of Michigan’s Institute for Social Research.
LIL has chosen Source Cooperative as the ideal repository for its Data.gov archive for a number of reasons. Built on cloud object storage, the repository supports direct publication of massive datasets, making it easy to share the data in its entirety or as discrete objects. Additionally, LIL has used the Library of Congress standard for the transfer of digital files. The “BagIt” principles of archiving ensure that each object is digitally signed and retains detailed metadata for authenticity and provenance. Our hope is that these additional steps will make it easier for researchers and the public to cite and access the information they need over time.
In the coming month, we will continue our work, fine-tuning the interface and incorporating feedback. We also continue to explore various modes of access to large government datasets, and so we are exploring, for example, how we might create greater access to the 710 TB of Smithsonian collections data we recently copied. Please be in touch with questions or feedback.