RBSE: NASA's Repository-Based Software Engineering Web Spider
In 1994, researchers at NASA's Software Engineering Laboratory at Goddard Space Flight Center presented the RBSE spider, an automated web crawler built in C with Oracle and WAIS back-ends, designed to index and analyse software repositories on the World Wide Web for reuse research.
Background
By the early 1990s, the World Wide Web was growing fast enough that nobody could keep track of it by hand. Most people found pages through word of mouth, manually maintained lists, or early directories that relied on site owners to submit their own links. That worked when the web had a few hundred sites. It stopped working when the number of sites started climbing into the thousands.
The research community had tools for searching text archives before the web arrived. WAIS (Wide Area Information Servers), a protocol for querying and retrieving documents across networked databases, was already in use. But WAIS assumed you knew which server to query. What nobody had built, at least not in a documented and systematic way, was software that would go out and find pages on its own, follow links, and bring back what it found in a form you could query later.
At NASA’s Software Engineering Laboratory at Goddard Space Flight Center, there was a specific problem driving this. The laboratory was involved in Repository-Based Software Engineering, or RBSE, a research programme focused on finding and reusing existing software rather than writing everything from scratch. The web was starting to host software repositories, but nobody had a reliable way to index them systematically. A solution that worked for general web pages would be useful. A solution built specifically for software repositories would be better.
What happened
E. James Whitehead Jr., working at NASA’s Software Engineering Laboratory at Goddard Space Flight Center, and Roy T. Fielding, then at the University of California, Irvine, built the RBSE spider to solve that problem. The spider was written in C. It crawled the web automatically, following links from page to page without human direction, and stored what it found in an Oracle relational database. Queries could then be run against that database through a WAIS interface, allowing researchers to search the indexed content using standard tools of the time.
The architecture kept three things separate: the crawling process that fetched pages, the Oracle back-end that stored them, and the query layer that let users retrieve results. That separation was a practical decision, not a theoretical one. It meant each part could be updated or replaced without rebuilding the whole system. Fielding and Whitehead presented the work at the Second International World Wide Web Conference in 1994.
The spider’s focus was narrow by design. It was not trying to index the whole web. It targeted software repositories specifically, because that was what the RBSE programme needed. The statistical analysis it performed on what it found was meant to support reuse research, helping engineers understand what software was out there and where. A crawler built for a general audience might have looked very different. But the underlying mechanism, automated link-following combined with structured storage and a searchable query interface, was essentially what later, much larger systems would also rely on.
Why it mattered
The RBSE spider was among the earliest documented automated web crawlers, predating most commercial search engines and demonstrating that structured information retrieval could be applied systematically to the nascent World Wide Web. Its design (separating crawling, storage, and query components) anticipated the architecture of later large-scale search infrastructure. By targeting software repositories specifically, it showed that domain-focused crawling could support knowledge management in engineering and scientific organisations.
People
E James Whitehead Jr NASA Goddard Space Flight Center, Roy T Fielding University of California, Irvine
Organisations
NASA Software Engineering Laboratory, Goddard Space Flight Center
Sources
- The Web as a Software Repository.ACM Digital Library, Proceedings of the 2nd International World Wide Web Conference.Primary source
- W3 Search Engines, Early Web Crawler Survey (includes RBSE).World Wide Web Consortium (W3C).institutional
- NASA Technical Reports Server, RBSE-related records.NASA Technical Reports Server (NTRS).Official
Cite this page
AI Achievements. (1994). RBSE: NASA's Repository-Based Software Engineering Web Spider. Retrieved 2026-08-22, from https://achievements.ai/milestone/rbse-first-web-spider-introduced-by-nasa
@misc{achievements_rbse_first_web_spider_introduced_by_nasa,
title = {RBSE: NASA's Repository-Based Software Engineering Web Spider},
author = {{AI Achievements}},
year = {1994},
url = {https://achievements.ai/milestone/rbse-first-web-spider-introduced-by-nasa}
}