BackRub Web Crawler Introduces Link-Based Page Ranking at Stanford
In March 1996, Larry Page, a PhD student at Stanford University, launched the BackRub web crawler to analyse the backlink structure of the web, laying the algorithmic foundation for what would become Google's PageRank system.
Background
By the mid-1990s, finding anything useful on the web was genuinely difficult. Search engines existed: AltaVista, Excite and WebCrawler could index millions of pages, but they ranked results almost entirely by looking at the words on the page itself. A page that repeated a search term many times would float to the top, whether it was authoritative or not. People gamed this constantly, stuffing pages with invisible or repeated keywords to climb the rankings.
The web was also growing faster than anyone had anticipated. Each new site added more pages and more links, and the engines of the time had no reliable way to tell a trusted source from a worthless one. Keyword frequency was cheap to measure, but it said almost nothing about whether a page was actually worth reading. Researchers knew this was a problem. Nobody had a working solution at scale.
What happened
In March 1996, Larry Page, then a PhD student at Stanford University, started crawling the web from his own Stanford homepage. The project was called BackRub. Its starting point was a question that sounds obvious in retrospect: if a page has many other pages linking to it, and those pages are themselves well-linked, does that tell you something real about its quality? Page thought it did, and he built a crawler to test the idea.
BackRub worked by following hyperlinks outward from a starting page and recording which pages pointed back to which. This produced a map of the web’s link structure, and from that map Page could calculate a score for each page based on how many inbound links it had and how authoritative those linking pages were. The score was computed iteratively, meaning a link from a highly-ranked page counted for more than a link from an obscure one. Page and Sergey Brin, a fellow Stanford PhD student who joined the work, described the method in detail in their 1998 paper “The Anatomy of a Large-Scale Hypertextual Web Search Engine”, published through the Stanford InfoLab.
The system they built to search this ranked index they named Google. At the time it was a research prototype running on Stanford’s network, not a public product. But the link-based ranking it used was demonstrably better at surfacing relevant results than the keyword-frequency approaches everywhere else. The gap was large enough that even in its prototype form, the system attracted attention well beyond Stanford’s computer science department.
Why it mattered
BackRub demonstrated that the number and quality of inbound hyperlinks to a page could serve as a reliable proxy for its authority, replacing keyword-frequency heuristics that dominated earlier search engines. This insight produced PageRank, which transformed information retrieval at web scale and remains central to how large search systems rank results. The project directly seeded Google Inc., making it one of the most consequential research prototypes in the history of computing and applied AI.
People
Organisations
Sources
- The Anatomy of a Large-Scale Hypertextual Web Search Engine.Stanford University InfoLab.Primary source
- Stanford and the Google Algorithm.IEEE Annals of the History of Computing.Secondary
- Google's Roots in Stanford Research.Stanford University Libraries.institutional
Cite this page
AI Achievements. (1996). BackRub Web Crawler Introduces Link-Based Page Ranking at Stanford. Retrieved 2026-08-22, from https://achievements.ai/milestone/backrub-started-indexing-backlinks-of-webpages
@misc{achievements_backrub_started_indexing_backlinks_of_webpages,
title = {BackRub Web Crawler Introduces Link-Based Page Ranking at Stanford},
author = {{AI Achievements}},
year = {1996},
url = {https://achievements.ai/milestone/backrub-started-indexing-backlinks-of-webpages}
}