Spinn3r was founded in late 2005 by web crawler and RSS
expert Kevin Burton. Mr. Burton is a serial
entrepreneur and sold his previous
to Six Apart in late 2006.
Spinn3r was originally built to power Tailrank, a real-time blog analysis
and topical relevance index which launched in early 2006. Tailrank monitored
and ranked live conversations across the web. Tailrank's algorithms correlate
and present the issues on the mind of the world's publishers every day to an
audience interested in breaking news, trends, and other influencing
The architecture behind Spinn3r was influenced by two large projects. One
was Rojo, which had a 500GB-1TB search index. The other was NewsMonster, one of the first
and still the most advanced client side aggregator, with a high performance
crawler integrated at its core.
Spinn3r was launched in August 2007 as a dedicated product after having
numerous requests to license its backend infrastructure.
Since launching, Spinn3r has been consistently adopted by new startups
needing access to the blogosphere. Spinner is now providing crawl
infrastructure for startups that have raised in excess of $100M in VC funding, as
well as dozens of universities and hundreds of researchers.
We present the backend architecture behind Spinn3r - our scalable web and blog
crawler. Most existing work in scaling MySQL has been around high read
throughput environments similar to web applications. In contrast, at Spinn3r
we needed to complete thousands of write transactions per second in order to
index the blogosphere at full speed.
The blogosphere is a noisy place filled with many
interconnected conversations on all sorts of disparate
topics. Tailrank, which calls itself a memetracker, is a
service that tracks the zeitgeist of conversations in the
blogosphere. I sat down with Kevin Burton, Tailrank's CEO and
founder, and Jonathan Moore, brilliant engineer and hacker, at
their home office to learn a bit about their
services. Tailrank is built upon Spinn3r, another service that
Burton and Moore built, that spiders and indexes the
Kevin Burton is a talented developer who has worked on a
variety of startups already including Rojo, and now TailRank
which he started to be able to see what bloggers were talking
about. Here I sit down with him for an interesting
conversation in the lobby of San Francisco's Palace Hotel.