Spinn3r
Company Overview
Spinn3r was founded in late 2005 by web crawler and RSS expert Kevin Burton. Mr. Burton is a serial entrepreneur and sold his previous company, Rojo, to Six Apart in late 2006.
Spinn3r was originally built to power Tailrank, a real-time blog analysis and topical relevance index which launched in early 2006. Tailrank monitored and ranked live conversations across the web. Tailrank's algorithms correlate and present the issues on the mind of the world's publishers every day to an audience interested in breaking news, trends, and other influencing factors.
The architecture behind Spinn3r was influenced by two large projects. One was Rojo, which had a 500GB-1TB search index. The other was NewsMonster, one of the first and still the most advanced client side aggregator, with a high performance crawler integrated at its core.
Spinn3r was launched in August 2007 as a dedicated product after having numerous requests to license its backend infrastructure.
Since launching, Spinn3r has been consistently adopted by new startups needing access to the blogosphere. Spinner is now providing crawl infrastructure for startups that have raised in excess of $100M in VC funding, as well as dozens of universities and hundreds of researchers.
Conferences
Our talk on the Spinn3r web crawler architecture entitled "Scaling MySQL and Java in High Write Throughput Environments" was presented at the 2008 MySQL Conference.
We present the backend architecture behind Spinn3r - our scalable web and blog crawler. Most existing work in scaling MySQL has been around high read throughput environments similar to web applications. In contrast, at Spinn3r we needed to complete thousands of write transactions per second in order to index the blogosphere at full speed.
In the Media
LunchMeetIn July 2007 Tailrank was featured on LunchMeet where we had a chance to talk about Spinn3r:
|
|
The Scoble ShowIn December 2006, Tailrank was featured on the Scoble Show.
|
|