Feeds

Sphinx - text search The Pirate Bay way

Like MySQL. But it can scale

Choosing a cloud hosting partner with confidence

In 2001, Andrew Aksyonoff had an itch. He ran a website that indexed song lyrics and allowed users to search them. At the time, none of the open source options for text search worked. Lucene, the standard response at an engineering team meeting when the problem of search comes up, was still a baby. Indexing took too long, and it was a bear to install on anything other than a Lucene developer's machine.

Mnogosearch - the search engine you probably still haven't heard of - was too bloated and took a full twenty-four hours to index Aksyonoff's meager 150 megabytes of data. On top of all this, the standard algorithms for text search were about as useful as a venereal disease when it came to searching song lyrics. Andrew had a different idea of how text search should work - and the spare time to try it out. He did, and today, his software powers the search engines behind Craigslist, The Pirate Bay, and many more.

It's called Sphinx, and it's on track to become the open source world's canonical answer to the question of text search. MySQL and Solr, the two popular solutions, are showing their age. MySQL introduced full-text search in late 2000 as a way to more intelligently search blobs of text stored in databases. You can work a full-text clause into a query, and MySQL will rank the result rows by how relevant it thinks they are to the query.

MySQL uses textbook search algorithms and doesn't allow for a lot of relevance tuning. It's like a drawing from a five year old: The heart is in the right place, but everybody knows that kids suck at drawing. Implementation details aside, MySQL still suffers from scalability problems. Having ignored the trend of chip manufacturers to build multiple cores into CPUs, hoping that this unpleasant trend that required them to actually think about multi-threading would just blow over sooner or later, MySQL's ability to handle parallelism is, well, see the five year old's drawing.

Internet-famous MySQL wonk Jeremy Zawodny, who had the foresight to jump from the ship's bow as Yahoo started to take on water, replaced MySQL fulltext search at Craigslist with Sphinx. Craigslist used 25 machines to handle roughly 50 million queries per day on MySQL. Under that kind of load, Zawodny found that MySQL wasn't using much CPU or doing much disk I/O, which means it's spending all of its time waiting on thread locks. Oops.

Maybe we should have paid attention to parallelism after all. The Sphinx implementation took those 25 machines down to 10, with plenty of room to grow. While Sphinx didn't handle the traffic out of the box at the time, Zawodny was able to patch it to handle Craigslist's specific need - and fix a few bugs along the way.

Secure remote control for conventional and virtual desktops

More from The Register

next story
ONE MILLION people already running Windows 10
A third of them are doing it in VMs, but early feedback focuses on frippery
Sign off my IT project or I’ll PHONE your MUM
Honestly, it’s a piece of piss
Netscape Navigator - the browser that started it all - turns 20
It was 20 years ago today, Marc Andreeesen taught the band to play
Torvalds CONFESSES: 'I'm pretty good at alienating devs'
Admits to 'a metric ****load' of mistakes during work with Linux collaborators
Sway: Microsoft's new Office app doesn't have an Undo function
Content aggregation, meet the workplace ... oh
Do Moan! MONSTER 6-day EMAIL OUTAGE hits Domain Monster
Customers freaked out by frightful service
Ploppr: The #VultureTRENDING App of the Now
This organic crowd sourced viro- social fertiliser just got REAL
Return of the Jedi – Apache reclaims web server crown
.london, .hamburg and .公司 - that's .com in Chinese - storm the web server charts
NetWare sales revive in China thanks to that man Snowden
If it ain't Microsoft, it's in fashion behind the Great Firewall
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Win a year’s supply of chocolate
There is no techie angle to this competition so we're not going to pretend there is, but everyone loves chocolate so who cares.
Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Saudi Petroleum chooses Tegile storage solution
A storage solution that addresses company growth and performance for business-critical applications of caseware archive and search along with other key operational systems.