Feeds

Sphinx - text search The Pirate Bay way

Like MySQL. But it can scale

Choosing a cloud hosting partner with confidence

Vowels are a scare resource

Before Sphinx, the other option for text search was Apache Solr. Solr, whose name-giver understands that vowels are a scarce resource and must be used sparingly, is a server that sits on top of Lucene. Solr is popular with the enterprise crowd, who love its Java. Being a Java program, Solr includes no shortage of technology whose acronyms contain the letters J and X.

This tickles the enterprise pink, because these sorts of developers love nothing more than hanging out around a whiteboard drawing boxes and arrows and, from time to time, writing XML to make it look like they're doing real work. Solr thrives in this environment, being an Apache Foundation project, the Apache Foundation, of course, widely known as a cruel experiment to see what happens when bureaucrats do open source.

Being backed by Lucene, Solr is bound by Lucene. It is easier to tune Solr for relevancy than it is to tune MySQL, but it still doesn't provide the search quality to justify its popularity. That, and Solr has scalability problems. Lucene was originally designed as an embeddable library, so scalability wasn't a top priority. As a network layer on top of Lucene, Solr must accommodate for this. "In my internal benchmarks, Solr routinely performed 2x-4x slower [than Sphinx], both for indexing and searching," Aksyonoff tells The Reg.

Sphinx takes a different approach to search. For example, using MySQL or Solr, each query will be trimmed of its stopwords. This is a method out of the information retrieval books, where you remove low-signal words (like "the" or "and") that can degrade search quality. This was one of Aksyonoff's original problems searching song lyrics, a query like "I feel you" would yield Cuil-like results. By default, Sphinx does not remove stopwords (but it can be configured to) and ranks results based on phrase matching.

Having its roots in database search, Sphinx speaks MySQL, which means you can connect to its search server with a MySQL client and issue queries that look very similar to SQL. It will easily index data out of a MySQL or Postgres databases, but also supports an XML format for arbitrary data. There are also clients for PHP, Python, Ruby, and any other language that's worth using. Scalability is no issue, both in searching and indexing: Sphinx can index 10 megabytes of data per second and can search up to 100 gigabytes of text on a single processor. It also supports multi-machine distributed searching, as in the case of Craigslist.

In the vicious meritocracy of open source software, the only yardstick of success is the distribution of your code. In this regard, Sphinx still has some way to go, but there's still profit to be made. High-volume installations at Craigslist and The Pirate Bay are a validation for Aksyonoff, who now makes his living on Sphinx.

"We aren't exactly bathing in money, but on the other hand, we're paying the salaries from our revenues instead of burning venture capital and have always been," he says. His company, Sphinx Technologies, provides consulting and support services, as well as custom development. Judging by how well Sphinx is performing in real-world installations, for Aksyonoff, the General Public Business Model might actually pay off.

You can download Sphinx here. ®

Ted Dziuba is a co-founder at Milo.com You can read his regular Reg column, Fail and You, every other Monday.

Secure remote control for conventional and virtual desktops

More from The Register

next story
ONE MILLION people already running Windows 10
A third of them are doing it in VMs, but early feedback focuses on frippery
Netscape Navigator - the browser that started it all - turns 20
It was 20 years ago today, Marc Andreeesen taught the band to play
Sway: Microsoft's new Office app doesn't have an Undo function
Content aggregation, meet the workplace ... oh
Sign off my IT project or I’ll PHONE your MUM
Honestly, it’s a piece of piss
Do Moan! MONSTER 6-day EMAIL OUTAGE hits Domain Monster
Customers freaked out by frightful service
Return of the Jedi – Apache reclaims web server crown
.london, .hamburg and .公司 - that's .com in Chinese - storm the web server charts
NetWare sales revive in China thanks to that man Snowden
If it ain't Microsoft, it's in fashion behind the Great Firewall
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Win a year’s supply of chocolate
There is no techie angle to this competition so we're not going to pretend there is, but everyone loves chocolate so who cares.
Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Saudi Petroleum chooses Tegile storage solution
A storage solution that addresses company growth and performance for business-critical applications of caseware archive and search along with other key operational systems.