Feeds

Sphinx - text search The Pirate Bay way

Like MySQL. But it can scale

Maximizing your infrastructure through virtualization

Vowels are a scare resource

Before Sphinx, the other option for text search was Apache Solr. Solr, whose name-giver understands that vowels are a scarce resource and must be used sparingly, is a server that sits on top of Lucene. Solr is popular with the enterprise crowd, who love its Java. Being a Java program, Solr includes no shortage of technology whose acronyms contain the letters J and X.

This tickles the enterprise pink, because these sorts of developers love nothing more than hanging out around a whiteboard drawing boxes and arrows and, from time to time, writing XML to make it look like they're doing real work. Solr thrives in this environment, being an Apache Foundation project, the Apache Foundation, of course, widely known as a cruel experiment to see what happens when bureaucrats do open source.

Being backed by Lucene, Solr is bound by Lucene. It is easier to tune Solr for relevancy than it is to tune MySQL, but it still doesn't provide the search quality to justify its popularity. That, and Solr has scalability problems. Lucene was originally designed as an embeddable library, so scalability wasn't a top priority. As a network layer on top of Lucene, Solr must accommodate for this. "In my internal benchmarks, Solr routinely performed 2x-4x slower [than Sphinx], both for indexing and searching," Aksyonoff tells The Reg.

Sphinx takes a different approach to search. For example, using MySQL or Solr, each query will be trimmed of its stopwords. This is a method out of the information retrieval books, where you remove low-signal words (like "the" or "and") that can degrade search quality. This was one of Aksyonoff's original problems searching song lyrics, a query like "I feel you" would yield Cuil-like results. By default, Sphinx does not remove stopwords (but it can be configured to) and ranks results based on phrase matching.

Having its roots in database search, Sphinx speaks MySQL, which means you can connect to its search server with a MySQL client and issue queries that look very similar to SQL. It will easily index data out of a MySQL or Postgres databases, but also supports an XML format for arbitrary data. There are also clients for PHP, Python, Ruby, and any other language that's worth using. Scalability is no issue, both in searching and indexing: Sphinx can index 10 megabytes of data per second and can search up to 100 gigabytes of text on a single processor. It also supports multi-machine distributed searching, as in the case of Craigslist.

In the vicious meritocracy of open source software, the only yardstick of success is the distribution of your code. In this regard, Sphinx still has some way to go, but there's still profit to be made. High-volume installations at Craigslist and The Pirate Bay are a validation for Aksyonoff, who now makes his living on Sphinx.

"We aren't exactly bathing in money, but on the other hand, we're paying the salaries from our revenues instead of burning venture capital and have always been," he says. His company, Sphinx Technologies, provides consulting and support services, as well as custom development. Judging by how well Sphinx is performing in real-world installations, for Aksyonoff, the General Public Business Model might actually pay off.

You can download Sphinx here. ®

Ted Dziuba is a co-founder at Milo.com You can read his regular Reg column, Fail and You, every other Monday.

Reducing security risks from open source software

More from The Register

next story
HIDDEN packet sniffer spy tech in MILLIONS of iPhones, iPads – expert
Don't panic though – Apple's backdoor is not wide open to all, guru tells us
Do YOU work at Microsoft? Um. Are you SURE about that?
Nokia and marketing types first to get the bullet, says report
Microsoft takes on Chromebook with low-cost Windows laptops
Redmond's chief salesman: We're taking 'hard' decisions
Cheer up, Nokia fans. It can start making mobes again in 18 months
The real winner of the Nokia sale is *drumroll* ... Nokia
EU dons gloves, pokes Google's deals with Android mobe makers
El Reg cops a squint at investigatory letters
Chrome browser has been DRAINING PC batteries for YEARS
Google is only now fixing ancient, energy-sapping bug
Big Blue Apple: IBM to sell iPads, iPhones to enterprises
iOS/2 gear loaded with apps for big biz ... uh oh BlackBerry
prev story

Whitepapers

Seven Steps to Software Security
Seven practical steps you can begin to take today to secure your applications and prevent the damages a successful cyber-attack can cause.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.