Feeds

Sphinx - text search The Pirate Bay way

Like MySQL. But it can scale

The Power of One Brief: Top reasons to choose HP BladeSystem

Vowels are a scare resource

Before Sphinx, the other option for text search was Apache Solr. Solr, whose name-giver understands that vowels are a scarce resource and must be used sparingly, is a server that sits on top of Lucene. Solr is popular with the enterprise crowd, who love its Java. Being a Java program, Solr includes no shortage of technology whose acronyms contain the letters J and X.

This tickles the enterprise pink, because these sorts of developers love nothing more than hanging out around a whiteboard drawing boxes and arrows and, from time to time, writing XML to make it look like they're doing real work. Solr thrives in this environment, being an Apache Foundation project, the Apache Foundation, of course, widely known as a cruel experiment to see what happens when bureaucrats do open source.

Being backed by Lucene, Solr is bound by Lucene. It is easier to tune Solr for relevancy than it is to tune MySQL, but it still doesn't provide the search quality to justify its popularity. That, and Solr has scalability problems. Lucene was originally designed as an embeddable library, so scalability wasn't a top priority. As a network layer on top of Lucene, Solr must accommodate for this. "In my internal benchmarks, Solr routinely performed 2x-4x slower [than Sphinx], both for indexing and searching," Aksyonoff tells The Reg.

Sphinx takes a different approach to search. For example, using MySQL or Solr, each query will be trimmed of its stopwords. This is a method out of the information retrieval books, where you remove low-signal words (like "the" or "and") that can degrade search quality. This was one of Aksyonoff's original problems searching song lyrics, a query like "I feel you" would yield Cuil-like results. By default, Sphinx does not remove stopwords (but it can be configured to) and ranks results based on phrase matching.

Having its roots in database search, Sphinx speaks MySQL, which means you can connect to its search server with a MySQL client and issue queries that look very similar to SQL. It will easily index data out of a MySQL or Postgres databases, but also supports an XML format for arbitrary data. There are also clients for PHP, Python, Ruby, and any other language that's worth using. Scalability is no issue, both in searching and indexing: Sphinx can index 10 megabytes of data per second and can search up to 100 gigabytes of text on a single processor. It also supports multi-machine distributed searching, as in the case of Craigslist.

In the vicious meritocracy of open source software, the only yardstick of success is the distribution of your code. In this regard, Sphinx still has some way to go, but there's still profit to be made. High-volume installations at Craigslist and The Pirate Bay are a validation for Aksyonoff, who now makes his living on Sphinx.

"We aren't exactly bathing in money, but on the other hand, we're paying the salaries from our revenues instead of burning venture capital and have always been," he says. His company, Sphinx Technologies, provides consulting and support services, as well as custom development. Judging by how well Sphinx is performing in real-world installations, for Aksyonoff, the General Public Business Model might actually pay off.

You can download Sphinx here. ®

Ted Dziuba is a co-founder at Milo.com You can read his regular Reg column, Fail and You, every other Monday.

Seven Steps to Software Security

More from The Register

next story
Secure microkernel that uses maths to be 'bug free' goes open source
Hacker-repelling, drone-protecting code will soon be yours to tweak as you see fit
KDE releases ice-cream coloured Plasma 5 just in time for summer
Melty but refreshing - popular rival to Mint's Cinnamon's still a work in progress
NO MORE ALL CAPS and other pleasures of Visual Studio 14
Unpicking a packed preview that breaks down ASP.NET
Cheer up, Nokia fans. It can start making mobes again in 18 months
The real winner of the Nokia sale is *drumroll* ... Nokia
Put down that Oracle database patch: It could cost $23,000 per CPU
On-by-default INMEMORY tech a boon for developers ... as long as they can afford it
Another day, another Firefox: Version 31 is upon us ALREADY
Web devs, Mozilla really wants you to like this one
Google shows off new Chrome OS look
Athena springs full-grown from Chromium project's head
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Securing Web Applications Made Simple and Scalable
Learn how automated security testing can provide a simple and scalable way to protect your web applications.