Feeds

Sphinx - text search The Pirate Bay way

Like MySQL. But it can scale

Providing a secure and efficient Helpdesk

Vowels are a scare resource

Before Sphinx, the other option for text search was Apache Solr. Solr, whose name-giver understands that vowels are a scarce resource and must be used sparingly, is a server that sits on top of Lucene. Solr is popular with the enterprise crowd, who love its Java. Being a Java program, Solr includes no shortage of technology whose acronyms contain the letters J and X.

This tickles the enterprise pink, because these sorts of developers love nothing more than hanging out around a whiteboard drawing boxes and arrows and, from time to time, writing XML to make it look like they're doing real work. Solr thrives in this environment, being an Apache Foundation project, the Apache Foundation, of course, widely known as a cruel experiment to see what happens when bureaucrats do open source.

Being backed by Lucene, Solr is bound by Lucene. It is easier to tune Solr for relevancy than it is to tune MySQL, but it still doesn't provide the search quality to justify its popularity. That, and Solr has scalability problems. Lucene was originally designed as an embeddable library, so scalability wasn't a top priority. As a network layer on top of Lucene, Solr must accommodate for this. "In my internal benchmarks, Solr routinely performed 2x-4x slower [than Sphinx], both for indexing and searching," Aksyonoff tells The Reg.

Sphinx takes a different approach to search. For example, using MySQL or Solr, each query will be trimmed of its stopwords. This is a method out of the information retrieval books, where you remove low-signal words (like "the" or "and") that can degrade search quality. This was one of Aksyonoff's original problems searching song lyrics, a query like "I feel you" would yield Cuil-like results. By default, Sphinx does not remove stopwords (but it can be configured to) and ranks results based on phrase matching.

Having its roots in database search, Sphinx speaks MySQL, which means you can connect to its search server with a MySQL client and issue queries that look very similar to SQL. It will easily index data out of a MySQL or Postgres databases, but also supports an XML format for arbitrary data. There are also clients for PHP, Python, Ruby, and any other language that's worth using. Scalability is no issue, both in searching and indexing: Sphinx can index 10 megabytes of data per second and can search up to 100 gigabytes of text on a single processor. It also supports multi-machine distributed searching, as in the case of Craigslist.

In the vicious meritocracy of open source software, the only yardstick of success is the distribution of your code. In this regard, Sphinx still has some way to go, but there's still profit to be made. High-volume installations at Craigslist and The Pirate Bay are a validation for Aksyonoff, who now makes his living on Sphinx.

"We aren't exactly bathing in money, but on the other hand, we're paying the salaries from our revenues instead of burning venture capital and have always been," he says. His company, Sphinx Technologies, provides consulting and support services, as well as custom development. Judging by how well Sphinx is performing in real-world installations, for Aksyonoff, the General Public Business Model might actually pay off.

You can download Sphinx here. ®

Ted Dziuba is a co-founder at Milo.com You can read his regular Reg column, Fail and You, every other Monday.

Internet Security Threat Report 2014

More from The Register

next story
UNIX greybeards threaten Debian fork over systemd plan
'Veteran Unix Admins' fear desktop emphasis is betraying open source
Netscape Navigator - the browser that started it all - turns 20
It was 20 years ago today, Marc Andreeesen taught the band to play
Redmond top man Satya Nadella: 'Microsoft LOVES Linux'
Open-source 'love' fairly runneth over at cloud event
Chrome 38's new HTML tag support makes fatties FIT and SKINNIER
First browser to protect networks' bandwith using official spec
Admins! Never mind POODLE, there're NEW OpenSSL bugs to splat
Four new patches for open-source crypto libraries
Torvalds CONFESSES: 'I'm pretty good at alienating devs'
Admits to 'a metric ****load' of mistakes during work with Linux collaborators
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.