Original URL: https://www.theregister.com/2011/05/25/the_once_and_future_mongodb/
MongoDB daddy: My baby beats Google BigTable
The web is built on objects. Not tables
Posted in SaaS, 25th May 2011 19:41 GMT
After a decade as chief technology officer at DoubleClick – the internet ad giant he cofounded in 1995 – Dwight Merriman set out to build a "platform cloud" along the lines of Google App Engine or Microsoft Azure. But this was before people called them platform clouds, before anyone knew about App Engine or Azure.
It was late 2007, and the idea was to create an online service for developing, hosting, and automatically scaling web applications. "What we were building was very similar to what App Engine eventually became," Merriman tells us. But unlike App Engine, the service would be underpinned by an entirely open source software stack, and somewhere along the way Merriman and his team realized that no open source database platform was suited to such a service.
"We felt like a lot of existing databases didn't really have the 'cloud computing' principles you want them to have: elasticity, scalability, and ... easy administration, but also ease of use for developers and operators," Merriman says. "[MySQL] doesn't have all those properties."
So they set out to build a database of their own, one that would discard the familiar relational database model in favor of a distributed platform tailored for moden-day web applications. "By reducing transactional semantics, we could still solve an interesting set of problems, but we could also scale," Merriman explains. After a year of work, the database was in place, and they decided it had as much potential as the cloud service it was designed for – if not more. The cloud service was never finished. But the database was open sourced as MongoDB.
Dwight Merriman and his team, including ShopWiki founder Eliot Horowitz, built MongoDB under the aegis of the New York–based startup 10gen, and the company now offers support, training, and consulting services for the database in addition to serving as the open source project's primary steward. This week, 10gen held its second annual San Francisco developer conference, and with his Tuesday-morning keynote, Merriman described the origins of MongoDB and explained why the database was built the way it was.
The split from the relational model was essential, Merriman says, because you can't do distributed joins in a way that readily scales. "I'm not smart enough to do distributed joins that scale horizontally, widely, and are super fast. You have to choose something else," Merriman explained during his talk. "We have no choice but to not be relational."
It was equally important, he says, to limit the database's transactional semantics. "You can do distributed transactions, but if you do them with no loss of generality and you do them across a thousand machines, it's not going to be that fast."
Get off the Table
10gen is hardly alone in splitting from MySQL's path – so-called NoSQL databases are the hype du jour – but Merriman argues that MongoDB is better suited than many to running modern day "cloud" applications because it uses a document-oriented data model. Mapping object-oriented code to a relational database is "painful work," Merriman says, and he argues that you have much the same problem with tabular NoSQL databases such as Google's proprietary BigTable, which underpins App Engine, and BigTable's open source doppelgänger, HBase
"Today, 95 per cent of the code we're writing is in an object-oriented language," he says. "We're to the point where object-oriented programming is ubiquitous enough, having a database that works well with that sort of thing is important." There are certain advantages to a tabular model – it's closer to what the average database programmer is used to – but you still have to map from objects in programs to tables.
MongoDB's document-oriented model is based on JSON, the lightweight JavaScript-based data exchange format. This setup not only plays nicely with object-oriented code, it ensures that your data is kept separate from your code. "It fits very elegantly with object-oriented programming models, but it's not too tightly coupled. If you're using Java and Python and Scala, and you're talking to Mongo in all three of those, the data is easily manipulated from all three of those," Merriman told The Register after his talk.
"JSON gives us a standards-based, language-independent way to store object-style data. So we're not storing methods. We're not storing class hierarchies. We're not storing the object 'graph' of your app. But we are storing the documents that are data of objects, including a good amount of embedding and nesting and lists. So the documents are pretty rich."
Speaking with The Register, Merriman acknowledged that MongoDB isn't the only NoSQL that uses a document-oriented model. CouchDB is the other obvious example. And he made it clear that MongoDB and other NoSQL database can't completely replace relational databases. They're not suited to complex transactions, and ad hoc queries aren't as easy. But MongoDB can scale, he says, and it can still handle ad hoc queries, and it offers strong consistency.
"We want Mongo to be a full-featured product, but it's never going to be quite as full-featured as a traditional RDMS. We're always going to leave a few things out so we can scale," Merriman says. Mongo may be suited to some tasks behind the firewall, but it is particularly suited for services in the cloud. Like the one it was originally built for. Today, MongoDB can be used in tandem with the new platform clouds from VMware (Cloud Foundry) and Red Hat (OpenShift), and there's a plug-in for the Ruby on Rails cloud from Heroku. "We want Mongo to be used in many, many platform-as-a-service stacks," Merriman says.
You can't run it with App Engine. Google's platform is necessarily tied to the company's own BigTable-based back-end infrastructure. Apparently, MongoDB owes a debt to BigTable. Merriman indicates that 10gen's distributed data–sharing setup is based on Google's famous BigTable research paper. But data sharing aside, MongoDB is very much a different animal.
With the introduction of Megastore – the high-availability layer built atop BigTable – Google's setup is closer to Mongo. "We read [Google's Megastore research paper] and we were almost laughing at the similarities," Merriman says. But mapping object-oriented code to a tabular database like BigTable, he argues, still lacks a certain elegance. And we would add that MongoDB is open source. ®