Feeds

Hadoop's little buddy Nutch 2.0 gulps down web's big data

Apache projects united

The essential guide to IT transformation

Hadoop daddy Doug Cutting's Nutch, the open-source web-search engine written in Java, has been updated to crawl through piles of big data on the web.

Apache Software Foundation (ASF) has released Nutch 2.0 featuring a data abstraction technique that plugs into big-data stores and frameworks Apache Accumulo, Avro, Cassandra, HBase and, yes, the Hadoop Distributed File System (HDFS).

The abstraction layer that was employed is yet another Apache project, Gora – a framework that provides an in-memory data model and persistence layer for big data.

Gora works with NoSQL column stores, key value stores and document stores, as well as with RDBMSes.

The ASF website where Gora makes its home states its goal as becoming "the standard data representation and persistence framework for big data".

Nutch 2.0 also builds on the Apache open-source search server Soir, which adds a crawler, and a link-graph database with parsing support handled by the Apache Tika project.

Cutting wrote Nutch in 2003 with Mike Cafarella, while the pair were also developing Hadoop – using Google's MapReduce distributed data processing framework to make Hadoop system work at scale. Cutting also wrote Lucene, but it was Hadoop that made his name and he was brought in by Yahoo! to implement the system on its servers.

Nutch has since been somewhat eclipsed by Hadoop, which is used by Amazon.com, Facebook and Yahoo! to name just three web giants. Search engines written using Nutch include Krugle and mozDex. ®

Secure remote control for conventional and virtual desktops

More from The Register

next story
Apple promises to lift Curse of the Drained iPhone 5 Battery
Have you tried turning it off and...? Never mind, here's a replacement
Mozilla's 'Tiles' ads debut in new Firefox nightlies
You can try turning them off and on again
Linux turns 23 and Linus Torvalds celebrates as only he can
No, not with swearing, but by controlling the release cycle
Scratched PC-dispatch patch patched, hatched in batch rematch
Windows security update fixed after triggering blue screens (and screams) of death
This is how I set about making a fortune with my own startup
Would you leave your well-paid job to chase your dream?
prev story

Whitepapers

Endpoint data privacy in the cloud is easier than you think
Innovations in encryption and storage resolve issues of data privacy and key requirements for companies to look for in a solution.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Advanced data protection for your virtualized environments
Find a natural fit for optimizing protection for the often resource-constrained data protection process found in virtual environments.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.