Feeds

Insecure indexing risk dissected

How did THAT get out?

  • alert
  • submit to reddit

Choosing a cloud hosting partner with confidence

It's embarrassing when future PR items, upcoming security advisories or boilerplates for obituaries that are not meant to be visible to external users drift into the public domain. These documents might get accidentally uploaded to the wrong part of a website but mischievous attacks can also play a role.

Web application security researcher Amit Klein this week published a paper explaining how "insecure indexing" allows attackers to expose hidden files on web servers. Some site-installed search engines index files that search engines are programmed to ignore. Typically search engines look in a root domain for a special file called "robots.txt" which tells the robot (spider) which files it may download.

If an attacker can get to internal search engines he can get around files denied to him by the Robots Exclusion Standard. Klein explains that these attacks are "fundamentally different from exploiting external (remote) search engines".

Klein explains various attack techniques, ranging all the way from guessing a file name from names that already exist to targeted search strings and far more complicated traffic-intensive attacks, and concludes with methods for detecting insecure indexing and suggested defences. "Crawling style indexing should be preferred over direct file indexing. If file-level indexing cannot be avoided, more consideration should be made when deploying a search engine that facilitates it. In particular those search engines should be systematically limited to the visible resources (or at the very least, to accessible resources)," he writes.

The paper - Insecure Indexing Vulnerability: Attacks Against Local Search Engines - can be found on the Web Application Security Consortium's site here. ®

Related stories

Botnets strangle Google Adwords campaigns
Phishers suspected of eBay Germany domain hijack
Interview with a link spammer
Google's No-Google tag blesses the Balkanized web
Google exposes web surveillance cams
Major flaw found in Google Desktop

Beginner's guide to SSL certificates

More from The Register

next story
FYI: OS X Yosemite's Spotlight tells Apple EVERYTHING you're looking for
It's on by default – didn't you read the small print?
Russian hackers exploit 'Sandworm' bug 'to spy on NATO, EU PCs'
Fix imminent from Microsoft for Vista, Server 2008, other stuff
Edward who? GCHQ boss dodges Snowden topic during last speech
UK spies would rather 'walk' than do 'mass surveillance'
Microsoft pulls another dodgy patch
Redmond makes a hash of hashing add-on
'LulzSec leader Aush0k' found to be naughty boy not worthy of jail
15 months home detention leaves egg on feds' faces as they grab for more power
China is ALREADY spying on Apple iCloud users, claims watchdog
Attack harvests users' info at iPhone 6 launch
Carders punch holes through Staples
Investigation launched into East Coast stores
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Cloud and hybrid-cloud data protection for VMware
Learn how quick and easy it is to configure backups and perform restores for VMware environments.
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.