Feeds

Google: 'We'll track EVERY task on EVERY data center server'

Chip-level performance tracking in thousand-server Googly clusters

Combat fraud and increase customer satisfaction

Google has wired its worldwide fleet of servers up with monitoring technology that inspects every task running on every machine, and eventually hopes to use this data to selectively throttle or even kill processes that cause disruptions for other tasks running on the same CPU.

The search giant gave details on how it had developed the planet-spanning technology in a technical paper (PDF) due to be published next week – and its contents will be of major interest to anyone running massive Linux-based infrastructure clouds.

"Performance isolation is a key challenge in cloud computing. Unfortunately, Linux has few defenses against performance interference in shared resources such as processor caches and memory buses, so applications in a cloud can experience unpredictable performance caused by other program's behavior," the researchers write.

"Our solution, CPI2, uses cycles-per-instruction (CPI) data obtained by hardware performance counters to identify problems, select the likely perpetrators, and then optionally throttle them so that the victims can return to their expected behavior. It automatically learns normal and anomalous behaviors by aggregating data from multiple tasks in the same job."

In essence, CPI2 lets Google engineers isolate poor performance down to a single task running on a single processor within a cluster of thousands, then drill down to it and select to throttle that task, without causing a CPU overhead of more than 0.1 per cent. It requires no special hardware and its only software dependency appears to be use of Linux.

CPI2 lets Google gather information on the expected CPU cycles-per-instruction (CPI) of any particular task, build standard resource profiles from this data, and then use these profiles to help the web giant identify tasks that are taking more cycles-per-instruction than usual to get executed ("victims") and the tasks that may be causing this disruption ("antagonists"). Software agents can then throttle the antagonists so the victims stop fretting and get back to work (Sounds a bit rough–Ed).

The "vast majority" of Google's machines run multiple tasks, the company wrote. These jobs are either latency-sensitive or batch-based, and are themselves typically composed of multiple tasks. 96 per cent of the tasks running on Google servers are part of a job with at least 10 tasks, and 87 per cent of the tasks are part of a job with 100 or more tasks.

But these tasks can interfere with each other through foul-ups relating to processor caches and memory allocation problems, causing the latency of a task within an application to skyrocket – something that ad-slinger Google wants to avoid at all costs.

To help it control latency spikes on each processor on a task-by-task basis, Google has rolled out CPI monitoring across all of its production servers. It derives its CPI data by measuring processor hardware counters, according to CPU_CLK_UNHALTED.REF divided by INSTRUCTIONS_RETIRED.

Google gathers this data for a 10 second period once every minute via a perf_event tool in counting mode, rather than sampling mode. Total CPU overhead of the system is less than 0.1% and leads to no visible latency impact.

CPI2 lets Google inspect every task on every chip

Don't worry 'victim', the agents will save you!

CPI is calculated to take account of variances in the type of CPU being run on, Google said, as typical clusters will run across a large range of platforms. CPI values are measured and analyzed locally by an agent that runs on every machine (pictured). This agent is always given the most current expected CPI distribution for the jobs it is running tasks for, so that it can spot aberrations without having to phone home.

If this agent spots a "victim" task that is being slowed, it looks once per second for the "antagonist" tasks disrupting it. It uses an algorithm to do this that lets it work out whether there's a relationship between an antagonist's increasing use of CPU and a "victim" task spiking, in terms of the cycles needed per instruction.

If it identifies an antagonist and finds that it is a batch job, then the system will "forcibly reduce the antagonist's CPU usage by applying CPU hard-capping".

It is more than likely that this agent is given tasks from a component of Google's mega-infrastructure-management system named Omega, especially when you consider that the architect of Omega (computer science luminary John Wilkes) also happens to be one of the co-authors of the CPI2 paper.

The CPI2 system can even automatically throttle disruptive tasks, but only if they have been marked as being eligible for what the search giant terms "hard-capping" – something that not many tasks are tagged as. Alternately, system operators can interface with CPI and can manually hard-cap suspects.

CPI data is logged and stored offline, along with profiles of antagonist tasks, so admins can query it via Google's major internal analysis tool, Dremel (of which Cloudera's Impala is a public implementation).

Dremel is used by Google engineers for performance forensics to let them identify particularly aggressive antagonists for their tasks. In the future, it could be possible to reschedule antagonists to different machines and put the most disruptive ones on their own subset of machines, then feed this configuration pattern to the scheduler to avoid problems.

One area for improvement is dealing with multiple antagonists, as this currently confuses the algorithm. Another is to introduce a feedback-driven policy for capping tasks, and expanding the CPI2 method to deal with disk and I/O conflicts as well.

"Even before these enhancements are applied, we believe that CPI2 is a powerful, useful tool," the researchers wrote.

CPI2 looks to be a less consumptive way of getting workable info on app performance than other Google schemes. There's a parallel technique called "Google-Wide Profiling" that tracks both hardware and software performance, but is only in use on a fraction of Google machines due to concerns about performance.

From the mile-high perspective of Vulture West, CPI2 provides an illuminating look at the types of invention needed to not only manage massive IT estates, but make them work better. Next time you perform a search, or check your email, or look up an address via Google services and notice the page takes a bit longer to load than normal, you might spare a thought for the antagonist that is now being seen to by a cold, heartless CPI2 agent. ®

Combat fraud and increase customer satisfaction

More from The Register

next story
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Kingston DataTraveler MicroDuo: Turn your phone into a 72GB beast
USB-usiness in the front, micro-USB party in the back
Dropbox defends fantastically badly timed Condoleezza Rice appointment
'Nothing is going to change with Dr. Rice's appointment,' file sharer promises
BOFH: Oh DO tell us what you think. *CLICK*
$%%&amp Oh dear, we've been cut *CLICK* Well hello *CLICK* You're breaking up...
AMD's 'Seattle' 64-bit ARM server chips now sampling, set to launch in late 2014
But they won't appear in SeaMicro Fabric Compute Systems anytime soon
Amazon reveals its Google-killing 'R3' server instances
A mega-memory instance that never forgets
Cisco reps flog Whiptail's Invicta arrays against EMC and Pure
Storage reseller report reveals who's selling what
Microsoft builds teleporter weapon to send VMware into Azure
Updated Virtual Machine Converter now converts Linux VMs too
prev story

Whitepapers

Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Top three mobile application threats
Learn about three of the top mobile application security threats facing businesses today and recommendations on how to mitigate the risk.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.