Analysis On Friday, President Obama gave his long-awaited speech on plans to reform the activities of the US intelligence services and how they monitor the rest of the world.

You can watch the entire speech here, but words are tricky things – never more so than when national security is involved. As such we've taken a transcript of the president's words and, given what we know about today's mass surveillance operations, tried to work out what was actually said. Prez Obama's speech is presented below in bold, with our annotations throughout.

First, a history lesson from the President

At the dawn of our Republic, a small, secret surveillance committee borne out of the "The Sons of Liberty" was established in Boston. And the group's members included Paul Revere. At night, they would patrol the streets, reporting back any signs that the British were preparing raids against America's early Patriots.

It's fair to say that if the British had the capabilities of the NSA today, there wouldn’t have been an American revolution and the citizens of the North American continent would be sipping warm beer and spelling color with a 'u' along with the rest of Anglo-Saxon society.

The British wouldn't have needed to monitor content of the letters sent by Paul Revere and others, just tracked his movements, examined the metadata of his associates, and then swooped. Revere and others would have been up a tree with a hemp necktie for carrying out acts of terrorism against a national government, since these "Sons of Liberty" weren't above violence when it came to furthering their aims.

U.S. intelligence agencies were anchored in a system of checks and balances – with oversight from elected leaders, and protections for ordinary citizens. Meanwhile, totalitarian states like East Germany offered a cautionary tale of what could happen when vast, unchecked surveillance turned citizens into informers, and persecuted people for what they said in the privacy of their own homes.

The US has always had some checks and balances, to be sure. Whether or not they have always been followed, however, is another question entirely (see the history of J. Edgar Hoover for more details). If they had been, it's probable that Obama's Friday schedule would not have included this speech.

The Stasi example is also an unfortunate one to pick. The reports that the US was spying on not only its European friends, but also on the private phone lines of other governments' leaders, led to accusations that the NSA had taken a leaf out of the Stasi's playbook – and is doing a much more thorough job of it than the East Germans ever did.

How we got here

The horror of September 11th brought all these issues to the fore. Across the political spectrum, Americans recognized that we had to adapt to a world in which a bomb could be built in a basement, and our electric grid could be shut down by operators an ocean away. We were shaken by the signs we had missed leading up to the attacks – how the hijackers had made phone calls to known extremists and traveled to suspicious places. So we demanded that our intelligence community improve its capabilities, and that law enforcement change practices to focus more on preventing attacks before they happen than prosecuting terrorists after an attack.

The intelligence community did receive a drubbing in the wake of the attacks on September 11, 2001, and there were serious failings, although elected politicians should also shoulder a fair measure of blame.

But the 9/11 Commission and others have pointed out that the information to detect the attacks was out there – the problem was that the intelligence agencies weren't sharing that data with each other. Since then, it seems, little has changed: two amateur teenagers were able to pull off the Boston Marathon bombing last year despite the massive collection facilities of the NSA.

Relationships with foreign intelligence services have expanded, and our capacity to repel cyber-attacks have been strengthened. And taken together, these efforts have prevented multiple attacks and saved innocent lives – not just here in the United States, but around the globe.

Intelligence certainly has saved lives, but the mass-monitoring program instituted hasn't had that much success.

When the Snowden scandal broke, General Keith Alexander claimed that more than 50 attacks had been stopped by his agency, in the US and overseas. This number has been steadily reduced as the months have progressed, and a detailed report from the nonprofit think tank New America Foundation found 17 plots had been stopped, and only one by the US spying on its own citizens.

Too long, didn't watch: Trust us, we're the NSA

I maintained a healthy skepticism toward our surveillance programs after I became President. I ordered that our programs be reviewed by my national security team and our lawyers, and in some cases I ordered changes in how we did business. We increased oversight and auditing, including new structures aimed at compliance. Improved rules were proposed by the government and approved by the Foreign Intelligence Surveillance Court. And we sought to keep Congress continually updated on these activities.

There have been some improvements made in this area, but, as the Snowden documents have shown, the NSA and others have been equally adept at finding out new ways to get around them. Documents declassified by the Foreign Intelligence Surveillance Court show even the judges involved felt the NSA was lying to them and prevaricating.

As for keeping Congress informed, the US Director of National Intelligence James Clapper was caught out lying to his Congressional overlords in a pretty barefaced manner. Senator Ron Wyden asked Clapper whether or not the NSA is collecting data on US citizens – a question he had given to Clapper 24 hours before so that he could consider his reply.

Clapper response was a simple "No," and it was only after the existence of the mass-collection of phone metadata was revealed by whistleblower Edward Snowden months later that Clapper was forced to explain that is all depends on how you use the term "collect".

In an extraordinarily difficult job – one in which actions are second-guessed, success is unreported, and failure can be catastrophic – the men and women of the intelligence community, including the NSA, consistently follow protocols designed to protect the privacy of ordinary people. They're not abusing authorities in order to listen to your private phone calls or read your emails.

That last sentence is factually untrue. NSA inspector general Dr. George Ellard reports that some agents routinely abused their surveillance capabilities; going as far as to use government data to spy on people they were interested in wooing – so-called "loveint".

Examples of this include eavesdropping on phones of potential paramours or checking into their communications background before going on a date. General Alexander has confirmed this, but says it only happens once a year on average.

The NSA has also refused to deny that it is using its surveillance powers on member of Congress charged with overseeing the agency. Such monitoring has happened before, and many are worried that it's also happening now. When Obama was a senator, his communications were under NSA surveillance, according to one agency whistleblower.

Reasons for change

I indicated in a speech at the National Defense University last May that we needed a more robust public discussion about the balance between security and liberty. Of course, what I did not know at the time is that within weeks of my speech, an avalanche of unauthorized disclosures would spark controversies at home and abroad that have continued to this day.

Take a look at that speech from last year, just before the Snowden leaks started appearing. In more than 6,000 words, the President devotes just two short paragraphs to the topic of domestic surveillance: he said the US would have to "keep working hard to strike the appropriate balance between our need for security and preserving those freedoms that make us who we are. That means reviewing the authorities of law enforcement, so we can intercept new types of communication, but also build in privacy protections to prevent abuse."

Thus, we argue that today's NSA "reforms" are not the result of a speech given nearly a year ago – but instead triggered by the bespectacled elephant in the room who made the activities of the US and UK spying agencies public since June.

I'm not going to dwell on Mr Snowden's actions or his motivations; I will say that our nation's defense depends in part on the fidelity of those entrusted with our nation's secrets. If any individual who objects to government policy can take it into their own hands to publicly disclose classified information, then we will not be able to keep our people safe, or conduct foreign policy. Moreover, the sensational way in which these disclosures have come out has often shed more heat than light, while revealing methods to our adversaries that could impact our operations in ways that we may not fully understand for years to come.

Ex-NSA contractor Snowden has stated that he undertook his decision to go public, and to leak a trove of Uncle Sam's spying documents, because he had no other choice: his attempts to raise concerns about the surveillance programs through the usual official channels had been ignored, it's claimed. Based on the experience of other NSA whistleblowers, fleeing to Hong Kong (and then Russia) with the evidence he needed wasn't such a bad idea.

Take, for example, the case of William Binney, a 30-year NSA veteran who rose to the rank of Technical Director. In 2002, Binney and two associates played it by the book and complained through formal channels that the NSA's activities were unconstitutional. As a result he was arrested at gunpoint, had his business shut down, and was only cleared of wrongdoing after a five-year legal battle.

Snowden's not an idiot. He gave up a six-figure salary, a girlfriend, and a posting in the not-unpleasant surroundings of Hawaii to right what he sees are wrongs being committed by his former employers. The documents that have been published under Snowden's guidance have been carefully redacted, and most (although by no means all) have avoided revealing specific activities against direct enemies of the US.

Given the unique power of the state, it is not enough for leaders to say: Trust us, we won't abuse the data we collect. For history has too many examples when that trust has been breached. Our system of government is built on the premise that our liberty cannot depend on the good intentions of those in power; it depends on the law to constrain those in power.

This is Snowden's point, and one that the rest of the world seems to be coming around to. The defense of civil liberties depends on the rule of law, and Snowden and others contend that this hasn't been the case of late.

Let's get to the meat of the reforms

First, I have approved a new presidential directive for our signals intelligence activities both at home and abroad. This guidance will strengthen executive branch oversight of our intelligence activities. It will ensure that we take into account our security requirements, but also our alliances; our trade and investment relationships, including the concerns of American companies; and our commitment to privacy and basic liberties. And we will review decisions about intelligence priorities and sensitive targets on an annual basis so that our actions are regularly scrutinized by my senior national security team.

Increased oversight is certainly needed, although the details of the presidential directive aren't yet known. Certainly Obama seems to be saying that the new regimen will fix some of the problems being detailed, and companies like Cisco, whose profits have been hit by the perception that their products are easily accessible by NSA agents, will breathe easier.

We will reform programs and procedures in place to provide greater transparency to our surveillance activities, and fortify the safeguards that protect the privacy of U.S. persons. Since we began this review, including information being released today, we have declassified over 40 opinions and orders of the Foreign Intelligence Surveillance Court, which provides judicial review of some of our most sensitive intelligence activities – including the Section 702 program targeting foreign individuals overseas, and the Section 215 telephone metadata program.

The declassification of such documents is very welcome news. What we've seen so far has been partially reassuring and, in parts, worrying. The secret NSA oversight court has complained that judges are being lied to, and that the US G-men aren't answering questions properly – and that's just in the opinions released.

Opening up secret courts

Going forward, I'm directing the Director of National Intelligence, in consultation with the Attorney General, to annually review for the purposes of declassification any future opinions of the court with broad privacy implications, and to report to me and to Congress on these efforts. To ensure that the court hears a broader range of privacy perspectives, I am also calling on Congress to authorize the establishment of a panel of advocates from outside government to provide an independent voice in significant cases before the Foreign Intelligence Surveillance Court.

Given Clapper's record as Director of National Intelligence, the news he'll be overseeing the distribution of declassified documents doesn't exactly inspire confidence, but the inclusion of the Attorney General is a welcome step.

Creating a panel of advocates for the Foreign Intelligence Surveillance Court is a very good move, and one that privacy groups have been calling for. Such advocates still have to be picked however, and it's going to be interesting to see who gets the final say on who is chosen. But the phrase "significant cases" does raise the question as to how often these advocates will actually be called.

We will provide additional protections for activities conducted under Section 702, which allows the government to intercept the communications of foreign targets overseas who have information that's important for our national security. Specifically, I am asking the Attorney General and DNI to institute reforms that place additional restrictions on government's ability to retain, search, and use in criminal cases communications between Americans and foreign citizens incidentally collected under Section 702.

Reform of Section 702, or even a clarification as to how far it can be used, is certainly welcome. But when you reread the statement, you see there's no solid commitment to reform, and it could be that nothing meaningful will change, particularly with the DNI in the driving seat.

I have directed the Attorney General to amend how we use national security letters so that this secrecy will not be indefinite, so that it will terminate within a fixed time unless the government demonstrates a real need for further secrecy. We will also enable communications providers to make public more information than ever before about the orders that they have received to provide data to the government.

Google, Microsoft, Yahoo!, and others will be pleased to hear this. Ever since Google started doing transparency reports (and others in the industry followed them), there has been a lot of interest from consumers and businesses in the results.

Companies looking to do business internationally have faced a very tricky situation ever since Snowden started singing. Cloud vendors in the US complain that they are treated like pariahs when it comes to selling their services overseas. The reform will go some way to easing those concerns.

Managing mass slurping of metadata

This brings me to the program that has generated the most controversy these past few months – the bulk collection of telephone records under Section 215. Let me repeat what I said when this story first broke: This program does not involve the content of phone calls, or the names of people making calls. Instead, it provides a record of phone numbers and the times and lengths of calls – metadata that can be queried if and when we have a reasonable suspicion that a particular number is linked to a terrorist organization.

That the NSA doesn't listen to the content of calls has been central to the arguments of pro-NSA types. But you could argue that phone numbers (for most of us, at least) are directly personally identifiable and the amount of data you can get from numbers, call times and dates, and other metadata records makes its collection highly intrusive. You don't need to know what was said in a call, in other words; just knowing who is talking to whom and when and where can be enough to discern your intentions.

The mass hoarding of this metadata isn't being done in a targeted way against suspected terrorists, nor is it solely being analyzed on a strict suspect-by-suspect basis, if Snowden's documents are to be believed.

The telephone metadata program under Section 215 was designed to map the communications of terrorists so we can see who they may be in contact with as quickly as possible. And this capability could also prove valuable in a crisis. For example, if a bomb goes off in one of our cities and law enforcement is racing to determine whether a network is poised to conduct additional attacks, time is of the essence. Being able to quickly review phone connections to assess whether a network exists is critical to that effort.

We call this the Jack Bauer defense. Such metadata would be very useful, provided you know which number called the mobile phone that set off the first explosive, but the cellphone-activated bomb cliché is a poor one to choose. In such situations it would be easy to grab this data directly from the phone company with a single request. It shows mass collection has more to do with convenience than immediate need – and at least one judge agrees.

Having said that, I believe critics are right to point out that without proper safeguards, this type of program could be used to yield more information about our private lives, and open the door to more intrusive bulk collection programs in the future. They're also right to point out that although the telephone bulk collection program was subject to oversight by the Foreign Intelligence Surveillance Court and has been reauthorized repeatedly by Congress, it has never been subject to vigorous public debate.

Or, in fact, any public debate at all until Snowden started leaking. Such talk of mass snooping was dismissed as the province of conspiracy theorists and tinfoil hat–sporting nutjobs. At the Black Hat hacker conference two years ago, merely mentioning to a former FBI director the possibility of the NSA spying on Americans on US soil, let alone every foreigner abroad, sparked a tirade of abuse at this correspondent.

For all these reasons, I believe we need a new approach. I am therefore ordering a transition that will end the Section 215 bulk metadata program as it currently exists, and establish a mechanism that preserves the capabilities we need without the government holding this bulk metadata. The review group recommended that our current approach be replaced by one in which the providers or a third party retain the bulk records, with government accessing information as needed.

In other words, the mass collection of metadata will continue but the method of storage will change.

Of the two ideas floated, keeping this data in the hands of the telcos looks to be the preferable one: they are already required it hold anyway, and it would be comparatively simple to arrange access for the intelligence services. How long that data is stored, and who pays the cost of doing so, could well be a sticking point, however. This assumes the telcos have kept their systems secure, and will be able to do so in future.

Creating a third-party organization to handle this data, while possible, would be expensive and cumbersome. The organization would have to be set up, have really good security, and the owners of the repository would have to be carefully vetted and screened.

Because of the challenges involved, I've ordered that the transition away from the existing program will proceed in two steps. Effective immediately, we will only pursue phone calls that are two steps removed from a number associated with a terrorist organization instead of the current three. And I have directed the Attorney General to work with the Foreign Intelligence Surveillance Court [FISC] so that during this transition period, the database can be queried only after a judicial finding or in the case of a true emergency.

The change to a two-step rule is a slight improvement, but still one that leaves a sufficiently wide dragnet to build up accurate mapping of social connections. But adding the need for a FISC thumbs-up is a major improvement.

That said, the FISC is notorious for not turning down requests for investigations – but at least there's some oversight involved, as opposed to the current situation where it's a free-for-all. Including the "true emergency" codicil will allow action in the unlikely event of a ticking time-bomb situation, should one ever arise.

Maintaining the highest standard

The new presidential directive that I've issued today will clearly prescribe what we do, and do not do, when it comes to our overseas surveillance. To begin with, the directive makes clear that the United States only uses signals intelligence for legitimate national security purposes, and not for the purpose of indiscriminately reviewing the emails or phone calls of ordinary folks.

No doubt this will come as a relief to the citizens of the UK, France, Germany, Brazil, and others who have had huge amounts of their communications data slurped by the NSA in the past. However, it will take some time before those "ordinary folks" take the US at its word, given the abuses of the past.

In this directive, I have taken the unprecedented step of extending certain protections that we have for the American people to people overseas. I've directed the DNI, in consultation with the Attorney General, to develop these safeguards, which will limit the duration that we can hold personal information, while also restricting the use of this information.

This has the potential to be very welcome news indeed. What has been striking about Congressional attempts to strengthen data protection in the light of the Snowden leaks is that any additional safeguards have only applied to US citizens, and giving the rest of the world some privacy is welcome.

I have made clear to the intelligence community that unless there is a compelling national security purpose, we will not monitor the communications of heads of state and government of our close friends and allies. And I've instructed my national security team, as well as the intelligence community, to work with foreign counterparts to deepen our coordination and cooperation in ways that rebuild trust going forward.

So if you're the head of a friendly government you should be feeling a little more secure about using your personal mobile phone. Then again, the US defines who is friendly and who isn’t, and there's that "nation security" caveat again.

I have also asked my counselor, John Podesta, to lead a comprehensive review of big data and privacy. And this group will consist of government officials who, along with the President's Council of Advisors on Science and Technology, will reach out to privacy experts, technologists and business leaders, and look how the challenges inherent in big data are being confronted by both the public and private sectors.

This debate is sorely needed, and needs to include a variety of competing specialists. No doubt the government will have its say, but the input of businesses, security experts, and privacy specialists is going to be key.

No one expects China to have an open debate about their surveillance programs, or Russia to take privacy concerns of citizens in other places into account. But let's remember: We are held to a different standard precisely because we have been at the forefront of defending personal privacy and human dignity.

Agreed. No one expects the Chinese government not to be doing this stuff because it's largely unaccountable and has shown a willingness to play fast and loose with the rules in the past. Similarly, Putin's Russia is hardly a haven of openness and democracy.

There are reasons why the US is held to a higher standard. Firstly, it claims the position and has, in the past, been a valuable force for maintaining human rights and freedoms. But it also has effective control of many parts of the digital world and with that comes a certain amount of responsibility to do things right, and to be seen to be doing so.

For more than two centuries, our Constitution has weathered every type of change because we have been willing to defend it, and because we have been willing to question the actions that have been taken in its defense. Today is no different. I believe we can meet high expectations. Together, let us chart a way forward that secures the life of our nation while preserving the liberties that make our nation worth fighting for.

We hope so. We really, really do. ®

Related stories

• Spooks add Uber algorithms to their research wish-list (12 August 2015)

  https://www.theregister.com/2015/08/12/spooks_add_uber_algorithms_to_their_research_wishlist/

• Senator: Backdoor for the Feds is a backdoor for hackers (15 December 2014)

  https://www.theregister.com/2014/12/15/us_senator_to_congress_close_all_the_backdoors/

• CIA snoops snooped on Senate to spy spy torture report – report (5 March 2014)

  https://www.theregister.com/2014/03/05/cia_senate_watchdog_spying_row/

• Yahoo! Mail! users! change! your! passwords! NOW! (31 January 2014)

  https://www.theregister.com/2014/01/31/yahoo_mail_users_change_your_password_now/

• Snowden speaks: NSA spies create 'databases of ruin' on innocent folks (24 January 2014)

  https://www.theregister.com/2014/01/24/snowden_speaks_nsa_whistleblower_calls_for_global_privacy_standards/

• US govt watchdog slams NSA snooping as illegal, useless against terrorism (23 January 2014)

  https://www.theregister.com/2014/01/23/us_government_watchdog_report_finds_nsa_spying_is_illegal_and_useless/

• Obama reveals tiny NSA reforms ... aka reforming your view of the NSA (17 January 2014)

  https://www.theregister.com/2014/01/17/obama_promises_limited_reforms_of_nsa_powers_against_ordinary_americans/

• NSA: It's TRUE, we grab 200 MILLION of your text messages A DAY globally (16 January 2014)

  https://www.theregister.com/2014/01/16/snowden_leak_shows_nsa_collects_tiny_fraction_of_sms_messages/

• Campaign to kick NSA man from crypto standards group fails (8 January 2014)

  https://www.theregister.com/2014/01/08/nsa_bod_crypto_standard_co_chair_controversy/

• NSA refuses to deny spying on members of Congress (6 January 2014)

  https://www.theregister.com/2014/01/06/nsa_refuses_to_deny_spying_on_members_of_congress/

• Snowden docs: NSA building encryption-cracking quantum computer (3 January 2014)

  https://www.theregister.com/2014/01/03/snowden_docs_show_nsa_building_encryptioncracking_quantum_system/

• Blame Silicon Valley for the NSA's data slurp... and what to do about it (31 December 2013)

  https://www.theregister.com/2013/12/31/blame_silicon_valley_for_the_nsas_data_slurp_and_what_to_do_about_it/