Research Duquesne Advisory delivers in-depth analyses of Information and Communications Technologies, their implementations and their markets. Research is based on critical observation of the market by the analysts and their on-going contacts with the vendor community, together with hands-on, practical experience in consulting engagements.

Amazon Web Services: protecting privacy with “customer controlled” encryption

Amazon Web Services: protecting privacy with “customer controlled” encryption
Despite persistent concerns about the security of public clouds, most IT professionals will admit that major US providers such as AWS, Microsoft, IBM and Verizon are in fact very good at security, as are European players like Orange and T-Systems.

In the case of the US cloud players, however, being good at security may not be good enough, when a major threat to the privacy of the data of their European customers is perceived - right or wrong - as coming from an increasingly intrusive American government.

Although business clouds would not seem to be the most obvious “target” for US surveillance, this is a very real and very big issue in the privacy sensitive European market. A lot of business is at stake, as demonstrated by the recent decision of the German government to cancel a major Internet services contract with Verizon amid concerns that American companies might be giving data to US authorities.

American carriers and Internet firms have now begun to “push back” against US government demands for customer information. In addition to legal recourse, the option for customers to use powerful encryption technologies under their own control, and in particular those proposed by AWS, is now part of the discussion.

“Push back” from US carriers and Internet firms

The newly installed director of the National Security Agency (NSA), Admiral Michael Rogers, acknowledged in an interview published in the NY Times (June 29, 2014), that the quiet and secretive working relationships between the NSA and American telecommunications and high technology firms had been sharply changed by the Snowden disclosures... US companies now insist that “you are going to have to compel us,” Admiral Rogers said, to turn over data, so that they can demonstrate to foreign customers that they do not voluntarily cooperate.

While all of this is entirely laudable, it is still insufficient to meet the concerns of European customers. No one expects a right to absolute protection from legal investigation, but they do expect that access to their data stored in Europe will be governed by the laws of their own jurisdiction and that they will- at the very least - be informed of any governmental demand for their data.

Two major statutes in the legislative arsenal of the US government can be used to compel cooperation from US service providers while keeping the customer in the dark: the famously controversial Patriot Act which can apply to anything considered to be “foreign intelligence”, and the less well known American Stored Communications Act (SCA), which can apply to any type of federal investigation.

Given this statutory framework, US service providers– assuming they want to keep European business –may have to stand up for their customers in federal courts and some are beginning to do so.

Breach in the “data center location argument”

The “first line of defense” of many big American players (including IBM) has been to claim that customer data stored in their European data centers is not subject to US jurisdiction, since the federal government has no authority to conduct search and seizure outside of US territory.

This was also the argument made by Microsoft in a recent case in a New York federal magistrate court in challenging an SCA warrant for customer e-mails stored in the company’s Dublin data center. (See our analysis on this site: US law in European datacenters: Microsoft in federal court). The Judge, however, rejected the challenge and ordered Microsoft to turn over the content of the Dublin e-mails.

In a groundbreaking ruling, the court maintained that traditional search and seizure protections cannot apply in the interconnected digital world. Since “electronic property” is in reality “just a block of ones and zeroes stored somewhere on somebody else’s computer” that is accessible over “an electronic medium that disregards geographical boundaries”, it should in effect be considered as storedat the headquarters of the service provider. In short, the physical location of the data, for the purposes of US federal law, simply does not matter.

On July 31, 2014, this decision was - not unexpectedly - upheld on first appeal by the federal District Court of the New York Southern District which ruled, in particular, that the magistrate court order was not an extra territorial application of U.S. law. The District Court Judge stayed the ruling to give Microsoft time to appeal to the Second Circuit Court of Appeals.

No one knows how all of this will play out on appeal or how widely this extraordinary new legal doctrine might be used (and even extended) by federal prosecutors, but the ruling as it stands has opened a major breach in the “data center location argument” of American service providers in Europe.

Amazon Web Services and US jurisdiction in Europe

Until recently, Amazon Web Services has not been especially involved (at least not publicly) in the issue of US jurisdiction over data in Europe, presumably since business clouds have been less of a priority for public authorities than e-mail services, social networking or carrier metadata (“who called whom, from where and when”.)This may well change as both public authorities and “persons of interest” come to better understand the cloud.

Unlike some competitors, AWS has never (to our knowledge) publicly claimed that its data centers outside the US were out of the reach of American laws such as the Patriot Act or the SCA. While the company is very discreet about details, it does admit that it receives a variety of requests for cooperation from US federal and other authorities and that the number is trending upwards. As an American company, AWS does say that it must comply with proper US warrants, including those that come with “gag orders”, although it notes that it also receives a certain number of “misdirected” requests.

To take a hypothetical example of a misdirected request, AWS might receive an SCA warrant for e-mails under the control of an e-mail service provider hosted in the Amazon cloud, for example in Dublin. In such a case, it would explain to the federal authorities that – in line with the SCA - the warrant should be served on the e-mail provider who has control of the e-mails rather than on AWS. If the provider is an American company, in the current state of federal law, it would have to comply.

This example raises an interesting point. European Internet service providers (and for that matter, European SaaS providers) are clearly not subject to US jurisdiction in their operations outside of the United States, except of course through an appropriate MLAT (Mutual Legal Assistance Treaty) process involving their own national authorities. Although there is no public case law on the subject, the decision of such providers to be hosted in the European region of the Amazon cloud does not appear to imply a risk of US jurisdiction over the data of their customers.

For IT users considering AWS as a cloud provider in Europe, however, it’s a different story.In our opinion, AWS is keenly aware that the controversy over US jurisdiction and surveillance could handicap its European business development. The company clearly wants to win acceptance by large IT departments in Europe, and it needs to find an answer to the concerns of a good number of potential customers.

Since technology companies tend to believe that the answer to most problems is more technology, it is not entirely surprising that AWS is now busily promoting extremely powerful encryption solutions for customers who want top security – whatever the source of the threat – for sensitive data stored in the Amazon cloud.

“Customer controlled” encryption in the Amazon cloud

Amazon Web Services affirms that security is its top priority and that it invests accordingly. Even so, the level of security depends in large part on the choices made by the individual customer. Since the Amazon cloud is fundamentally a “shared responsibility infrastructure”, the company necessarily takes a “shared responsibility” approach to security: AWS is responsible for the security of the cloud as a whole while customers are responsible for security of what they put into the cloud.

At the individual customer level, the company does provide a wide choice of technological building blocks (together with security advice) and actively promotes what it sees as best practices. Nonetheless, the customer has the ultimate responsibility for choosing and managing things like user identity management, virtualnetwork firewalls, IDS/IPS, application firewalls, and so forth.

An increasingly important issue for customers is whether to encrypt data in the cloud and how to manage the keys. AWS imposes nothing but does see encryption of data in the cloud as an emerging de facto standard. The digital world is a dangerous place, as numerous high profile data breaches – often carried out with remarkable technical sophistication - have amply demonstrated. Most of these breaches have taken place within “on premises” enterprise information systems, but as valuable data moves to the cloud, criminals will presumably “follow the money”. In either case, strong encryption makes stealing data essentially worthless to criminals.

Given the decision to encrypt sensitive data, key management necessarily becomes a process that must be strictly controlled. In this spirit, AWS offers solutions with (what one might call) “customer controlled” encryption, in which the encryption keys are kept totally under the control of the customer and are not accessible even to AWS. However much confidence one might have in AWS internal controls, this option – in line with the venerable principle of “segregation of duties” - has the unquestionable merit of drastically reducing risk from a “service provider insider”.

The AWS team is careful to present these solutions as security “best practice”, intended for customers who want top security – whatever the source of the threat – for sensitive data stored in the cloud. Nonetheless they do not hesitate to point out a major collateral benefit: even in the case of a legal governmental demand, AWS would be completely unable to turn over the encryption keys.

CloudHSM: a high-end service

In March 2013, AWS launched CloudHSM, a serviceallowing customers with very exacting data security requirements to use a dedicated, tamper-resistant Hardware Security Module (HSM) appliance – based on well-respected HSM technology from Safenet - for key storage within the Amazon cloud. Previously, their only option was to store sensitive data – or the encryption keys to it – in their own data centers, making it hard for them to fully migrate applications to the cloud.

The CloudHSM service requires the use of the AWS Virtual Private Cloud (VPC) and the appliances are provisioned as resources inside the user’s VPC.The big technical advantage of this approach is that, since the HSMs are close to the user’s applications, network latency is low and transaction processing is a realistic use case.

All of this doesn’t come cheap and – by AWS standards anyway - the upfront and hourly costs can appear to be high, especially for smaller customers. CloudHSM is clearly a high end service that makes sense primarily for large and sophisticated customers with exacting security requirements. In addition, the use of CloudHSM complicates the picture for Business Continuity. If the customer’s dedicated HSM module was damaged, a good deal of data could be irretrievably lost, unless there are proper precautions for maintaining a duplicate HSM or at least a copy of the keys elsewhere.

On an architectural level, whatever the merits of the Safenet HSM appliances, they were clearly designed primarily for on premises use in an organization’s own data center. The architectural “fit” is less natural in a massive, highly mutualized service provider environment. One can reasonably expect, therefore, that CloudHSM will evolve over time, especially on the hardware side.

S3 Server-Side Encryption with customer managed keys

AWS has for some time also provided various low cost options for the protection of S3 object data, including client-side encryption of data before transmission and server-side encryption with AWS managed keys. In June 2014, AWS announced the enhancement of S3's support for server-side encryption with the possibility for customers to provide and manage their own keys.

The new feature is easily accessible to user applications via the S3 APIs (GET object, PUT object). When the encryption key is supplied as part of a PUT, S3 uses the key to apply AES-256 encryption to the data, computes and stores a key checksum, and then immediately destroys its copy of the key. When the customer supplies the same key as part of a GET, S3 will decrypt the object (after verifying that the checksums match) and return the decrypted object, once again destroying its copy of the key.

The most obvious use case is the secure management of active archives, with large volumes of structured and unstructured data This use case is not as limited as it might sound. Simply “storing stuff” in the cloud – digitized documents, recordings of call center or trading room conversations, archives of internal slide presentations and spreadsheets, old transaction data and so forth – in low cost S3 object storage, with easy access and strong security, can be a good value proposition for many customers.

Of course, it should be kept in mind that some sort of system will presumably be needed for client-side key management. If the customer loses the key, the object is lost for good. Once again, there are business continuity issues to deal with.

“Customer controlled” encryption and jurisdictional risk: can it work?

The first big question, of course, is how this AWS approach to protecting the data (more precisely, the encryption keys) of European customers from US jurisdiction will work out in practice and especially before the federal courts.

A first reaction to this approach might be, to use a British expression, that it could well be seen by public authorities as “too clever by half”. In the case of a hypothetical SCA warrant concerning the AWS European region, a federal judge might well not appreciate the fact that a US service provider had put into place solutions that essentially make it impossible to make practical use of the “customer content” demanded by the warrant. Even so, however annoying this may be for the court, the judge can only apply the law as it is written and there is no provision in the SCA to forbid strong encryption and customer managed keys.

A second and more measured reaction, therefore, is that the AWS approach appears to have a good chance of working in practice. It is hard to deny that “customer controlled“ encryption - in which the customer alone manages the keys - is indeed “best practice” for organizations with exacting requirements who need the highest level of security against a variety of threats.

In addition, it is “settled law” in the United States that any kind of warrant or subpoena can only oblige a person or company to turn over something that it really controls. In the case of the hypothetical SCA warrant, AWS would comply fully by turning over only what it effectively controlled, in other words, the encrypted data but not the keys.

The second big question is whether the approach will work on a commercial level in the European market. The answer to this question is probably: a “qualified yes”.

Of course, some enterprises and public authorities may well decide that it’s just not worth the additional effort and opt instead for private clouds or European based public cloud solutions.

Others, who see important value in what AWS can offer, may readily buy into these solutions, especially if they want best practice security against a variety of threats including, for example, criminal data breaches.

Finally, a large number of customers and prospective customers will simply find these capabilities to be reassuring. Even if they don’t plan to implement them in the short term, the simple fact that customer controlled encryption solutions are available could remove or at least diminish an important obstacle to moving forward with AWS in cloud computing.


The controversy over US jurisdiction over data stored in Europe is an issue that American service providers most heartily wish would just somehow go away. This appears unlikely, unless of course Microsoft finally manages to win its Dublin e-mail case against the US government, in what looks to be a lengthy appeals process in the federal court system.

In the meantime, Amazon Web Services has come up with an ingenious and very “techy” line of defense, based on strong encryption technology with keys under the sole control of the customer. The AWS solutions may not yet be perfectly optimal, but they should at the least provide considerable reassurance to prospective European customers who want access to their data to be governed by the laws of their own national jurisdictions.

Monday, August 4th 2014
Duquesne Advisory
Newsletter To subscribe to the Duquesne Advisory Newsletter, please enter your e-mail address.

Duquesne Advisory

Duquesne Advisory is a European firm, dedicated to researching, understanding and advising clients worldwide on opportunities and trends in Information and Communications technology.


Duquesne Advisory delivers in-depth analyses of Information and Communications Technologies, their implementations and their markets. Research is based on critical observation of the market by the analysts and their on-going contacts with the vendor community, together with hands-on, practical experience in consulting engagements.


The analysts of Duquesne Advisory leverage the Firm’s ongoing market and technology research to undertake high added value consulting engagements for both ICT users and ICT providers. Focused on client service, their approach is rigorous and methodical, and at the same time pragmatic and operational.