Feel secure with SSL? Think again.

Recently, we’ve heard a lot of discussion about the trust we place in public binary repositories. For example, Maven Central, a popular legacy repository maintained by Sonatype, was recently compromised by a successful MITM attack. In response, Sonatype set up an https access to central (removing the demand for a $10 donation to the Apache Foundation for using SSL). This has no impact on millions of maven installations in the world, which will continue access Maven Central via http unless manually reconfigured, but the interesting question is – is that enough?

While SSL is important in guaranteeing the integrity of downloaded files, it doesn’t say anything about the integrity of the files in the repository itself. The only(!) verification mechanism in the repository itself that Maven Central and other legacy repositories suggest you blindly trust, is the set of signature files uploaded with the files. Interestingly enough, modern repositories such as RubyGems.org, npm-registry and Bintray don’t force you to sign your files at all. Let’s try to understand why.

Here’s tl;dr if you need one.

Is SSL access enough for us to feel secure?

To answer this question, let’s consider Maven Central. This is a repository that works with SSL and “secures” the files with PGP signatures. In theory, these signature files strongly identify the signer (assuming that both the jar files and the signatures are served over SSL). But do they really?

Let’s see how it works:

  • The author uses a gpg tool to generate a keypair for identification.
  • The author then uploads the public key to a trusted key server (one of MIT, SKS OpenPGP Public Key Server or PGP Global Directory).
  • Anyone who wants to download a package and verify its integrity runs a signature check against one of the servers, and gets the unique and verified identity of the author.

Is it so?

Let’s run a couple of experiments, and see how securely the “secured” Maven Central is really maintained.

Here’s one of Sonatype’s latests jars: nexus-core-2.8.1-01.

Let’s download it see what’s the signature says, after importing the signer’s public key:

>gpg nexus-core-2.8.1-01.jar.asc
 gpg: Signature made 05/27/14 18:00:54 Central Daylight Time using DSA key ID 8DD1BDFD
 gpg: Good signature from "Sonatype, Inc. (Sonatype release key) <dev@sonatype.com>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.
 Primary key fingerprint: 2BCB DD0F 23EA 1CAF CC11  D486 0374 CF2E 8DD1 BDFD

Whoa! What happened?! “There is no indication that the signature belongs to the owner”?!

Wait a second. Does that mean that anyone can generate a keypair for Sonatype, Inc. “(Sonatype release key) <dev@sonatype.com>” pretending to be the Sonatype release team?

Here’s an amazing picture for you:

heinrichh@uni-duesseldorf.de

Click to see

Hiene writes code

 

According to the “trusted” key server, all those signatures belong to the German poet Heinrich Heine. Well, I would say that’s quite unlikely since he passed away quite some time ago1.

Let’s run a couple of more tests (go figure, maybe it’s only Sonatype who can’t generate a “truly trusted” signature):

Here’s a signed Eclipse artifact:

>gpg aether-1.0.0.v20140518-source-release.zip.asc
 gpg: Signature made 05/18/14 12:54:52 Central Daylight Time using DSA key ID A7FF4A41
 gpg: Good signature from "Benjamin Bentmann (CODE SIGNING KEY) <bentmann@apache.org>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.
 Primary key fingerprint: BA92 6F64 CA64 7B6D 853A  3867 2E20 10F8 A7FF 4A41

Same again!

Here’s Oracle’s OpenJDK tool:

>gpg jmh-core-0.9.5.jar.asc
 gpg: Signature made 07/24/14 13:53:36 Central Daylight Time using RSA key ID 060CF9FA
 gpg: Good signature from "Evgeny Mandrikov (CODE SIGNING KEY) <mandrikov@gmail.com>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.
 Primary key fingerprint: A413 F67D 71BE EC23 ADD0  CE0A CB43 338E 060C F9FA

Whoa, who’s Evgeny Mandrikov? An Oracle employee? Nope. Why does he use gmail? Let’s see, maybe Sonatype pre-verified him when they gave him access to Maven Central? Did he establish a relationship with Oracle or something? Nothing, not a single question. “I have nothing to do with Oracle, but still want to publish OpenJDK artifacts!” “OK, go ahead”. Can you trust him to provide you with authentic OpenJDK artifacts? Not sure2.

You get the picture. You can’t trust those self-generated key-pairs. And the Trusted Key Servers themselves acknowledge it! Here’s a quote from the Key Verification Policy of PGP Global Directory:

…there is always a risk that the verified key in the PGP Global Directory is not actually owned by the person who appears to own it. While the verification mechanisms in the directory are suitable for many purposes, you should endeavor to use additional mechanisms…

That’s exactly what modern repositories, like Bintray and GitHub do. But we’ll get to that shortly.

But what about WoT?

Ah,” one can say, “that’s because you don’t have a clue about how pgp signatures work! You need to establish a Web of Trust with the signer and voila, the message is gone!

While technically correct, this makes very little sense in our context.

WoT works for the original usage of pgp signatures – authenticating content from people that you know directly, or indirectly through your contacts. For example, it works great for signing emails. You are likely to get email from people in your first, second or third circle of connections, but almost never from complete strangers. With packages from an Internet repository, this concept breaks completely. Chances that a developer personally knows the creator of a package, even if its indirectly to one or two levels, are close to zero. Same works for the creators of a package. As an author, you have no idea who is going to use your package. You can only hope that they’ll be a bunch of strangers that you don’t know :-D.

I am intrigued. Show me how can I maliciously upload a forged artifact to Maven Central without even using MITM!

Tedious, but straight forward:

  1. Invent a fake identity (with a fake, but functional email address).
  2. Generate a keypair for it.
  3. Upload to a public key server (that’s where your email is needed).
  4. Follow the guide. The trickiest part is faking a story about your artifact to get your account set up in oss.sonatype.org through the Sonatype JIRA. Well, be creative!
  5. Within a couple of hours (or days) your artifact will get to Maven Central. If the description you invented  looks realistic enough, no questions will be asked. Somewhere during  the process of uploading and releasing files, Sonatype will check that the signatures with which you signed your artifacts match the email you used for your oss.sonatype.org account. But of course, they match.
  6. Mazal Tov! Your fake artifacts are now in Maven Central. From there they will be securely(!) downloaded over SSL to your victim computers by Maven.

Easy? Well, not really. Looks like the MITM attack was easier. But that’s what it takes to add a jar to Maven Central.
Doable? Absolutely.

But how is Bintray different?

Well, we realize that in the modern world, where identity theft is one of the more popular crimes, you can’t blindly trust an online identity. So, like many other services, we recommend that you assess how trustable any particular content is (a jar in our case), based on the credit the community gives to the Internet identity of the author.

Here’s what I mean:

Let’s say you want to download Groovy. Here it is. Let’s see how can you establish trust in Groovy binaries:

  • You can see the links to a website and GitHub.
  • You can see that this package belongs to an organization: Groovy. You can clearly see the list of members, and check each and every one of them.
  • Here’s Guillaume Laforge.
  • Check out his twitter account – almost 8.5K followers.
  • Here’s his blog, all around Groovy.
  • LinkedIn? Looks good.

So, can you trust him? Probably. He’s an admin of the Groovy organization, so his reputation is the guarantee that the organization and the files in it are authentic. You’re good to go. Or not?

That’s up to you! We give you the information. You decide if you can trust it or not.

 But Maven just goes out there and brings stuff!

To create a safe build you must use an in-house binary repository manager:

  1. Install it.
  2. Configure a “dirty” and a “clean” repository.
    1. The dirty repository should be able to proxy a remote source of artifacts that gives you a good insight on artifact producers’ identities (not just a pile of files with self-generated signatures).
      Consider proxying Bintray’s JCenter instead of Maven Central. JCenter is a superset of Maven Central, so finding packages won’t be a problem.
    2. The clean repository is a local repository with no Internet access.
  3. Configure your build tool (Maven?) to be able to build against either of them.
  4. Use a “sandbox” (e.g. a vm) to run a “dirty” build against the dirty repository. Its cache will be populated with all the artifacts needed for the build.
  5. Examine the identities of the publishers of those artifacts (remember to select a binary repository that can generate a list of dependencies used for every build) against the information you have about them (as you understood, it’d better be more than self-signed signatures).
  6. Promote the artifacts that you trust to the clean repository (remember to select a binary repository with deduplicated storage that gives you free and immediate “move” operations).
  7. Now you can safely build against the clean repository!

Here’s the summary (a.k.a. tl;dr):

  1. SSL is an important way to verify that the files you trust get safely from a server to your machine.
  2. Don’t let SSL give you a false sense of trust in the origin or author of the files.
  3. Don’t blindly trust self -issued signatures. One can generate a signature for any identity, sign any file with it and upload it to Maven Central.
  4. Go with a platform that enables and encourages web identity verification. Check the author and decide for yourself.
  5. Use a binary repository manager to discover and control the artifacts needed (and select it wisely).

1A gpg tool to work with pgp signatures was developed in Heinrich Heine University of Dusseldorf. In honor of the late poet after which the university was named, all defaults in the tool were initialized to Heinrich Heine, with the expectation that people would replace them their own details. Apparently, that didn’t work. People just enter-enter-enter through the settings generating dozens of key-pairs for the poor German, illustrating the absurdity of  “authentic”, self generated key-pairs.
Thank you, Heinrich!

2Actually, you can. Oracle do not distribute binaries of OpenJDK, and Evgeny volunteered, and was empowered by OpenJDK committers to do it for them.
But can you know without being personally familiar with Evgeny?! No, you can’t.

19 thoughts on “Feel secure with SSL? Think again.

  1. I accept my own incomplete understanding of the matter, but after reading your post, I’m not convinced you fully understand how gpg and Web-of-Trust networks are supposed to work (https://www.gnupg.org/gph/en/manual/x547.html). Sure, this also shows that few people really do, so the whole point of understanding when you should trust signatures and when you shouldn’t could be seen as a failure to properly raise awareness of the people behind such authentication/signing schemes.

    In particular, gpg issues those warnings (WARNING: This key is not certified with a trusted signature!) because you as a person have not built a “web-of-trust”: a network of people that you claim to trust, against which any signatures could be validated through the transitivity principle (if one of the people signing a package has had his/her key signed by one of the people you fully trust).

    Your point of people having a false sense of security when they don’t really understand the process is right though – and I admit things are broken that way (for the public at large). However, developers should be educated about this, and it might be more productive to point them in the right way instead of showing them how they can deceive others – If you are careful, you *can* get to a point where you trust that the person behind a signature is really who he/she claims to be.

    The alternative is of course using the centralized way certificate authorities work, but that comes at a (high) price for the people needing to buy signing keys periodically.

    • Thank you, Lucian, for this comment. The WoT works for the original usage of pgp signatures – authenticating content from people that you might know, directly or through your contacts. For example, it works great for signing emails. Chances are that you get people from your first, second of third-level connections, but almost never – from complete strangers. With packages from an Internet repository this concept breaks completely. Chances that a developer knows personally or even in a second or third degree, the creator of a package are almost always close to zero. Same works for the creators of a package. As an author, you have no idea who is going to use your package. You can only hope those will be bunch of strangers that you don’t know 😀

      It looks like those pgp signatures of packages from a repository are intended to serve as an analogue to site certificates. If one has it, it means that it’s OK and you can trust it. Well, that’s not the case with pgp signatures, is it?

      And yes, the alternative of buying a CA certificate for every package created is absurd. Although I belive Bintray provides a valid alternative of a “web identity” that the consumer should decide whether to trust or not.

      P.S. I’ll add a paragraph about why WoT won’t work with repositories. Thanks again!

      • Well, yes – and no. The way I see it, in the end you will have to place your trust in somebody (which is independent of the underlying signature/authentication scheme). You place your trust in the CA, you place your trust in your (let’s say) first-degree connections. In the case of binary packages, you could place your trust in a prominent developer (see Linus for the Linux kernel for example) or in the key of the repository maintainer (I agree this is problematic, but trust is inherently problematic at a human level). This doesn’t have to be one of your “contacts”, but somebody who is taking responsibility for package integrity within an organization. He will then need to sign-off packages as they become available in the repository, or delegate towards a small list of other trusted people.

        So when you see a package signed by him/her or signed by somebody else whose key was signed by this person you “trust”, you would accept the package as valid.

        What’s the alternative?

      • You are right. The whole idea of “Internet Identity” is to give you the ability to decide on whom you trust without knowing the name (as in case with Linus). For example, do you know who Trustin Lee is? How can you decide about trusting him and his packages? That’s the idea behind the Bintray user profile – to give you the tools to decide.

      • However, my point was that the idea of “Internet Identity” is not at odds with SSL/GPG/WoT as you were suggesting in your post (by showing how those things fail to work), but more complementary in nature: for example, perhaps I could get the correct pgp key for Trustin Lee from his bintray profile page? If I’m satisfied with his proof of identity I will then download & mark the key as trusted in the WoT sense.

        Or in other words – the underlying technical infrastructure is there, we need ways of making its use reasonable and failsafe. Would you agree with that?

      • I see “Internet Identity” as a replacement for the WoT that doesn’t (and can’t) work in case of public repositories (you can’t know the publisher). Once you decided you trust the publisher, the signatures work fine to establish the authorship.

  2. I could not agree more. SSL only provide a safe channel. But it doesn’t prevent you for downloading a malicious jar.

    About the MITM attack, the signatures could provide a way to ensure integrity if only we could trust these PGP keys. Currently, the attacker can simplify resign the files.

    Unless I’ve miss something, with your method, you’ve proven you could download anything to maven central but you can’t deploy a new version of an existing groupId/artifactId. So if I’ve right groupId/artifactId, then I can trust where does it comes from. So the question is how to retrieve this information ?

    I found your solution (clean/dirty repository) a bit constraining at the least for a start. With basic measures, I think you can stay safe. The same way as your example with Bintray, you can use the community to trust content as you said. If I want to download the spring framework, I will simply search “spring framework” and find on the developer website the download link where the groupId and artifactId is given. I may also search on mvnrepository.com and check the popularity. With this I can say, I’m pretty safe (I agree not as good as Bintray): I think I’ve the right artifactId and groupId (not a forgery).

    For transitive dependencies, you could trust the developers: for major open source projects this is not a problem (most dependencies are pretty common anyway). The issue arise if I want to use a jar not too common. In this case, I should audit the code as well as their dependencies. But even with common jars, I think, it’s a good practice to always keep an eye on resolved dependencies (as you do with your clean/dirty repository).

    It’s true adding dependencies is not something to be taken too lightly. It’s so easy with build tools that we have forgotten the risks behind it.

    • Thanks for your comment, Sylvain.
      You are right, there are bunch of other ways to verify the integrity of the packages you are interested in. Blindly trusting Maven Central is not one of them. We just try to make your life, as a developer, a bit easier. I hope we are on a same page that what you propose with searching the internetz and checking the mvnrepository.com can be achieved with Bintray in easier way.

  3. I see one change. You intend to stand by the incorrect statement that Central was compromised? Any informed reader who follows the actual story will deduce that this is sensationalism at best, blatent misinformation at worst and calls into question the rest of the blog.

    • I am not sure what we are arguing about? Does the term “compromised” describe what happened accurately? I think it does. MITM attack was successfully executed on Maven Central. Here’s a quote from the original post:

      When dilettante sees a JAR coming from Maven Central it replaces the original JAR with a backdoored version that runs malicious code on the victim’s computer.

      If that’s not a security breach, I don’t know what is.

      It’s good that we discuss it here, readers can judge whether this calls into question the rest of the blog or not.
      Thanks for your concern about the integrity of my posts!

      • It depends on whether this software was ran or not. The name “proof of concept” implies that it was ran (otherwise it’s not a proof). Once a software that exploits a vulnerability was ran, it became a compromise.

  4. Pingback: Canada Post community mailbox security concerns raised again - Nova Scotia

  5. Pingback: Feeling secure with Bintray downloads | Blog @Bintray

  6. This is a good analysis of the issues, but the current state of bintray’s solution here has some serious issues. First and foremost, asking anyone to upload their private PGP key defeats the whole purpose of using PGP signing in the first place: decentralizing trust. Bintray runs jcenter, stores the private keys, and via javascript can see the passwords to those keys. This makes Bintray the central spot to exploit in order to inject malware to the masses. Bintray should remove entirely the ability to upload or store the private PGP key and instead only store the public key and allow users to upload the PGP signatures directly (i.e. .sig and .asc files).

    Second, it is great to link up things like email address, twitter, etc. but right now this is done using OAuth. That means we have to rely purely on Bintray and the other service provider (e.g. Twitter) for the verification of the link. A better model is how keybase.io does the verification: people post public messages which anyone can verify is linked to a PGP key. Since a core goal of the verification system in Bintray is to provide verifiable link to a PGP key, the exact same system applies.

    • Thanks for your comments. I’d like to address both of them:

      • Bintray does not force to store your pgp keypair. If you don’t trust Bintray, that’s fine, you can sign on the client side, or not to sign at all! As I mentioned, we are not obsessed with the asc signatures and don’t require them. Also, Bintray does not store the keyphrase, so there is no javascript in the world that can show your password 🙂
      • I guess you are referring to this? I am not sure I understand why trusting Twitter for OAuth is not good enough, but trusting them for a public tweet is OK? Either you trust Twitter to verify the author or not.

      Thanks again for the insightful comment!

  7. I should start off by saying that my work requires me to think a lot about trust in systems, so my language often sounds paranoid since I’m always talking about trying to trust no one ;-). Based on your blog posts, forced HTTPS, and support for PGP, it sounds like the people behind Bintray are thinking about security and are trustworthy.

    The questions I’m posing are not about trusting the people behind Bintray, that is unavoidable, but rather removing as many things from the system that are based on believing people are doing the right thing. Whenever possible, these things should be replaced by things that anyone can verify. That makes for a much less fragile system in terms of security.

    Storing the secret key on your servers, then also sending that secret key’s password to the server means there is a single point where an attacker can get lots of signing keys plus their passwords, plus a software distribution mechanism. This means a huge liability for Bintray since you are responsible for all those keys.

    As for Twitter and web services, of course having some trust in Twitter when using Twitter in unavoidable. The problem I’m trying to highlight is that only Twitter and Bintray can validate the relationship, there is no public proof. The only way for someone else to check that connection is to look at Bintray or Twitter. I cannot verify the connection by looking at someone’s public Twitter posts and running the crypto process myself. The system that keybase.io and others like it have setup allows for exactly that.

    The point of all this is to design a system that is not brittle and does not completely fall apart if Bintray’s servers are exploited. That’ll take a load off of your shoulders because it means exploiting Bintray becomes less valuable.

    About the signature, I see that my misunderstanding is because bintray only supports .asc signature files, and not .sig signature files. Uploading .asc files works, so that’s a good option. I prefer .sig files because they are unambiguously always a detached signature. A .asc file can be a detached signature, signed content, encrypted content, or even a key.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s