Even more Vagrant love in Bintray

You, of course, know, that for nearly the last two years, you have been downloading your Vagrant software from JFrog Bintray. But recently, Bintray has taken Vagrant support to a whole new level; it is now is a fully fledged Vagrant repository allowing you to distribute your public and private Vagrant boxes from Bintray! As for everything in Bintray, it’s simple and powerful:

Publishing Vagrant Boxes

1. Create a Vagrant repository (if you’re a new user it is likely that you already have a default Vagrant repository named “boxes”):

Create a Vagrant repository

2. Click on “Set Me Up!” and copy/paste the REST command to upload the boxes:

Create a Vagrant repository

Consuming Vagrant Boxes

Follow the “Downloading” section under “Set Me Up!” to configure box resolution for downloading Vagrant boxes, and be able to enjoy automatic box update during ‘vagrant up’ a box either by an explicit call to  ‘vagrant box update’. That’s it, now you can benefit from all the power behind Bintray distribution: CDN, stats, logs, version notifications and more. So give it a try, put a box or two on JFrog Bintray today!

Enterprise Level Access Control with Keys and Entitlements

entitlements200x190

“Private repositories”, “Teams and Organizations”, “Permissions”…, sounds like that’s all you need to provide secure private downloads. Well, not quite. Those are great features that fit the bill if your consumer is a Bintray user. But what if she isn’t? Well, then there are signed URLs. Those should do the trick. Just sign your file and send your consumer the URL. But what if you want to share an entire repository, package or version with a group of people, but need to give each of them different privileges. Some can only view or download your files, but others should be able to delete your files or upload new ones. Signed URLs don’t cover that kind of control. They are great for single files, but for more fine-grained access control, you need a more sophisticated feature. That’s where entitlements and keys come in.

“Entitlements,” you said? What are those?

Entitlements are privileges you can give anyone…yes anyone, not only Bintray users, to entities in your private repositories. “Entities” means anything that can contain files – a whole repository, a path within the repository, a specific package or a specific version. “Privileges” means“rw”  – download, upload and delete, or “r” – download only. If you didn’t notice – the combination of entities and privileges gives you any level of granularity that you need for providing access.

But how do you unlock entitlements?

I guess you get the hint. Keys unlock entitlements. You generate a key along with its password (or Bintray can generate one for you automatically). Your user will have to provide the username and password to enable the key that unlocks the entitlement. That’s where the security lies. Only users who have both the username and the password of the key that you provide to them can access your repository entities according to the entitlements you created.

So how does it all work?

Two simple steps using Bintray’s REST API:

  1. Create keys. You can supply a password for each key you create, or Bintray can generate one for you.

  2. Create entitlements. When you create entitlements, you specify which keys to apply to them.

Now all you need to do is provide your user with the username and password for one of those keys. Your user now applies a REST API to access your private Bintray resource while including the key and password you provided as parameters to the API call. Bintray will check if there is an entitlement that allows access to that resource, and if that entitlement has the key that the user specified associated with it.

Let’s see an example at work.

Let’s say user “ACMECorp” has a private repository called “acme”.

This private repo contains several versions of acprod under the “acmecorprod” directory that are protected from public access.

acmecorp wants to authorize a “platinum customer” to download the different versions. Needless to say, “Ms. Platinum” does not have a Bintray account.

First, ACMECorp needs to create a key. Bintray offers two ways to maintain control over keys that are distributed to outside users. The easy way is to just put an expiry date on the key. A more advanced method is to set up an internal server that is used to validate and authenticate keys, and provide the server URL to Bintray when a key is created. Every time a user tries to access ACMECorp’s repositories, Bintray will validate the key using  the URL that ACMECorp provided when creating it. Since ACMECorp is very careful with its key, let’s assume they want to validate keys with their own systems:

curl -XPOST -uacmecorp:APIKEY “https://api.bintray.com/users/acmecorp/download_keys
{
“id”: “key1″,

“expiry”: 7956915742000

“existence_check”: {

  “url”: “http://callbacks.myci.org/username=:username,password=:password”,

  “cache_for_secs”: 60

}


}

Bintray creates a key and its associated password

Status: 201 Created
{
“username”: “key1@acmecorp“,
“password”: “8fdf84d2a814783f0fc2ce869b5e7f6ce9f286a0″
}

acmecorp now creates an entitlement that provides download privileges to the acmecorprod directory, while assigning key1 that was just created

curl -XPOST -uacmecorp:APIKEY “https://api.bintray.com/packages/acmecorp/acme/acprod/entitlements

{
“access”: “r”,
“download_keys”: [“key1″]
}

Bintray responds:

Status: 201 Created
{
“id”: “7f8d57b16c1046e38062ea3db91838ff77758eca”,
“access”: “r”,
“download_keys”: [“key1″]
}


Basically, that’s it. acmecorp can now just provide the username “key1@acmecorp” and its password to Ms. Platinum who can now use them to access the acmecorprod directory in acmecorp’s repository.

For example, to download version 1.0 of acprod, Ms. Platinum would use:

curl -XGET “https://dl.bintray.com/acmecorp/acme/acmecorprod/1.0/acprod.exe” -ukey1@acmecorp -p”8fdf84d2a814783f0fc2ce869b5e7f6ce9f286a0”

But what happens now if acmecorp and Ms. Platinum have a falling out. When that happens, acmecorp can just delete the download key from their validation server and “Hey presto”, Ms. Platinum is now locked out of acmecorp’s repositories.

Try doing that on Docker Hub, RubyGems.org, NuGet Gallery, Maven Central or any other repository or download center out there. Bintray is the only one that provides you with this level of control over access to your private resources.

Is Docker Hub really the best way to distribute your images?

Docker is definitely one of the biggest things to hit the software industry in the last few years. Everyone is using Docker, and Docker Hub is growing rapidly serving over 45,000 images by now. But did you ever ask yourself if Docker Hub is really the best platform to distribute your Docker images. It stands to reason, but reason be damned; We think Bintray has an edge over Docker Hub and here are the reasons why.

1. Fine-grained access control
Yes, Docker Hub recognizes organizations and teams. You can have all the relevant people open an account on Docker Hub and then organize them in whatever way serves your needs to expose or hide registries as you see fit.

But then you want to send your #1 strategic client an early release

How does Docker Hub handle that? Well, you can ask for your customer’s account, and then include him in the right organization or team. But then, you need to be really careful with what you put in the repository you have exposed to him. Or you need to add him to a team, and later remove him. Manageable so far… But now he also wants his boss to see it, and later the CTO, and two weeks later, you find that your guy has just changed jobs and is working for your competitor. Did you remember to remove him from that team or can he still access your private repository?

Bintray removes this pain-point with entitlements and download keys

Bintray lets you create entitlements which define, in minute detail, exactly what part of a repository can be accessed – from a whole repository down to the level of a single path within a repository. You can define who has access by name, geographic region, IP domain and pretty much anything else. You can also limit that access to a very specific period of time. Once you have defined your entitlements, you can then create keys which unlock the access the entitlements provide. Your user needs to provide the key and its associated password each time he accesses a resource which he is entitled to. You can even define what kind of access each entitlement provides. It can be for download only, or for download, upload and delete.

2. Image and Version Level Stats and Logs
Ever wondered how many times your image has been downloaded? Of course you have; it’s a measure of success. But going beyond self-flattery, it’s not enough to know that magic number of downloads. Wouldn’t it be helpful to be able to know who downloaded your images and from where? Bintray gives you that. The stats pages on Bintray give you total numbers, but download logs let you segment the downloads by region, country, IP address, and even down to the specific user and organization. For your public repositories this can be really useful, but for your private repositories it’s critical for you to know who is accessing your packages and what they are doing in your repositories. You can read more about stats and logs in this post the Bintray Blog.

3. There is more to the world than Docker
allforone4Docker may be the best container to have ever hit the software docks, but at the end of the day, you are only using Docker to create a virtual world in which to run your products. And your products are built with packages that are Java, Ruby, NuGet, Python, Debian … and new formats hit the news all the time. How are you going to distribute those? Not on Docker Hub. So should you use Docker Hub for Docker and something else for all the rest? Well, if you can use that something else, aka Bintray, for Docker images too, then why not simplify your life (and scripts) and distribute all packages through Bintray.

All for one and one download center for all.

3.5. For Bintray, distribution is a core competency
Docker is making huge waves in container technology. This is their core competency; not software distribution. You can use Docker Hub to distribute your images, but you must ask yourself, “Will Docker Hub receive the same level of attention from its makers as container technology does?” Bintray’s scalable infrastructure is deployed on clustered servers in multiple data centers around the world, and currently serves over 200 million downloads per month of 200 thousand packages that reside in 50 thousand repositories.

That’s what we call a core competency!

Like Docker Hub, Bintray offers a host of great features like fast CDN downloads, a rich REST API for automation, searchable metadata and more. But like Docker is redefining container technology, Bintray is redefining software distribution. The service tiers offered by Bintray make it the best choice for distribution of both free OSS images as well as for commercial organizations that need enterprise grade software distribution features. Bintray lives and breathes software distribution; that’s what it does, and that’s where it will continue to evolve to stay the best solution for a download center, for Docker images and indeed for all other software package formats.  

Download stats and logs – now with deep user insights

Ever wondered who exactly downloaded your software? I don’t mean just “someone from the United States.” I’m talking about getting down to the organization level in terms of “someone from Acme Corp. NY office”.

Now you can get this information, from Bintray:
Bintray Live Download Feed

This information is available for any package type for Bintray Premium users, but that’s not all. It is also available, for free, for every OSS package in JCenter!!! This means that if you own a package that is linked to JCenter you already have this information available today!

And not only can you get rich live logs of your downloads, you can retrieve fully parsable logs in CSV or Apache common format! Just use the REST API, or click the preferred log format link:
Bintray Download Logs

Log format options include the Apache-style logs, which can be analyzed by dozens of tools out there, or you can download the logs as CSV files. CSV files are usually easier to understand, and actually contain more data, such as ZIP code, geolocation etc. Download the CSV log and hack on! Or, you can even open it in Excel and show it to your boss:
Bintray Logs in CSV Format
Excel jokes aside, that’s the most powerful downloads analytics one can get and it’s there at your fingertips.

With Live Logs and Download Logs you get much more detail about activity within your repositories. This can be very helpful in analyzing peaks as it provides immediate feedback on the popularity of your software, and how this popularity is distributed by version, geo-location and organization. Naturally, it can also be used to direct you to where you should be focusing your marketing efforts.

It’s time to learn about your users!

Another one bites the Maven Central dust (and saved by Bintray)

Today, I encountered another very detailed blog post on the woes of publishing on Maven Central. Jose Maria Arranz (@jmarranz) explains why he doesn’t like Maven in general and publishing to Maven Central in particular (I am with him on a lot of valid points).

I can’t help quoting:

Fortunately when searching for how-to articles and commenting in Twitter, @jbaruch an employee of JFrog contacts with me offering Bintray.com alternative to publish to Maven Central, the people behind JCenter, I read the article “The easy way to maven central” and I was sold. Bintray provides a GUI to upload and self sign your artifacts if you provides your public and private GnuPG keys, and with a simple UI action you are published in JCenter repository, and providing the Sontaype user and password you can finally easily publish in Maven Central.

Bintray helped me to break the wall of Sonatype process. I’m saved!!

Currently I’ve released RelProxy on JCenter and Maven Central. For releasing I use Ant calling to Maven tasks to generate the required Maven artifacts and to generate a distribution zip with everything. Everything could be automated, I could add signing and uploading from Ant (or maybe by the POM) without Bintray, but Bintray auto-signing and uploading UI is enough for me, releasing is done from time to time and most of releasing process is already automated, and releasing in JCenter is a plus.

Note: Don’t forget JCenter, for instance Maven Central is no longer pre-configured in Google Android environment.

That’s what I call “success”!

Thanks for the kind words, Jose Maria! I am happy that we managed to help :)

Android Studio – Migration from Maven Central to JCenter

This post was originally published in Techno Talkative blog by Paresh Mayani. Feel free to comment here or there.

 

During the android workshop, in the office and in the chat with some of the android developers, I have received some questions around build script and repository:

  • Why earlier versions of android studio were using maven central?
  • Why android project, getting created using android studio, is using jcenter?
  • What are the reasons/benefits/purpose that android studio has gone with JCenter repository?
  • JCenter Vs. Maven Central

I had shared some of the points at that time, but I tried to go into the deep to know more about the migration made from maven central to jcenter. I have found interesting details from the couple of links:

Before going ahead and we discuss about the migration made from maven central to jcenter, here is brief about the repository (in case if you really don’t know!)

What is repository?

In Maven terminology, a repository is a place i.e. directory where all the project jars, library jar, plugins or any other project specific artifacts are stored and can be used by Maven easily.

In build.gradle file, you need to specify which repository to use when resolving dependencies for building your project, so you need to put something like the following in your build.gradle file:

repositories {
    jcenter()
}    

Yes, you need to specify at least one repository before you use any external dependency. Read more about Dependency Management Basics and repositories.

Maven central to jcenter migration

Now, here are the details and interesting points about the jcenter and the transition made from maven central to jcenter:

  • As said earlier, jcenter is the new default repository used with Android’s gradle plugin.
  • jcenter is a Java repository in Bintray, which is the largest repo in the world for Java and Android OSS libraries, packages and components.
  • All the content in jcenter is served over a CDN, with a secure https connection. Back in the time of the migration (Android Studio 0.8) The central maven 2 repository was HTTP only and HTTPS wasn’t supported. Reference: 51.6.2. Maven central repository. (This might be one of the reasons as Google is a fan of HTTPs!).
  • jcenter() is a superset of mavenCentral(), that encompasses many additional repositories and artifacts. Reference: Blog @ Bintray.
  • The jcenter guys claim, that they have a better performance than maven central.
  • Bintray has a different approach to package identification than the legacy Maven Central.
  • If you really need to get your package to Maven Central (for supporting legacy tools) you can do it from Bintray as well, in a click of a button or even automatically.

Regarding performance improvements, couple of android developer advocates had faced/noticed the issue of huge indexing with maven central.

In the words of Tor Norbye:

I ran AndroidStudio with a brand new settings directory, so it went and connected maven central and downloaded an index of the available artifacts.

Then I happened to look at the size of my directory.

My ~/Library/Cache/AndroidStudioPreview is 1.5G, and 1.2G of those are taken by the “Maven” subdirectory.

That’s ridiculous. We barely use the index at all. The primary use for it is the Dependency editor in the Project Structure Dialog, but we really don’t need to have a precomputed index for it. MavenCentral has a fast online JSON search we can use on demand when somebody searches for artifacts. In https://android-review.googlesource.com/#/c/94843/ we added a lint check which checks whether the dependencies are up to date, and the search for a handful of artifacts is near instant.

In short, we really don’t need the cache; it may help with code completion in .gradle and maven .pom files, but that’s not a super important usecase, and certainly not something *all* users should have to sacrifice 1.5G of download speed and diskspace to have the possibility of one day doing.

Read more on: The Maven index is *huge*!

I must say, after this migration from maven cetnral to jcenter, performance of the android studio is noticeable. Earlier loading android studio in even 4GB RAM configuration laptop, was all like:

code-21

Any way, this is not the ending of this study here, I still want to know more about the improvements or strong points (if any). If you know or have played enough with jcenter and maven central, please share your inputs. And also, as this is almost an undocumented point, I am really looking forward to hear about this migration, from the android developers advocates or engineers working in android tools team.