Github + EC2: Integration Opportunities

Before the holidays, I was tasked with creating the first iteration of a tool that allows our developers to spin up their own private servers, in our Amazon Virtual Private Cloud. In my first sketch of the idea, I called it “Thinglauncher” and (to my delight and coworkers chagrin) they name has stuck.

Meanwhile, we are nailing down our installation of Github Enterprise– a behind-the-firewall version of Github.com, with all the things that make Github useful: code repositories, gist, wiki’s, and social features. Trust me, It’s awesome.

What follows are some ways the two systems work together.

Authentication

One of my first thoughts was: this new launcher tool should use Github as an login mechanism– it avoids two kinds of authentication pain that I’ve dealt with:

  • Making users remember a new password
  • and the alternative, making me figure out how to authenticate against something like  ActiveDirectory

Win, win. See also the “oauth as identity” section of How Github uses Github to build Github.

Key management

When you create a (linux) EC2 instance, you specify an SSH public key, which gets added to the authorized_keys file on the newly created server. As long as the corresponding private key is installed on your workstation, you’ll be able to log in.

Github uses SSH keys as well– once you add your public key to your Github account, you’re able to push and pull code from any workstation that has your private key. The Github API also exposes your SSH keys. This means that an app that authenticates against Github, can also grab a users public keys via the API. If what your app does is launch EC2 instances, the key can be grabbed from Github and used to launch the EC2 instance. You’ve allowed your user to log in to the newly created instance without making them create a new password or new SSH key.

More key management

If we are in fact creating servers for development work, then the servers need to be able to push and pull code to and from Github. We could expect users to do the whole ssh-keygen dance for each server, or to remember to upload their private key to each server they create. No fun, right?

After some digging, I found a handy recipe for generating new key pairs in Python. The first time a user creates a server, a new key is created and stored behind the scenes. The server template (What Amazon calls an “AMI”) we’re using includes Ubuntu’s CloudInit bootstrapping system. The new private key is installed in the users SSH configuration, and the public key is added to the users list of keys in Github. Users can log in to the new server, and push and pull to github just like they can on their local machine.

Code checkout during provisioning

There’s not much magic here: When the user creates a server, they can specify a repository and branch name that get checked out to the users home directory on the new server.

Server configuration in git

This part is a little magic. If the repository you checkout includes a file called ‘fabfile.py’ in the root folder, it assumes it’s a Fabric script, and runs it against your new server. This can be used to further customize the server for the code you want to run. For example, if you’re developing a wordpress theme, Fabric can be used to install PHP, MySQL, the libraries that integrate the two, wordpress itself, and import a wordpress database.

I hate to use the word “synergy”, but…

It was striking to me how all these nice integration points appeared. Whenever I scratched my head about how to solve a particular problem, the answer was usually “use this feature of Github” or “that feature of EC2”. I’m sure there are more that I haven’t discovered, yet.

This should all get less abstract, soon: we’re working on our process and policies to open source (really, public domain) all of our code, and releasing Thinglauncher is on the roadmap.

Leave a Reply