django-nudge

We have some interesting security restrictions at work. On particular requirement is that we can’t have an administrative interface exposed on our public web servers. For WordPress (which powers most of consumerfinance.gov) we use a firewall to block access to sensitive URL’s, and RAMP to push content from a separate staging server.

The non-Wordpress parts of our site are powered by Django. We wanted (and couldn’t find) something like RAMP for Django. So, we built it. It’s called Nudge (named by Josh, who wrote much of the code).

Much of the actual hard work is offloaded to the excellent django-reversion, which handles tracking changes in objects. What Nudge adds is the ability to package up a set of changes (a “batch”, terminology borrowed from RAMP) , and move those changes between servers. Thanks to the design of reversion, this works nicely for new objects, changes, and even deletions.

Nudge is released into the public domain– if you have ideas, problems, or questions, let us know, or fork away!

Github + EC2: Integration Opportunities

Before the holidays, I was tasked with creating the first iteration of a tool that allows our developers to spin up their own private servers, in our Amazon Virtual Private Cloud. In my first sketch of the idea, I called it “Thinglauncher” and (to my delight and coworkers chagrin) they name has stuck.

Meanwhile, we are nailing down our installation of Github Enterprise– a behind-the-firewall version of Github.com, with all the things that make Github useful: code repositories, gist, wiki’s, and social features. Trust me, It’s awesome.

What follows are some ways the two systems work together.

Authentication

One of my first thoughts was: this new launcher tool should use Github as an login mechanism– it avoids two kinds of authentication pain that I’ve dealt with:

  • Making users remember a new password
  • and the alternative, making me figure out how to authenticate against something like  ActiveDirectory

Win, win. See also the “oauth as identity” section of How Github uses Github to build Github.

Key management

When you create a (linux) EC2 instance, you specify an SSH public key, which gets added to the authorized_keys file on the newly created server. As long as the corresponding private key is installed on your workstation, you’ll be able to log in.

Github uses SSH keys as well– once you add your public key to your Github account, you’re able to push and pull code from any workstation that has your private key. The Github API also exposes your SSH keys. This means that an app that authenticates against Github, can also grab a users public keys via the API. If what your app does is launch EC2 instances, the key can be grabbed from Github and used to launch the EC2 instance. You’ve allowed your user to log in to the newly created instance without making them create a new password or new SSH key.

More key management

If we are in fact creating servers for development work, then the servers need to be able to push and pull code to and from Github. We could expect users to do the whole ssh-keygen dance for each server, or to remember to upload their private key to each server they create. No fun, right?

After some digging, I found a handy recipe for generating new key pairs in Python. The first time a user creates a server, a new key is created and stored behind the scenes. The server template (What Amazon calls an “AMI”) we’re using includes Ubuntu’s CloudInit bootstrapping system. The new private key is installed in the users SSH configuration, and the public key is added to the users list of keys in Github. Users can log in to the new server, and push and pull to github just like they can on their local machine.

Code checkout during provisioning

There’s not much magic here: When the user creates a server, they can specify a repository and branch name that get checked out to the users home directory on the new server.

Server configuration in git

This part is a little magic. If the repository you checkout includes a file called ‘fabfile.py’ in the root folder, it assumes it’s a Fabric script, and runs it against your new server. This can be used to further customize the server for the code you want to run. For example, if you’re developing a wordpress theme, Fabric can be used to install PHP, MySQL, the libraries that integrate the two, wordpress itself, and import a wordpress database.

I hate to use the word “synergy”, but…

It was striking to me how all these nice integration points appeared. Whenever I scratched my head about how to solve a particular problem, the answer was usually “use this feature of Github” or “that feature of EC2”. I’m sure there are more that I haven’t discovered, yet.

This should all get less abstract, soon: we’re working on our process and policies to open source (really, public domain) all of our code, and releasing Thinglauncher is on the roadmap.