Mathematics, philosophy, code, travel and everything in between. More about me…

I write about

How to use Google App Engine with virtualenv

Google App Engine is a pretty interesting PaaS offering. However, its Python environment leaves a lot to be desired. The biggest pain point is its seeming incompatibility with virtualenv – a tool that is almost indispensable in Python development. Luckily, there is a workaround.

Virtual environments are perfect for isolating your project from the global Python installation and managing its package dependencies separately. Unfortunately, most App Engine guides will tell you to “just download package ZIPs and unpack them in your project directory.” Did that make you wince? It should. Managing dependencies manually is tedious, dangerous, and difficult to synchronize in a team. We have pip, virtualenv and requirements.txt solve precisely these problems.

App Engine SDK doesn’t play nice with virtualenv because it’s a sandbox trying to emulate the environment on GAE production servers. When you run it inside a virtualenv it does its best to break out, access the system-wide Python and occasionally deny access to the virtualenv packages.

The workaround

tl;dr: get linkenv to symlink virtualenv packages to your project dir.


When you deploy an app to GAE the deploy script will resolve any symbolic links inside your project directory and upload their actual targets. This allows us to work around the virtualenv incompatibility:

  1. First we create a virtual environment and install packages inside it as usual, taking full advantage of requirements.txt, pip, automatic dependency installation etc.
  2. Then we create symlinks pointing to our virtualenv packages and place them inside our project directory.
  3. We make sure the symlinks are at the beginning of sys.path.
  4. When we deploy the actual packages from the virtualenv will be uploaded.

Step 1 is easy:

$ virtualenv env
$ source bin/env/activate
(env)$ pip install djangoappengine six ...
(env)$ pip install -r requirements.txt ...

To automate step 2 I’ve created a small script called linkenv. Install it in your virtual environment:

(env)$ pip install git+https://github.com/ze-phyr-us/linkenv.git

To create package symlinks in directory gaenv (for example), go to the root of your project and run

(env)$ linkenv env/lib/python2.7/site-packages gaenv

The first argument indicates where your virtualenv packages are installed. After the script finishes you need to make sure that gaenv is in your import path. One easy way is to add this to your appengine_config.py:

import os, sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'gaenv'))

After this you will be able to use the virtualenv in your local environment, as well as deploy the packages to production servers.

A few more tips

SDK location

I keep the SDK outside project directories, so I need to ensure it’s in my $PATH before running the development server:

$ source env/bin/activate
(env)$ export PATH=$PATH:/path/to/google_appengine

Ignore the virtual environment in deployment

It’s better to list your virtualenv in skip_files in app.yaml to prevent it from being deployed unnecessarily (add this line: - ^env/.*). However, it has to be allowed for local development, otherwise the SDK’s sandbox will deny access to it. At the moment I’m just keeping the ignore line commented out and only enable it for deployment.

Weird tkinter imports

I had a lot of trouble with the development server trying to import binary .so modules from my global Python installation (like _tkinter). Due to the sandbox restrictions this would obviously fail. Amusingly (or not), most “relevant” Google results for this problem were people actually trying to use Tkinter (a native GUI library) and expecting it to work in their GAE web application. Anyway, after a frustrating afternoon I decided to simply put empty .py files of the same name in my import path. They would then get quietly imported instead of the shared libraries. On my system (Mavericks) this meant touch gaenv/{_tkinter,gdbm,_winreg}.py. I’m still not quite clear why the SDK is trying to import everything it can find…

February 10, MMXIV — Python, Programming.