I wish someone wrote django-static-upstream… maybe even… me!

I used to think serving static files (aka static assets) is really easy: configure nginx to serve a directory and you’re done. But things quickly became more complicated as issues like asset compilation, CDNs/scalability, file-specific custom headers, deployment complexity and development/production parity rear their ugly heads. Judging by the huge number of different asset management packages on djangopackages, it seems like I’m not the only one who ran into this problem and felt not-entirely-satisfied with all the other solutions out there. Things actually took a turn for the worse ever since I started drinking the Heroku cool-aid, and for the live of me I just can’t make sense of their best-practice with regard to serving static files. The heroku-django quickstart conveniently sidesteps the issue of statics, and while there are a few support articles that lightly touch on neighboring subjects, nothing I found was spot-on and hands-on (this is an exception to the rule; the Heroku cool-aid is otherwise very tasty and easy to drink, to my taste so far). Ugh, why can’t there be a silver bullet to solve all this? Let me tell you about my “wishlist” for the best static serving method evar.

First, I want to be able to take any checkout from my VCS, maybe run an easy bootstrap function, and get a working development environment with statics served. In production, I want to serve my statics from a CDN with aggressive caching, so I need some versioning system, but I’d like to minimize deployment complexity and I want fine-grained cache invalidation of my statics. I want my statics to be served the same way (same headers, same versioning mechanism) in development and production without having to update two different locations (i.e., my S3 syncing script and my development nginx configuration). I also don’t want to have to “garbage-collect” my old statics from S3 every so and so days. Like I said, I’d like some of my statics to be served with some bells and whistles, like various custom headers (Access-Control-Allow-Origin, anyone?) or gzip compression. Speaking of bells and whistles, how about a whole marching band, since I want to serve statics that require compilation (minify/concatenate/compile scss+coffe/spritalize/etc), but I don’t want to have to rerun a ‘build process’ every time I touch a coffee script file in development. Finally, and this isn’t something I’m not very adamant about, but I prefer my statics to be served from a different subdomain (static, not www), I think it’s cleaner, I don’t need my clients’ cookies with every static request and it allows for some tricks (like using a CDN with support for a custom origin).

And nothing I found does all that, definitely not easily. In my dream, there’s a package called django-static-upstream, which is designed to provide a holistic approach to all these issues. I’m thinking:

  • a pure Python/django static HTTP server (probably just django.views.static with support for the bells and whistles as mentioned above), and yeah, I think I should bloody use this server as a backend to serve my in production
  • a “vhost” middleware that will replace request.urlconf based on the Host: header; if the host starts with some prefix (say, static), the request will be served by the webserver above
  • a couple of template tags like {% static "images/logo.png" %} that will create content-hashed links to the static webserver (i.e., //static.example.com/829dd67168a3/images/logo.png); the static server will know to ignore the content-hash bit
  • this isn’t really up to the package, but it should be built to support easily setting a custom-origin supporting CDN (like CloudFront) as the origin URL; this is both to serve the statics from nearby edges CDN but also (maybe more importantly) to serve as a caching reverse proxy so the dynamic server will be fairly idle
  • support for compiling some static types on the fly (coffeescript, scss, etc) and returning the rendered result; the result may have to be cached using django’s cache (keyd by a content hash), but this is more to speed up multi-browser development where there is no CDN to serve as a reverse caching proxy than because I worry about production

So now I’m thinking maybe I should write something like this. There are two reasons I’m blogging about a package before I even wrote it; first, since I wanted to flesh out in my mind what is it that I want from it. But second, and more importantly, because I’d like to tread carefully (1) before I have the hubris to start yet another assets related django package, (2) before I start serving static content with a dynamic language (what am I, mad?) and (3) because compiling static assets on runtime in violation of the fifth factor in Adam Wiggins’ twelve-factor app manifesto (what, you didn’t read it yet? what’s wrong with you?). These are quite a few warning signs to cross, and I’d like to get some feedback before I go there. But I honestly think I’ll be happier if I had a package to do all this, I don’t think writing it should be so hard and I hope you’d be happy using it, too.

The ball’s in your court, comment away.


Comments

7 responses to “I wish someone wrote django-static-upstream… maybe even… me!”

  1. I find that using boto, django-storages, and Amazon S3/Cloudfront works very well for me, and is pretty easy to set-up, configure and provide custom headers to set caching max-age etc. I just configure a storage instance with the custom domain, headers, access policy and then set the STATICFILES_STORAGE setting to use this in my production environment.

    Using Django 1.3’s ‘collectstatic’ command with ‘–settings project.productionsettings’ allows for easy upload of static files to S3 before I then push changes to the production server.

    Invalidation/compilation is still an issue. Haven’t really used compilation/minification much, and I rename my css files when I update then to include the date/time in the file name – this is a bit of a pain.

  2. Wolfgang Schnerring Avatar
    Wolfgang Schnerring

    I’m not very familiar with the Django ecosystem, but you might want to take a look at http://fanstatic.org, an asset management system that hooks up to the WSGI pipeline.

  3. I just use django-storages and django-compressor (with offline mode enabled for production). Everything is automatically minified/compressed without having to configure much, and it just works. Some people mix it with django-staticfiles to gain the {% static %} tag you mention, or just to have collectstatic to gather their files when they are scattered all over the place.

    IMHO, it would be better to fix the flaws/missing stuff in the existing apps if possible. There are already too many django assets management apps already!.

  4. I would love to work with you on this. I have a media manager I wrote with exactly the same motivations in mind.

    I would be willing to put it out on GitHub and share it if you think it is a good basis for what you want.

    Literally all of your requirements are the same as mine. We do media serving via Django for development, but we serve from a cookieless domain for production. In the future this might be a CDN, but for now is just a dedicated machine.

    Static files are pre-processed, I currently do minification and gzipping (so Apache can serve pre-compressed files where appropriate). I also do an additional step, which is to combine javascript files that are used together into bundles.

    This is all done by a management command that scans your templates. It finds instances of the {% media %} template tag and extracts the contents to build up a list of media files. Once it has this list, it groups files by type and by template. Then it groups all combinations of files that are used together (that don’t contain other smaller groups). These bundles are concatenated, and compiled.

    The {% media %} then serves the bundle in production, or individual files in development mode. It also uses the .min flavor for production, but the non-minified version for development.

    Deploying to the CDN is also handled by my media management command. So the process is:

    python manage.py media –compress –minify –combine –deploy

    Which preps and uploads everything to the CDN. Versioning is handled differently than you describe, but I like your method better (hash before filename which is ignored by media server).

    I provided my email with this comment, and will monitor it for follow-ups. If you are interested, I will put this on GitHub and we can continue there.

    1. Sorry, a couple other points. We deploy our production and beta assets differently. So the deploy command actually takes an environment parameter so that it can invoke the correct deployment method.

      python manage.py media –deploy=production
      – or –
      python manage.py media –deploy=beta

      Currenly my deployment is built around lftp, which syncs all the assets to the server via FTP. But this is configuable in settings.py:

      MEDIA_DEPLOY_COMMANDS = {
      ‘production’: ‘lftp …’,
      ‘beta’: ‘lftp …’,
      }

      I have not yet automated deploying of the app itself, but when I do, then pushing to production will be just two management commands… I will probably end up using fab for that.

      I do agree with the others concerning re-inventing the wheel, but for me it was easier to write what I needed than to piece together a solution from various parts then fix all the parts for my requirements. I did not open source my code because I did not want to pollute the Django ecosystem with another wheel written specifically for my requirements.

      However, if you feel the same way I did, perhaps the code is useful outside my walls.

  5. I would use this if it existed.

  6. Hi there just wanted to give you a quick heads up. The text in your content seem to be running off the screen in Internet explorer. I’m not sure if this is a formatting issue or something to do with internet browser compatibility but I thought I’d post to let you know. The design look great though! Hope you get the problem resolved soon. Kudos