Introducing django-ajaxerrors

2011/01/31 § Leave a comment

Today I finished a small django middleware that facilitates development of AJAX applications. I reckon most if not all Django developers know about Django’s useful debug-mode unhandled error page (you know, the one that looks like this). However, when an AJAX request reaches a faulty view, the error page will not be rendered in your browser but instead be received by your AJAX error handler (assuming you even had one), which is almost always not what you want. This forces you you to find some other way to reach your traceback information. For example, before I wrote this package, I used to regularly open Chrome’s developer tools, find the failed resource in the Resources tab, and then either read through the raw HTML (yuck) or copy and paste it to a file and double click it (tedious).

As you can see, this bothered other people, too, but I couldn’t find a decent solution on the web. Thankfully, since the problem is really about ease of development and not very relevant in environments where DEBUG is false (you’d get the traceback via email anyway), and since I do most of my development work locally (and I suspect so do many other Django developers), I figured the solution can take advantage of the server being a full fledged desktop with a modern browser and a GUI. Enter ajaxerrors.middleware.ShowAJAXErrors.

This little middleware intercepts all unhandled view exceptions, pickles the technical error page and uses Python’s webbrowser module to direct a new browser tab at a special URL that will serve (and delete) the previously stored page. All this is only triggered if DEBUG and request.is_ajax() is true, so pretty much everything you’re used to in your development flow should stay the same. Sweet. ShowAJAXError can also be configured to run arbitrary user-defined handlers with the error information, and even comes with a handler for growlnotify and a handler that replies to all failed AJAX calls with HTTP OK result containing an empty JSON object (I use it for development, YMMV).

django-ajaxerrors is on PyPI, so you can install it with pip or easy_install with ease. You can clone/fork the code (and read the more detailed readme) here. Also, if you’d like to see what django-ajaxerrors is like without any special effort, you can download a simple AJAX app I’ve written for the purpose of developing django-ajaxerrors itself. The app has django-ajaxerrors contained within it, so this is pretty much all you need to see django-ajaxerrors in action about 1 minute from now:

$ virtualenv -q demo
$ pip install -q -E demo django
$ cd demo
$ source bin/activate
(demo)$ curl -s -o - | tar jx
(demo)$ cd ajaxerrors-demo/
(demo)$ python runserver
# demoserver running...

Now point your browser at the development server, and play with the calculator a bit. Hint: the calculator is not protected against division by zero. :)

I hope you like django-ajaxerrors, it’s MIT licensed, so do what you will with it. Let me know if you need assistance or ran into a snag, or, even better, if you want to contribute something to it.

Something’s horribly wrong with Dell (and HP)

2010/12/09 § 4 Comments

Maybe it’s not the most interesting content I’ve ever published, but I’d like to share something with my readers, secretly hoping that maybe some Dell or HP employee would ever get to see this. My aunt, who lives in far-away London, asked me for help choosing her new computer. Her requirements from the new computer are dead simple: it should have dual screen support (she’s a translator and prefers the original and translated text open side-by-side) and preferably the chassis should have something purple on it. That’s it. I reckon if she wasn’t in London I’d pop by one of the small computer shops on the street near my home and pick up a brandless computer to fit her needs. I reckon I could do fine with the equivalent of something around 250GBP.

Since she’s far away and since my mom has a surprisingly long-lived Dell OptiPlex GX260 (Pentium 4!) and since I have a friend whose been badgering me for years that bitsie computers are bullocks and that going with a brand, preferably Dell (in his opinion) is the only sensible option, I asked my aunt if she’d like to shell the extra money for a brand name and she agreed. So I went to Dell UK’s stupid website, which keeps shoving deals I don’t want in my face while not letting me understand really what is it that I’m going to buy and for what price. Admittedly, I haven’t really bought a computer I cared much about in years, but still I thought that I knew my way around computers, and for the life of me I couldn’t be certain what’s the cheapest Dell with to which you can connect two monitors, and that’s all the dude ever wanted anyway.

I clicked the banner which was flying on the screen (you think I’m kidding, but there was a banner and it was flying) which read “Chat with a sales representative now”. As I expected I was taken to a chat with some poor sap who seemed intent on failing the Turing test. Honestly, I think Dr. Sbaitso would have helped me better than this person did1. After specifying my extremely simple requirements and after receiving several links that didn’t work because the Dell website was built by faceless and nameless gnomes who obviously never heard of beautiful URLs, I literally had to force the sales rep to give me just the name of the model they recommended and Google it. Namely, turns out I was looking for an Dell Inspiron 560 MT (model number: D005619, if the link won’t work for you guys either), priced at a whooping 500GBP. The computer was an obvious overkill for my needs, and still I couldn’t find a proper specification sheet saying simply that it had dual monitor support. I didn’t feel like I could trust the sales rep anyway, I felt our interests were horribly misaligned and he really couldn’t care. Every other sentence the sales rep uttered was “I hope I helped you today sir, would you like to proceed with me to checkout?” or something to that effect.

So I tried figuring things out myself, going backwards, first choosing the cheapest Dell with an ATI GPU (a Radeon 3000), and then searching about the Radeon 3000, and the AMD 760G chipset, and other (non-Dell) manufacturers that make boards with the 760G – all in order to find a picture of the board where I could see what will I be receiving, to no avail. There are many specs lying around the Internet and many boards and some had both DVI/VGA and some didn’t and I couldn’t come up with a simple, definite answer for my aunt, who’se a really kind person but would really rather not donate 500GBP to Dell needlessly. Amazingly, Dell thinks it makes sense to add a sentence like NOTE: Offerings may vary by region. For more information regarding the configuration of your computer, click Start -> Help and Support and select the option to view information about your computer. to a spec sheet. Good thing RFC791 didn’t read [the Identification field is] an identifying value assigned by the sender to aid in assembling the fragments of a datagram, but it may vary by region and be something completely different, or may not. We’ll see, or else the friggin’ Internet probably wouldn’t have existed as we know it.

Visiting HP’s website yielded frighteningly similar results. I’m pretty sure I’d tell her to go Apple by now, but she’s far away and OSX is a whole new interface and I don’t have much time these days for support calls and she needs good Hebrew support (for the translations), which I’m afraid Apple lacks. Besides, Apple scares me, and that’s a reason for another post but not now. So I’m stuck. Ya hear that, fatso Dell and HP boneheaded executives? I’m stuck! There’s a guy here who literally grew up with computers and probably bought more than a hundred of them over the years, he wants to give you money to finance your unjustified bonuses, and he’s unable to, because you couldn’t keep your product lines reasonably small, your model names and submodel number sensible (and sub-sub-model number, and sub-sub-sub-model number, repeat at stupidum), your specifications sheets straight, your website functional and your sales representatives earning more than the equivalent of 10 pennies an hour! What the heck is wrong with you guys? How hard do you think it is to sell me a computer?

Screw this dude, let’s go bowling. But what will I tell my aunt?!

1 <evil>Suddenly I’m thinking that was I tasked with covert development of software that would pass the Turing test, I’d probably either train it against these ‘sales reps’ or try to get it a job as such a sales rep. Hmm.</evil>

zsh and virtualenv

2010/10/14 § 8 Comments

A week ago or so I finally got off my arse and did the pragmatic programmer thing, setting aside those measly ten minutes to check out virtualenv (well, I also checked out buildout, but I won’t discuss it in this post). I knew pretty much what to expect, but I wanted to get my hands dirty with them so I could see what I assumed I’ve been missing out on for so long (and indeed I promptly kicked myself for not doing it sooner, yada yada, you probably know the drill about well-known-must-know-techniques-and-tools-that-somehow-you-don’t-know). Much as I liked virtualenv, there were two things I didn’t like about environment activation in virtualenv. First, I found typing ‘source bin/activate’ (or similar) cumbersome, I wanted something short and snazzy that worked regardless of where inside the virtualenv I am so long as I’m somewhere in it (it makes sense to me to say that I’m ‘in’ a virtualenv when my current working directory is somewhere under the virtualenv’s directory). Note that being “in” a virtualenv isn’t the same as activating it; you can change directory from virtualenv foo to virtualenv bar, and virtualenv foo will remain active. Indeed, this was the second problem I had: I kept forgetting to activate my virtualenv as I started using it or to deactivate the old one as I switched from one to another.

zsh to the rescue. You may recall that I already mentioned the tinkering I’ve done to make it easier to remember my current DVCS branch. Briefly, I have a function called _rprompt_dvcs which is evaluated whenever zsh displays my prompt and if I’m in a git/Mercurial repository it sets my right prompt to the name of the current branch in blue/green. You may also recall that while I use git itself to tell me if I’m in a git repository at all and which branch I’m at (using git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/(\1)/'), I had to resort to a small C program (fast_hg_root) in order to decide whether I’m in a Mercurial repository or not and then I manually parse the branch with cat. As I said in the previous post about hg and prompt, I’m not into giving hg grief about speed vs. git, but when it comes to the prompt things are different.

With this background in mind, I was perfectly armed to solve my woes with virtualenv. First, I changed fast_hg_root to be slightly more generic and search for a user-specified “magic” filename upwards from the current working directory (I called the outcome walkup, it’s really simple and nothing to write home about…). For example, to mimic fast_hg_root with walkup, you’d run it like so: $ walkup .hg. Using $ walkup bin/activate to find my current virtualenv (if any at all), I could easily add the following function to my zsh environment:

act () {
        if [ -n "$1" ]
                if [ ! -d "$1" ]
                        echo "act: $1 no such directory"
                        return 1
                if [ ! -e "$1/bin/activate" ]
                        echo "act: $1 is not a virtualenv"
                        return 1
                if which deactivate > /dev/null
                cd "$1"
                source bin/activate
                virtualenv="$(walkup bin/activate)" 
                if [ $? -eq 1 ]
                        echo "act: not in a virtualenv"
                        return 1
                source "$virtualenv"/bin/activate

Now I can type $ act anywhere I want in a virtualenv, and that virtualenv will become active; this saves figuring out the path to bin/activate and ending up typing something ugly like $ source ../../bin/activate. If you want something that can work for you without a special binary on your host, there’s also a pure-shell version of the same function in the collapsed snippet below.

function act() {
    if [ -n "$1" ]; then
        if [ ! -d "$1" ]; then
            echo "act: $1 no such directory"
            return 1
        if [ ! -e "$1/bin/activate" ]; then
            echo "act: $1 is not a virtualenv"
            return 1

        if which deactivate > /dev/null; then
        cd "$1"
        source bin/activate
        while [ ! -f bin/activate ]; do
            if [ $(pwd) = / ]; then
                echo "act: not in a virtualenv"
                cd "$stored_dir"
                return 1
            cd ..
        source bin/activate
        cd "$stored_dir"

This was nice, but only solved half the problem: I still kept forgetting to activate the virtualenv, or moving out of a virtualenv and forgetting that I left it activated (this can cause lots of confusion, for example, if you’re simultaneously trying out this, this, this or that django-facebook integration modules, more than one of them thinks that facebook is a good idea for a namespace to take!). To remind me, I wanted my left prompt to reflect my virtualenv in the following manner (much like my right prompt reflects my current git/hg branch if any):

  1. If I’m not in a virtualenv and no virtualenv is active, do nothing.
  2. If I’m in a virtualenv and it is not active, display its name as part of the prompt in white.
  3. If I’m in a virtualenv and it is active, display its name as part of the prompt in green.
  4. If I’m not in a virtualenv but some virtualenv is active, display its name in yellow.
  5. Finally, if I’m in one virtualenv but another virtualenv is active, display both their names in red.

So, using walkup, I wrote the virtualenv parsing functions:

function active_virtualenv() {
    if [ -z "$VIRTUAL_ENV" ]; then
        # not in a virtualenv

    basename "$VIRTUAL_ENV"

function enclosing_virtualenv() {
    if ! which walkup > /dev/null; then
    virtualenv="$(walkup bin/activate)"
    if [ -z "$virtualenv" ]; then
        # not in a virtualenv

    basename $(grep VIRTUAL_ENV= "$virtualenv"/bin/activate | sed -E 's/VIRTUAL_ENV="(.*)"$/\1/')

All that remained was to change my lprompt function to look like so (remember I have setopt prompt_subst on):

function _lprompt_env {
    local active="$(active_virtualenv)"
    local enclosing="$(enclosing_virtualenv)"
    if [ -z "$active" -a -z "$enclosing" ]; then
        # no active virtual env, no enclosing virtualenv, just leave
    if [ -z "$active" ]; then
        local color=white
        local text="$enclosing"
        if [ -z "$enclosing" ]; then
            local color=yellow
            local text="$active"
        elif [ "$enclosing" = "$active" ]; then
            local color=green
            local text="$active"
            local color=red
            local text="$active":"$enclosing"
    local result="%{$fg[$color]%}${text}$rst "
    echo -n $result

function lprompt {
    local col1 col2 ch1 ch2

    local _env='$(_lprompt_env)'

    local col_b col_s

$ch2$rst \
$col2%#$rst "

A bit lengthy, but not very difficult. I suffered a bit until I figured out that I should escape the result of _lprompt_virtualenv using a percent sign (like so: "%{$fg[$color]%}${text}$rst "), or else the ANSII color escapes are counted for cursor positioning purposes and screw up the prompt’s alignment. Meh. Also, remember to set VIRTUAL_ENV_DISABLE_PROMPT=True somewhere, so virtualenv’s simple/default prompt manipulation functionality won’t kick in and screw things up for you, and we’re good to go.

The result looks like so (I still don’t know how to do a terminal-“screenshot”-to-html, here’s a crummy png):

Voila! Feel free to use these snippets, and happy zshelling!

Eulogy to a server

2010/10/01 § 2 Comments

You don’t know it, but I’ve started writing this blog several times before it actually went live, and every time I scraped whatever post I started with (the initial run was on I just didn’t think these posts were all too interesting, they were about my monstrous home server, donny. Maybe this is still not interesting, but I’m metaphorically on the verge of tears and I just have to tell someone of what happened, to repent me of my horrible sin. You may not read if you don’t want to. I bought donny about 2.5-3 years ago, to replace my aging home storage server (I had about 3x250GB at the time, no RAID). There’s not much to say about donny’s hardware (Core 2 Duo, 2GB of RAM, Asus P5K-WS motherboard), other than the gargantuan CoolerMaster Stacker 810 chassis with room for 14 (!) SATA disks. Initially I bought 8×0.5TB SATA Hitachi disks for it, and added more as I had the chance. I guess I bought it because at the time I’d hang around disks all day long, I must’ve felt the need to compensate for something (my job at the time was mostly around software, but still, you can’t ignore the shipping crates of SATA disks in lab).

Anyway, most of its life donny ran OpenSolaris. One of our customers had a big ZFS deployment, I’ve always liked Solaris most of all the big Unices (I never thought it really better than Linux, it just Sucked Less™ than the other big iron Unices), I totally drank the cool-aid about “ZFS: the last word in File System” (notice how the first Google hit for this search term is “Bad Request” :) and dtrace really blew me away. So I chose OpenSolaris. Some of those started-but-never-finished posts were about whether I was happy with OpenSolaris and ZFS or not, I never found them interesting enough to even finish them. So even if I don’t wanna discuss that particularly, it should be noted that if we look at how I voted with my feet, I ended up migrating donny to Ubuntu 10.04.1/mdadm RAID5/ext4 when my wife and I got back from our long trip abroad.

Migration was a breeze, the actual migration process convinced me I’ve made the right choice in this case. Over the time with ZFS (both at work and at home) I realized it’s probably good but certainly not magical and not the end of human suffering with regard to storage. In exchange for giving up zfs and dtrace I received the joys of Ubuntu, most notably a working package management system and sensible defaults to make life so much easier, along with the most vibrant eco-system there is. I bought donny 4×2.0TB SATA WD Cavier Green disks, made a rolling upgrade for the data while relying on zfs-fuse (it went well, despite a small and old bug) and overall the downtime was less than an hour for the installation of the disks. At the time of the disaster, donny held one RAID5 array made of 4x2TB, one RAID5 array made of 4x.5TB, one soon-to-be-made RAID5 array made of 3x1TB+1×1.5TB (I bought a larger drive after one of the 1TB failed a while ago), and its two boot disks. I was happy. donny, my wife and me, one happy family. Until last night.

I was always eyeing donny’s small boot disks (what a waste of room… and with all these useless SATA disks I’ve accumulated over the years and have lying about…), so last night I wanted to break the 2x80GB mirror and roll-upgrade to a 2x1TB boot configuration, planning on using the extra space for… well, I’ll be honest, I don’t know for what. I’ll admit it – I got a bit addicted to seeing the TB suffix near the free space column of df -h at home (at work you can see better suffixes :). I just have hardware lying around, and I love never deleting ANYTHING, and I love keeping all the .isos of everything ever (Hmm… RHEL3 for Itanium… that must come in handy some day…) and keeping an image of every friend and relative’s Windows computer I ever fixed (it’s amazing how much time this saves), and never keeping any media I buy in plastic… and, well, the fetish of just having a big server. Heck, it sure beats farmville.

So, indeed, last night I broke that mirror, and installed that 1TB drive, and this morning I started re-mirroring the boot, and while I was at it I started seeing some of the directory structure was wrong so I redistributed stuff inside the RAIDs, and all the disks where whirring merrily at the corner of the room, and I was getting cold so I turned off the AC, and suddenly donny starts beeping (I didn’t even remember I installed that pcspkr script for mdadm) and I get a flurry of emails regarding disks failures in the md devices. WTF? Quickly I realized that donny was practically boiling hot (SMART read one of the disks at 70 degrees celsius), at which point I did an emergency shutdown and realized… that last night I disconnected the power cable running from the PSU to several fans, forgot to reconnect it, and now I’ve effectively cooked my server. Damn.

I’m not sure what to do now. I still have some friends who know stuff about harddisks (like, know the stuff you have to sign NDAs with the disk manufacturers in order to know), and I’m trying to pick my network about what to do next. Basically, from what I hear so far, I should keep donny off, let the disks cool down, be ready with lots of room on a separate host to copy stuff out of it, boot it up in a cool room, take the most critical stuff out and then do whatever, it doesn’t matter, cuz the disks are dead even if they seem alive. I’m told never to trust any of the disks that were inside during the malfunction (that’s >$1,000USD worth of disks…), once a disk reached 70 degrees, even far less, don’t get near it, even if it’s new. Admittedly, these guys are used to handling enterprise disk faults, where $1,000USD in hardware costs (and even many many times that amount) is nothing compared to data loss, but this is the gist of what I’m hearing so far. If you have other observations, let me know. It’s frustratingly difficult to get reliable data about disk failures on the Internet; I know just what to do in case of logical corruption of any sort; but I don’t know precisely what to do in a case like this, and in case of a controller failure, and a head crash, and so on, and so forth. I know it’s a lot about luck, but what’s the best way to give donny the highest chance of survival?

On a parting note, I’ll add that I’m a very sceptic kind of guy, but when it comes to these things I’m rather mystical. It all stems from my roots as a System Administrator; what else can comfort a lonely 19-year-old sysadmin trying to salvage something from nothing in a cold server room at 03:27AM on a Saturday? So now I blame all of this for the name I gave donny. I named it so because I name all my hosts at home after characters from Big Lebowski (I’m typing this on Dude, my laptop), and I called the server donny. The email address I gave it (so it could send me those FzA#$%! S.M.A.R.T reports it never did!) was named Theodore Donald Kerabatsos. The backup server, which is tiny compared to donny and doesn’t hold nearly as much stuff as I’d like to have back now, is called Francis Donnelly. The storage pools (and then RAID volumes) were called folgers, receptacle, modest and cookies (if you don’t understand, you should have paid more attention to Donny’s funeral in The Big Lebowski). And, indeed, as I predicted without knowing it, it ended up being friggin’ cremated. May Stallman bless its soul.

I guess I’m a moron for not thinking about this exact scenario; I was kinda assuming smartmontools and would be SMART (ha!) enough to shutdown when the disks reach 50 degrees, and maybe there is such a setting and I just didn’t enable it… I guess by now it doesn’t matter. I’m one sad hacker. I can’t believe I did this to myself.

Python’s Innards: Hello, ceval.c!

2010/09/02 § 5 Comments

The “Python’s Innards” series owes its existence, at least in part, to hearing one of the Python-Fu masters in my previous workplace say something about a switch statement so large that it was needed to break it up just so some compilers won’t choke on it. I remember thinking then: “Choke the compiler with a switch? Hrmf, let me see that code.” Turns out that this switch can be found in ./Python/ceval.c: PyEval_EvalFrameEx and it switches over the current opcode, invoking its implementation. If I had to summarize all of CPython into one line, I’d probably choose that switch (actually I’d refuse, but humour me by assuming I was at gunpoint or something). This choice is rather subjective, as arguably there are more complex/interesting bits in Python’s object system (explored here and there) or parser/compiler related code. But I can’t help seeing that line, and its surrounding function and file, as the ‘do-work’ heart of CPython.

The reason I didn’t start the series from this heart is that I thought it would be too hard (mostly for the author…). Thanks to what we (well, at least I) learned in the previous posts, I think we can now understand it quite well. I’ll try to link backwards as necessary throughout the article, but if you haven’t followed the series so far, you’d probably do much better if you went back and read some of the previous articles before tackling this one. Also, for brevity’s sake in this post, I won’t qualify the file ./Python/ceval.c and the function PyEval_EvalFrameEx in it. Finally, remember that usually in the series when I quote code, I may note that I edited it, and in that case I often prefer clarity and brevity over accuracy; this is true for this post as well, only much more so, excerpts here might bear only slight resemblance to the real code.

So, where were we… Ah, yes, monstrous switch statement. Well, as I said, this switch can be found in the rather lengthy file ceval.c, in the rather lengthy function PyEval_EvalFrameEx, which takes more than half the file’s lines (it’s roughly 2,250 lines, the file is about 4,400). PyEval_EvalFrameEx implements CPython’s evaluation loop, which is to say that it’s a function that takes a frame object and iterates over each of the opcodes in its associated code object, evaluating (interpreting, executing) each opcode within the context of the given frame (this context is chiefly the associated namespaces and interpreter/thread states). There’s more to ceval.c than PyEval_EvalFrameEx, and we may discuss some of the other bits later in this post (or perhaps a follow-up post), but PyEval_EvalFrameEx is obviously the most important part of it.

Having described the evaluation loop in the previous paragraph, let’s see what it looks like in C (edited):

PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)
    /* variable declaration and initialization stuff */
    for (;;) {
        /* do periodic housekeeping once in a few opcodes */
        opcode = NEXTOP();
        if (HAS_ARG(opcode)) oparg = NEXTARG();
        switch (opcode) {
            case NOP:
                goto fast_next_opcode;
            /* lots of more complex opcode implementations */
                /* become rather unhappy */
        /* handle exceptions or runtime errors, if any */
    /* we are finished, pop the frame stack */
    tstate->frame = f->f_back;
    return retval;

As you can see, iteration over opcodes is infinite (forever: fetch next opcode, do stuff), breaking out of the loop must be done explicitly. CPython (reasonably) assumes that evaluated bytecode is correct in the sense that it terminates itself by raising an exception, returning a value, etc. Indeed, if you were to synthesize a code object without a RETURN_VALUE at its end and execute it (exercise to reader: how?1), you’re likely to execute rubbish, reach the default handler (raises a SystemError) or maybe even segfault the interpreter (I didn’t check this thoroughly, but it looks plausible).

The evaluation loop may look fairly simple so far, but I kept back an important piece: I snipped about 1,450 lines of opcode implementations from within that big switch, all of them presumably more complex than a NOP. In order for you to be able to get a feel for what more serious opcode implementations look like, here’s the (edited) implementation of three more opcodes, illustrating a few more principles:

            case BINARY_SUBTRACT:
                w = *--stack_pointer; /* value stack POP */
                v = stack_pointer[-1];
                x = PyNumber_Subtract(v, w);
                stack_pointer[-1] = x; /* value stack SET_TOP */
                if (x != NULL) continue;
            case LOAD_CONST:
                x = PyTuple_GetItem(f->f_code->co_consts, oparg);
                *stack_pointer++ = x; /* value stack PUSH */
                goto fast_next_opcode;
            case SETUP_LOOP:
            case SETUP_EXCEPT:
            case SETUP_FINALLY:
                PyFrame_BlockSetup(f, opcode, INSTR_OFFSET() + oparg,

We see several things. First, we see a typical value manipulation opcode, BINARY_SUBTRACT. This opcode (and many others) works with values on the value stack as well as with a few temporary variables, using CPython’s C-API abstract object layer (in our case, a function from the number-like object abstraction) to replace the two top values on the value stack with the single value resulting from subtraction. As you can see, a small set of temporary variables, such as v, w and x are used (and reused, and reused…) as the registers of the CPython VM. The variable stack_pointer represents the current bottom of the stack (the next free pointer in the stack). This variable is initialized at the beginning of the function like so: stack_pointer = f->f_stacktop;. In essence, together with the room reserved in the frame object for that purpose, the value stack is this pointer. To make things simpler and more readable, the real (unedited by me) code of ceval.c defines several value stack manipulation/observation macros, like PUSH, TOP or EMPTY. They do what you imagine from their names.

Next, we see a very simple opcode that loads values from somewhere into the valuestack. I chose to quote LOAD_CONST because it’s very brief and simple, although it’s not really a namespace related opcode. “Real” namespace opcodes load values into the value stack from a namespace and store values from the value stack into a namespace; LOAD_CONST loads constants, but doesn’t fetch them from a namespace and has no STORE_CONST counterpart (we explored all this at length in the article about namespaces). The final opcode I chose to show is actually the single implementation of several different control-flow related opcodes (SETUP_LOOP, SETUP_EXCEPT and SETUP_FINALLY), which offload all details of their implementation to the block stack manipulation function PyFrame_BlockSetup; we discussed the block stack in our discussion of interpreter stacks.

Something we can observe looking at these implementations is that different opcodes exit the switch statement differently. Some simply break, and let the code after the switch resume. Some use continue to start the for loop from the beginning. Some goto various labels in the function. Each exit has different semantic meaning. If you break out of the switch (the ‘normal’ route), various checks will be made to see if some special behaviour should be performed – maybe a code block has ended, maybe an exception was raised, maybe we’re ready to return a value. Continuing the loop or going to a label lets certain opcodes take various shortcuts; no use checking for an exception after a NOP or a LOAD_CONST, for instance.

That’s pretty much it. I can’t really say we’re done (not at all), but this is pretty much the gist of PyEval_EvalFrameEx. Simple, eh? Well, yeah, simple, but I lied a bit with the editing to make it simpler. For example, if you look at the code itself, you will see that none of the case expressions for the big switch are really there. The code for the NOP opcode is actually (remember this series is about Python 3.x unless noted otherwise, so this snippet is from Python 3.1.2):


TARGET? FAST_DISPATCH? What are these? Let me explain. Things may become clearer if we’d look for a moment at the implementation of the NOP opcode in ceval.c of Python 2.x. Over there the code for NOP looks more like the samples I’ve shown you so far, and it actually seems to me that the code of ceval.c gets simpler and simpler as we look backwards at older revisions of it. The reason is that although I think PyEval_EvalFrameEx was originally written as a really exceptionally straightforward piece of code, over the years some necessary complexity crept into it as various optimizations and improvements were implemented (I’ll collectively call them ‘additions’ from now on, for lack of a better term).

To further complicate matters, many of these additions are compiled conditionally with preprocessor directives, so several things are implemented in more than one way in the same source file. In the larger code samples I quoted above, I liberally expanded some preprocessor directives using their least complex expansion. However, depending on compilation flags, these and other preprocessor directives might expand to something else, possibly a more complicated something. I can understand trading simplicity to optimize a tight loop which is used very often, and the evaluation loop is probably one of the more used loops in CPython (and probably as tight as its contributors could make it). So while this is all very warranted, it doesn’t help the readability of the code.

Anyway, I’d like to enumerate these additions here explicitly (some in more depth than others); this should aid future discussion of ceval.c, as well as prevent me from feeling like I’m hiding too many important things with my free spirited editing of quoted code. Fortunately, most if not all these additions are very well commented -actually, some of the explanations below will be just summaries or even taken verbatim from these comments, as I believe that they’re accurate (eek!). So, as you read PyEval_EvalFrameEx (and indeed ceval.c in general), you’re likely to run into any of these:

“Threaded Code” (Computed-GOTOs)

Let’s start with the addition that gave us TARGET, FAST_DISPATCH and a few other macros. The evaluation loop uses a “switch” statement, which decent compilers optimize as a single indirect branch instruction with a lookup table of addresses. Alas, since we’re switching over rapidly changing opcodes (it’s uncommon to have the same opcode repeat), this would have an adverse effect on the success rate of CPU branch prediction. Fortunately gcc supports the use of C-goto labels as values, which you can generally pass around and place in an array (restrictions apply!). Using an array of adresses in memory obtained from labels, as you can see in ./Python/opcode_targets.h, we create an explicit jump table and place an explicit indirect jump instruction at the end of each opcode. This improves the success rate of CPU prediction and can yield as much as 20% boost in performance.

Thus, for example, the NOP opcode is implemented in the code like so:


In the simpler scenario, this would expand to a plain case statement and a goto, like so:

        case NOP:
            goto fast_next_opcode;

But when threaded code is in use, that snippet would expand to (I highlighted the lines where we actually move on to the next opcode, using the dispatch table of label-values):

            opcode = NOP;
            if (HAS_ARG(NOP))
                oparg = NEXTARG();
        case NOP:
                if (!_Py_TracingPossible) {
                    f->f_lasti = INSTR_OFFSET();
                    goto *opcode_targets[*next_instr++];
                goto fast_next_opcode;

Same behaviour, somewhat more complicated implementation, up to 20% faster Python. Nifty.

Opcode Prediction

Some opcodes tend to come in pairs. For example, COMPARE_OP is often followed by JUMP_IF_FALSE or JUMP_IF_TRUE, themselves often followed by a POP_TOP. What’s more, there are situations where you can determine that a particular next-opcode can be run immediately after the execution of the current opcode, without going through the ‘outer’ (and expensive) parts of the evaluation loop. PREDICT (and a few others) are a set of macros that explicitly peek at the next opcode and jump to it if possible, shortcutting most of the loop in this fashion (i.e., if (*next_instr == op) goto PRED_##op). Note that there is no relation to real hardware here, these are simply hardcoded conditional jumps, not an exploitation of some mechanism in the underlying CPU (in particular, it has nothing to do with “Threaded Code” described above).

Low Level Tracing

An addition primarily geared towards those developing CPython (or suffering from a horrible, horrible bug). Low Level Tracing is controlled by the LLTRACE preprocessor name, which is enabled by default on debug builds of CPython (see --with-pydebug). As explained in ./Misc/SpecialBuilds.txt: when this feature is compiled-in, PyEval_EvalFrameEx checks the frame’s global namespace for the variable __lltrace__. If such a variable is found, mounds of information about what the interpreter is doing are sprayed to stdout, such as every opcode and opcode argument and values pushed onto and popped off the value stack. Not useful very often, but very useful when needed.

This is the what the low level trace output looks like (slightly edited):

>>> def f():
...     global a
...     return a - 5
>>> dis(f)
  3           0 LOAD_GLOBAL              0 (a) 
              3 LOAD_CONST               1 (5) 
              6 BINARY_SUBTRACT      
              7 RETURN_VALUE         
>>> exec(f.__code__, {'__lltrace__': 'foo', 'a': 10})
0: 116, 0
push 10
3: 100, 1
push 5
6: 24
pop 5
7: 83
pop 5
# trace of the end of exec() removed

As you can guess, you’re seeing a real-time disassembly of what’s going through the VM as well as stack operations. For example, the first line says: line 0, do opcode 116 (LOAD_GLOBAL) with the operand 0 (expands to the global variable a), and so on, and so forth. This is a bit like (well, little more than) adding a bunch of printf calls to the heart of VM.

Advanced Profiling

Under this heading I’d like to briefly discuss several profiling related additions. The first relies on the fact that some processors (notably Pentium descendants and at least some PowerPCs) have built-in wall time measurement capabilities which are cheap and precise (correct me if I’m wrong). As an aid in the development of a high-performance CPython implementation, Python 2.4’s ceval.c was instrumented with the ability to collect per-opcode profiling statistics using these counters. This instrumentation is controlled by the somewhat misnamed --with-tsc configuration flag (TSC is an Intel Pentium specific name, and this feature is more general than that). Calling sys.settscdump(True) on an instrumented interpreter will cause the function ./Python/ceval.c: dump_tsc to print these statistics every time the evaluation loop loops.

The second advanced profiling feature is Dynamic Execution Profiling. This is only available if Python was built with the DYNAMIC_EXECUTION_PROFILE preprocessor name. As ./Tools/scripts/ says, [this] will tell you which opcodes have been executed most frequently in the current process, and, if Python was also built with -DDXPAIRS, will tell you which instruction _pairs_ were executed most frequently, which may help in choosing new instructions. One last thing to add here is that enabling Dynamic Execution Profiling implicitly disables the “Threaded Code” addition.

The third and last addition in this category is function call profiling, controlled by the preprocessor name CALL_PROFILE. Quoting ./Misc/SpecialBuilds.txt again: When this name is defined, the ceval mainloop and helper functions count the number of function calls made. It keeps detailed statistics about what kind of object was called and whether the call hit any of the special fast paths in the code.

Extra Safety Valves

Two preprocessor names, USE_STACKCHECK and CHECKEXC include extra assertions. Testing an interpreter with these enabled may catch a subtle bug or regression, but they are usually disabled as they’re too expensive.

These are the additions I found, grepping ceval.c for #ifdef. I think we’ll call it a day here, although we’re by no means finished. For example, I’d like to devote a separate post to exceptions, which is where we can discuss the tail of the evaluation loop (everything after the big switch and before the end of the big for), which we merely skimmed today. I’d also like to devote a whole post to locking and synchronization (including the GIL), which we touched upon before but never covered properly. Last but really not least, there’s about 2,000 other lines in ceval.c which we didn’t cover today; none of them are as important as PyEval_EvalFrameEx, but we need to talk at least about some of them.

All these things taken into account, I think we can say that today we finally conquered the evaluation loop. This isn’t the end of the series, far from it, but I do see it as a milestone. “Hooray”, I believe the saying goes. I hope you’re enjoying the show, thanks for the supportive comments (they keep me going), and I’ll see you in the next post.

I would like to thank Nick Coghlan for reviewing this article; any mistakes that slipped through are my own.

1Lazy or timid readers may choose to defer to Nick Coghlan’s example of one way he did it; I urge you not to look there and solve it on your own, it’s rather easy.

The Curious Case of HID Malfunction

2010/08/21 § Leave a comment

A quick tidbit for any interested hardware wizards out there (I know no one is likely to care, this is really more of an excuse for why the next Python’s Innards post is progressing slowly). As some of you know, I’m currently on a long trip with my wife (a trip which is already nearing its end…). This means I’m rather poor in hardware, and that sometimes the environment is harsh – hot, cold, humid, occasionally vibrating (flights, boats), rich with small particles (sand, dust), etc. As I expected for a long time, the elements finally had their toll on my small but until now trusty Asus EeePC 1005HA. The thing is, the toll was taken in a rather odd way.

For about three days now the builtin trackpad stopped working – most of the time. Usually I get no cursor movement nor clicks, and there are no particular messages in dmesg//var/log/messages. On rare occasions the trackpad resumes working for a bit, I wasn’t able to find a pattern in what makes it work for these short periods of time (heating, cooling, sleeping, booting… nothing seems to make the short ‘work-periods’ predictable). On one occasion the trackpad worked but behaved erratically (jumping around, random clicks, etc), on others it works fine, but for a few seconds and up to a few minutes. I’m running Ubuntu 10.04, kernel 2.6.32-24, keeping it reasonably apt-get updated. I didn’t change anything significant in the software configuration of the computer before this happened, and booting a vanilla 10.04 from a USB stick I have around doesn’t help, so I’m pretty sure it’s not a vanilla software issue (despite the oddity listed below).

This is patently unpleasent but not entirely odd, and I would chalk it down to some undefined hardware damage and let it be. I could buy an external mouse for the remaining few weeks of the trip and otherwise ignore the issue, lest the builtin keyboard started showing similar behaviour. It works far more often than the mouse, but has spells of brokeness. An external USB keyboard works fine when plugged in. I don’t even know if my internal keyboard interfaces via some kind of internal USB controller or not; seems not, as even when it’s working it’s not listed in lsusb -v. /proc/bus/input/devices lists an “AT Translated Set 2 keyboard”, but I have no idea if this is really my keyboard or not. Anyway, the really weird thing is that the keyboard’s broken behaviour has a few extra odd quirks:

  • It works perfectly prior to loading the kernel: in the BIOS configuration screen, or GRUB’s menu, or the USB stick’s boot menu. It seems that as soon as the kernel is loaded, no more keyboard (X11 or console).
  • The “special” EeePC keys, like toggling wifi or changing screen brightness, work perfectly. They aren’t special keys, but rather a key combination, and the keys used in the combination don’t reach the OS discreetly.
  • When I open the laptop’s lid in sleep, I need to hit a key to bring it out of sleep. Any key works well enough for the computer to wakeup, and the very same key (or any other key) will promptly stop working when the OS is awake enough to ask for a password.

So what gives? My keyboard isn’t broken, but some kind of interface between the keyboard and the system which is circumvented by the BIOS but is used by the kernel is broken? Huh, WTF?

The bit of Googling I did yielded nothing, Internet here isn’t really scarce but it isn’t abundant and sure is not fast or pleasent (I’m on a beach in Thailand at the moment). I’m left with a big WTF and apt-get install keynav. Any tips will be greatly appreciated (and speed up the next post in the Python’s Innards series, too!).

Update: I’ve decided to disassemble and reassemble the keyboard, following these instructions, using a swiss-army knife, my wife’s fingernail file and a camping torch. Following the work both keyboard and touchpad are working for about 10 minutes now, one of the longer durations in the past few days. I can only hope I fixed the problem. Either way, I’m curious why the keyboard consistently didn’t work with a loaded kernel yet seemed to work fine using the BIOS (in the BIOS’ setup, GRUB’s boot menu, etc). Any explanations?

Hacker irked by reincarnation

2010/08/16 § 20 Comments

Today I chose a rather peculiar topic for a technology blog: the history of reincarnation research and its implications on science. This might seem a bit awkward or even off-topic, I think it’s neither (and I make up the rules here). Before we begin, I gather I should say that I’m a sceptic, I’ve always been a sceptic and I never saw myself much as a very spiritual or new-age kind of person. When my wife and I entered a 10-day course about Buddhism about a month ago, you can imagine I arrived with hefty sacks full of various grains of salt to take everything with.

I didn’t know much about Buddhism before the course, and I’m not any kind of an expert about it now, either. This post isn’t about Buddhism at all, actually – just about a small thing I ran into during the course. As it happened the course material mentioned a certain research by a Dr. Ian Stevenson from the University of Virginia, who dedicated much of his career to research of reincarnation. I don’t recall precisely how the wording went, but as best as I can recall the course material took his research to show that reincarnation is scientifically proven. Naturally, that was my queue for the Grand Entrance of the Grains of Salt.

So as soon as we left the course I spent a while Googling the late Dr. Stevenson, his work, and the work of others in the field. What I found was very disturbing: Dr. Stevenson seems, to the best of my ability to assess, as a reasonable researcher with reasonable methods who was never properly refuted. And he found some ‘disturbing’ (i.e., unexplainable, ‘supernatural’) results in his research. However, hardly anyone seems to have noticed or cared and hardly anyone continues his research today (I know Jim Tucker continues his work directly and that other researchers in the past and present also looked at the topic, but overall it seems to me like awfully too little).

OK, fine, so we have established evidence of reincarnation and we’re ignoring them. Uhm, what?! Hello? Am I missing something? Did a respectable member of the scientific community say (for several decades) that he found thousands of cases that are impossible to explain by modern science, cases that should shake our understanding of physics and/or biology and/or psychology and/or whatnot, and the collective response is to ignore this guy? How can the relevant scientific community look itself in the eye? Isn’t this an interesting and important subject? You think proof that actually P=NP would be big? How about friggin’ reincarnation!

I’ve read Bill Bryson’s A Short History of Nearly Everything lately; it has been rather sad to realize so many amazing discoveries and theories in history have been ignored for several decades (sometimes centuries) before anyone caught on to the fact that they’re actually true and significant, often yielding further discoveries and further results. I’m not at all saying the fact that many true discoveries were ignored means that reincarnation is true only on the basis of it being ignored. But whatever evidence Dr. Stevenson (and others) have found should be refuted seriously, not merely shrugged off.

To some extent, things like this undermine my (otherwise very strong) belief in science as a whole. What other grand truths are already known to a few, and we just didn’t hear of them because we are all collectively waiting for a bunch of old Professors to die or for someone to pick up where a deceased researcher left off? What kind of price might humanity be paying because this avenue of research is ‘weird’ or ‘unsexy’? What if rebirthing is possible, and the obvious logical reservations1 are somehow solvable, and we’re just ignoring it?

Call me naive, but I don’t get why people like you and me don’t demand from our governments and universities that this issue be cleared sooner rather than later (and the issues of other, similarly odd scientific observations, while we’re at it; the hairs on the back of my neck still stand when I recall a lecture from Prof. Shulamith Kreitler of TAU’s department of Psychology about long-range sub-awareness and super-awareness phenomena, which to a layman like me it sounded a lot like bloody Telepathy and I think were never thoroughly researched by a Neurologist/Physicist combo, as I think it should have). I can list a ton of reasons why understanding of reincarnation may be important, but above all else – we’re humans, aren’t we? Isn’t curiosity reason enough?

1 Where do new souls come from as the population is expanding? What would happen if we nuke every living being on Earth, where would all these souls go? Do we share souls with aliens? How do the physics of rebirthing work, that is, what kind of particles pass from the dead being to the newborn one? Why is rebirthing evidence so much more common in the East, where it is far more commonly believed to be true?