Python’s Innards: Interpreter Stacks

2010/07/22 § 4 Comments

Those of you who have been paying attention know that this series is spiraling towards what can be considered the core of Python’s Virtual Machine, the “actually do work function” ./Python/ceval.c: PyEval_EvalFrameEx. The (hopefully) last hurdle on our way there is to understand the three significant stack data structures used for CPython's code evaluation: the call stack, the value stack and the block stack (I've called them collectively "Interpreter Stacks" in the title, this isn't a formal term). All three stacks are tightly coupled with the frame object, which will also be discussed today. If you give me a minute to put on my spectacles, I'll read to you what Wikipedia says about call stacks in general: In computer science, a call stack is a stack data structure that stores information about the active subroutines of a computer program... A call stack is composed of stack frames (...). These are machine dependent data structures containing subroutine state information. Each stack frame corresponds to a call to a subroutine which has not yet terminated with a return. Hrmf. Jim, I don't understand... how does this translate to a virtual machine?

Well, since CPython implements a virtual machine, its call stack and stack frames are dependant on this virtual machine, not on the physical machine it's running on. And also, as Python tends to do, this internal implementation detail is exposed to Python code, either via the C-API or pure Python, as frame objects (./Include/frameobject.h: PyFrameObject). We know that code execution in CPython is really the evaluation (interpretation) of a code object, so every frame represents a currently-being-evaluated code object. We'll see (and already saw before) that frame objects are linked to one another, thus forming a call stack of frames. Finally, inside each frame object in the call stack there's a reference to two frame-specific stacks (not directly related to the call stack), they are the value stack and the block stack.

The value stack (you may know this term as an 'evaluation stack') is where manipulation of objects happens when object-manipulating opcodes are evaluated. We have seen the value stack before on various occasions, like in the introduction and during our discussion of namespaces. Recalling an example we used before, BINARY_SUBTRACT is an opcode that effectively pops the two top objects in the value stack, performs PyNumber_Subtract on them and sets the new top of the value stack to the result. Namespace related opcodes, like LOAD_FAST or STORE_GLOBAL, load values from a namespace to the stack or store values from the stack to a namespace. Each frame has a value stack of its own (this makes sense in several ways, possibly the most prominent is simplicity of implementation), we'll see later where in the frame object the value stack is stored.

This leaves us with the block stack, a fairly simple concept with some vaguely defined terminology around it, so pay attention. Python has a notion called a code block, which we have discussed in the article about code objects and which is also explained here. Completely unrelatedly, Python also has a notion of compound statements, which are statements that contain other statements (the language reference defines compound statements here). Compound statements consist of one or more clauses, each made of a header and a suite. Even if the terminology wasn't known to you until now, I expect this is all instinctively clear to you if you have almost any Python experience: for, try and while are a few compound statements.

So where's the confusion? In various places throughout the code, a block (sometimes "frame block", sometimes "basic block") is used as a loose synonym for a clause or a suite, making it easier to confuse suites and clauses with what's actually a code block or vice versa. Both the compilation code (./Python/compile.c) and the evaluation code (./Python/ceval.c) are aware of various suites and have (ill-named) data structures to deal with them; but since we're more interested in evaluation in this series, we won't discuss the compilation-related details much (or at all). Whenever I'll think wording might get confusing, I'll mention the formal terms of clause or suite alongside whatever code term we're discussing.

With all this terminology in mind we can look at what's contained in a frame object. Looking at the declaration of ./Include/frameobject.h: PyFrameObject, we find (comments were trimmed and edited for your viewing pleasure):

typedef struct _frame {
   PyObject_VAR_HEAD
   struct _frame *f_back;   /* previous frame, or NULL */
   PyCodeObject *f_code;    /* code segment */
   PyObject *f_builtins;    /* builtin symbol table */
   PyObject *f_globals;     /* global symbol table */
   PyObject *f_locals;      /* local symbol table */
   PyObject **f_valuestack; /* points after the last local */
   PyObject **f_stacktop;   /* current top of valuestack */
   PyObject *f_trace;       /* trace function */

   /* used for swapping generator exceptions */
   PyObject *f_exc_type, *f_exc_value, *f_exc_traceback;

   PyThreadState *f_tstate; /* call stack's thread state */
   int f_lasti;             /* last instruction if called */
   int f_lineno;            /* current line # (if tracing) */
   int f_iblock;            /* index in f_blockstack */

   /* for try and loop blocks */
   PyTryBlock f_blockstack[CO_MAXBLOCKS];

   /* dynamically: locals, free vars, cells and valuestack */
   PyObject *f_localsplus[1]; /* dynamic portion */
} PyFrameObject;

We see various fields used to store the state of this invocation of the code object as well as maintain the call stack's structure. Both in the C-API and in Python these fields are all prefixed by f_, though not all the fields of the C structure PyFrameObject are exposed in the pythonic representation. I hope some of the fields are intuitively clear to you, since these fields relate to many topics we have already covered. We already mentioned the relation between frame and code objects, so the f_code field of every frame points to precisely one code object. Insofar as structure goes, frames point backwards thus that they create a stack (f_back) as well as point "root-wards" in the interpreter state/thread state/call stack structure by pointing to their thread state (f_tstate), as explained here. Finally, since you always execute Python code in the context of three namespaces (as discussed there), frames have the f_builtins, f_globals and f_locals fields to point to these namespaces. These are the fields (I hope) we already know.

Before we dig into the other fields of a frame object, please notice frames are a variable size Python object (they are a PyObject_VAR_HEAD). The reason is that when a frame object is created it should be dynamically allocated to be large enough to contain references (pointers, really) to the locals, cells and free variables used by its code object, as well as the value stack needed by the code objects 'deepest' branch. Indeed, the last field of the frame object, f_localsplus (locals plus cells plus free variables plus value stack...) is a dynamic array where all these references are stored. PyFrame_New will show you exactly how the size of this array is computed.

If the previous paragraph doesn't sit well with you, I suggest you read the descriptions I wrote for co_nlocals, co_cellvars, co_freevars and co_stacksize - during evaluation, all these 'dead' parts of the inert code object come to 'life' in space allocated at the end of the frame. As we'll probably see in the next article, when the frame is evaluated, these references at the end of the frame will be used to get (or set) "fast" local variables, free variables and cell variables, as well as to the variables on the value stack ("fast" locals was explained when we discussed namespaces). Looking back at the commented declaration above and given what I said here, I believe you should now understand f_valuestack, f_stacktop and f_localsplus.

We can now look at f_blockstack, keeping in mind the terminology clarification from before. As you can maybe imagine, compound statements sometimes require state to be evaluated. If we're in a loop, we need to know where to go in case of a break or a continue. If we're raising an exception, we need to know where is the innermost enclosing handler (the suite of the closest except header, in more formal terms). This state is stored in f_blockstack, a fixed size stack of PyTryBlock structures which keeps the current compound statement state for us (PyTryBlock is not just for try blocks; it has a b_type field to let it handle various types of compound statements' suites). f_iblock is an offset to the last allocated PyTryBlock in the stack. If we need to bail out of the current "block" (that is, the current clause), we can pop the block stack and find the new offset in the bytecode from which we should resume evaluation in the popped PyTryBlock (look at its b_handler and b_level fields). A somewhat special case is a raised exception which exhausts the block stack without being caught, as you can imagine, in that case a handler will be sought in the block stack of the previous frames on the call stack.

All this should easily click into place now if you read three code snippets. First, look at this disassembly of a for statement (this would look strikingly similar for a try statement):

>>> def f():
...     for c in 'string':
...             my_global_list.append(c)
...
>>> diss(f)
 2           0 SETUP_LOOP              27 (to 30)
             3 LOAD_CONST               1 ('string')
             6 GET_ITER
       >>    7 FOR_ITER                19 (to 29)
            10 STORE_FAST               0 (c)

 3          13 LOAD_GLOBAL              0 (my_global_list)
            16 LOAD_ATTR                1 (append)
            19 LOAD_FAST                0 (c)
            22 CALL_FUNCTION            1
            25 POP_TOP
            26 JUMP_ABSOLUTE            7
       >>   29 POP_BLOCK
       >>   30 LOAD_CONST               0 (None)
            33 RETURN_VALUE
>>>

Next, look at how the opcodes SETUP_LOOP and POP_BLOCK are implemented in ./Python/ceval.c. Notice that SETUP_LOOP and SETUP_EXCEPT or SETUP_FINALLY are rather similar, they all push a block matching the relevant suite unto the block stack, and they all utilize the same POP_BLOCK:

       TARGET_WITH_IMPL(SETUP_LOOP, _setup_finally)
       TARGET_WITH_IMPL(SETUP_EXCEPT, _setup_finally)
       TARGET(SETUP_FINALLY)
       _setup_finally:
           PyFrame_BlockSetup(f, opcode, INSTR_OFFSET() + oparg,
                      STACK_LEVEL());
           DISPATCH();

       TARGET(POP_BLOCK)
           {
               PyTryBlock *b = PyFrame_BlockPop(f);
               UNWIND_BLOCK(b);
           }
           DISPATCH();

Finally, look at the actual implementation of ./Object/frameobject.c: PyFrame_BlockSetup and ./Object/frameobject.c: PyFrame_BlockPop:

void
PyFrame_BlockSetup(PyFrameObject *f, int type, int handler, int level)
{
   PyTryBlock *b;
   if (f->f_iblock >= CO_MAXBLOCKS)
       Py_FatalError("XXX block stack overflow");
   b = &f->f_blockstack[f->f_iblock++];
   b->b_type = type;
   b->b_level = level;
   b->b_handler = handler;
}

PyTryBlock *
PyFrame_BlockPop(PyFrameObject *f)
{
   PyTryBlock *b;
   if (f->f_iblock <= 0)
       Py_FatalError("XXX block stack underflow");
   b = &f->f_blockstack[--f->f_iblock];
   return b;
}

There, now you're smart. If you keep the terminology straight, f_blockstack turns out to be rather simple, at least in my book.

We're left with the rather esoteric fields, some simpler, some a bit more arcane. In the 'simpler' range we have f_lasti, an integer offset into the bytecode of the last instructions executed (initialized to -1, i.e., we didn't execute any instruction yet). This index lets us iterate over the opcodes in the bytecode stream. Heading towards the 'more arcane' area we see f_trace and f_lineno. f_trace is a pointer to a tracing function (see sys.settrace; think implementation of a tracer or a debugger). f_lineno contains the line number of the line which caused the generation of the current opcode; it is valid only when tracing (otherwise use PyCode_Addr2Line). Last but not least, we have three exception fields (f_exc_type, f_exc_value and f_exc_traceback), which are rather particular to generators so we'll discuss them when we discuss that beast (there's a longer comment about these fields in ./Include/frameobject.h if you're curious right now).

On a parting note, we can mention when frames are created. This happens in ./Objects/frameobject.c: PyFrame_New, usually called from ./Python/ceval.c: PyEval_EvalCodeEx (and ./Python/ceval.c: fast_function, a specialized optimization of PyEval_EvalCodeEx). Frame creation occurs whenever a code object should be evaluated, which is to say when a function is called, when a module is imported (the module's top-level code is executed), whenever a class is defined, for every discrete command entered in the interactive interpreter, when the builtins eval or exec are used and when the -c switch is used (I didn't absolutely verify this is a 100% exhaustive list, but it think it's rather complete).

Looking at the list in the previous paragraph, you probably realized frames are created very often, so two optimizations are implemented to make frame creation fast: first, code objects have a field (co_zombieframe) which allows them to remain associated with a 'zombie' (dead, unused) frame object even when they're not evaluated. If a code object was already evaluated once, chances are it will have a zombie frame ready to be reanimated by PyFrame_New and returned instead of a newly allocated frame (trading some memory to reduce the number of allocations). Second, allocated and entirely unused stack frames are kept in a special free-list (./Objects/frameobject.c: free_list), frames from this list will be used if possible, instead of actually allocating a brand new frame. This is all kindly commented in ./Objects/frameobject.c.

That's it, I think. Oh, wait: if you'd like to play with frames in your interpreter, take a look at the inspect module, maybe especially this part of it. In gdb, I used a rather crude method to look at the call stack (I dereferenced the global variable interp_head and went on from there). There's probably a better way, but I didn't bother looking. Now that's really it. In fact, I believe at last we covered enough material to analyze ./Python/ceval.c: PyEval_EvalFrameEx. Ladies and Gentlemen, we can read it. We have the technology.

But, alas, we'll only do it in the next post, and who knows when that will arrive. And until it does, do good, avoid doing bad and keep clearing your mind. Siddhārtha Gautama said that, and I tend to think that if that particular bloke lived today he'd have some serious Python-Fu going for him, so heed his words.


I would like to thank Nick Coghlan for reviewing this article; any mistakes that slipped through are my own.

Python’s Innards: for my wife

2010/07/04 § 1 Comment

The other day the best wife I’ll ever have had trouble sleeping and asked me to tell her something to put her to sleep. Since she’s not quite a hacker, I figured some discussion of what I usually write about may do the trick (okay, maybe ‘not quite a hacker’ is an understatement, she’s an un-hacker-hippie married to a hacker and it actually works out great thanks for asking). I don’t think this article is likely to be actually useful to the usual readers of this blog (unless you have a less-computer-initiated spouse with trouble sleeping), but I hope maybe this particular bedtime story might entertain some of you.

Yaniv: What is my blog about… Let’s start at the beginning. See, a computer is a machine, just a machine, you know, like one with gears and levers and stuff. Like all machines, it’s not smart, but unlike many machines, it’s has lots of pieces and is very very fast. Think about the musical boxes we saw today at the Museum’s souvenir store. They are also machines, but smaller and slower. When you twist the ratchet lever the pin cylinder turns around to make music. But even though the music is great, it isn’t a ‘smart’ machine, it just plays the music it was built to play. A machine with such and such a pin cylinder will play Mozart, and a machine with a different pin cylinder will play Bach. All right?

Inbal: An encouraging nod.

Yaniv: So think about a music box with replacable cylinder. Instead of a cylinder with small pins it could have a… eh… a punched card! A long strip of plastic or cardboard you insert into machine, and this strip has some holes punched in it. When the card is fed into the machine, the music box’s comb’s teeth encounter these holes like the encounter the pins and music is played. Computers are very much like that, only they don’t play music, they punch holes in a fresh card according to the card that was inserted. That’s how you program them, and that’s how they answer. You insert a card with holes arranged in just the right way, and you read the answer encoded as holes in the card the spew out. Not magic, just how the internals of the computer are built, like maybe an abacus controlled with cards. If you restart it ten times and feed it the same card, it will give the same answer each time. It’s a machine.

Inbal: Is that why people like you always tell people like me to restart the computer?

Yaniv: Yes, quite. We want to know exactly what kind of punched cards were fed into the computer from the moment it was last started, so we know what state it is in, so maybe we can actually figure out why it’s behaving like it is. Another encouraging nod, and wide open eyes. My wife shows general interest in my work, but never anything as low-level as this shit. I was wondering how far can we go on this topic. The thing is, you obviously know computers don’t really use punched cards anymore. Many years ago smart people figured out more elegant ways to talk with a computer, so we feed it with keyboards and USB sticks and wifi and we see the output on a screen or a printer. But to the computer, what you type and when you move the mouse and whatever you bring from the Internet or hard drive or USB stick – it all looks another part of an incredibly long series of holes in a gigantic punched card, and what you see on the screen is really just a series of new holes the computer punched as a result of what you fed it so far.

Inbal: Is this what you look at in the black screen with the green letters? The punched card inside the computer?

Yaniv: Uhm, not usually. I don’t need to. If a neurologist gets a head ache, she doesn’t undergo CT, she takes a pill. But even if I hardly ever look into whatever modern computers use to mimic their huge punched cards, I never forget this is how they are really built, and that all the wonderful things a computer can do are really performed by a giant abacus or music box with a series of holes in a long card instead of a cylinder.

Inbal: And that’s what you write about?

Yaniv: No, not yet. See, different computers, like my netbook or your iPhone react differently to cards punched in the same manner. One wouldn’t work with the other, because their internal machinery is different. You can think of it a bit like two music boxes with differently toothed combs. They’d play different music given the same pin cylinder, or, rather, one would play music and the other is likely to make icky noise, not unlike the noise you get when you tune the TV to the wrong channel.

Inbal: I hate that noise. A music box would never make that noise.

Yaniv: If it spun very quickly and had many many different teeth on the comb and had a bad cylinder inserted to it, then it would sound quite like the TV noise. The point is that different computers have different teeth combs and can play different cylinders. Python is a very special such cylinder. Python is a bit like a music box inside a music box. Python, the ‘inner’ music box affects the behaviour of the ‘outer’ music box, which is the computer. Silence. That didn’t go well. Look, you like the different apps I put on your iPhone, right?

Inbal: (frown) Mostly. Ugh. Come on, it was a farting app, I’m just a guy, how was I possibly supposed to resist it?!

Yaniv: Well, imagine an app that displays a music box on the screen and lets you tap on the pin cylinder to make your on music and play it from the iPhone. A music box inside the iPhone. That’s very imaginable, right?

Inbal: Uhm, yeah… that’s a nice idea.

Yaniv: Well, think about the pin cylinder of the music box in that app. If I would write such a program both for the iPhone and for the netbook, it makes sense that I could make it play the same music for the same pin cylinder, even tough it’s running on different computers. Right?

Inbal: Hmm-hmm. Yawn. Yikes, we better cut to the chase.

Yaniv: Well, as I said, and it’s sometimes even hard for me to believe but you have to take my word for it, that the iPhone just a punching machine itself, the music box app is really a punch card I put in the iPhone. So why not make a punching machine app? You could see it on the screen, and punch holes in a visual card, and drag it into the machine with your finger, and see the result coming out on the screen. A punching machine in a punching machine. I’d make a long punched card, this is the punching machine app, and after I feed it I will put in a second punched card, this is already running on the ‘inner’ punching machine, the app. OK?

Inbal: Mostly closed eyes, nod.

Yaniv: Well, so why not make a ‘first’ card for one machine, and a different ‘first’ card to another machine, such that the same ‘second’ card would work the same on both different machines. Just like I could program different music boxes that would behave the same way on different computers, just like Firefox acts the same on my Linux and on your Windows even though they’re different computers. Well, at least sort-of-ish.

Inbal: Long silence. Closed eyes. Ugh, she fell asleep just when I was really having fun. (sleepily) Only if the first card is really really long and the music box had many many teeth. Look, maybe her choice of words wasn’t so eloquent, but she did say a sentence very much to this effect almost in her sleep! I took that as proof she actually got what I was talking about so far.

Yaniv: (triumphant) Yes! Yes exactly! Well not exactly, but unbelievably close. Python is a very long punched card, and it runs on several different punching machines with many many teeth, but it makes all these machines behave the same way.

Inbal: My excitement must’ve woken her up a bit, because she frowned with closed eyes. All computers behave the same way? Isn’t that a bad thing?

Yaniv: For people like me, it can be a great thing! We can write software once, and run it on many different computers. And the second punch card is much easier to write compared to the first punched card we always reuse. Before things like Python, we had to think harder and punch cards differently for different computers. It wasted lots of time, so we couldn’t advance as rapidly. Today, we can do cool things in weeks. I know this is only very roughly true. But her breathing was getting very regular and I really wanted to finish the explanation. This is what I’m writing about. I’m explaining the structure of long long punched card which is Python, how it sets up the music box to be in just a particular way to eat the second punched card, which is a Python program, that’s easier to write and can run on any computer. That’s what’s my series of articles is all about, it lives happily ever after, the end.

Silence. I was wide awake and very happy. It’s so gratifying, to explain your work to your spouse. I think it’s even more so for those of us who have more arcane and less understood jobs. I stroked her dreadlocks (she puts all kinds of seashells and beads in them, it’s really nice) and finally fell asleep myself.

I can’t say her recollection the next day was perfect (You said something about music boxes… no? I remember it was a beautiful story!), but it was still quite a terrific experience for me. I wrote a very rough draft of this post in the morning when I woke up (the whole thing was a few weeks ago). As soon as I started writing it I realized this story is an obvious (and probably inferior) kin to Ryan Tomayko’s incredible “How I explained REST to my wife”. I dropped him a line to tell him I may write this piece, and he actually said some people don’t believe he had such a conversation with his wife.

Poor tossers! This is probably the most hard-core incident to date, but more often than not Inbal amazes me with how much she really knows about computers just from hanging around an aging geek like yours truly and hearing my occasional rants or explanations. Here is a plea to all you astrophysicists, microbiologists and mathematicians out there: explain your work, art and passion to your dancer, carpenter and education worker spouses! It’s fun (and you can blog about it, to boot)!

Python’s Innards: Code Objects

2010/07/03 § 5 Comments

This article, part of a series of articles about Python’s internals, will continue our preparation to engage the machinery of code evaluation by discussing Code Objects. To those of you who just now joined in and didn’t even read the introduction (but why?!), please note an important disclaimer: while the series as a whole is CPython 3.x centric and might not ‘apply cleanly’ to other Python implementations, matters of bytecode and evaluation (like this article discusses) are even more likely to deviate between implementations. So some of what I say in this post may apply to other implementations, some not – I’m not even checking at the moment; if and when we’ll discuss implementations like PyPy, Jython, IronPython, etc, I’ll highlight some of the differences. With this disclaimer in mind, we can get back to the plot: Code Objects. The compilation of Python source code emits Python bytecode, which is evaluated at runtime to produce whatever behaviour the programmer implemented. I guess you can think of bytecode as ‘machine code for the Python virtual machine’, and indeed if you look at some binary x86 machine code (like this one: 0x55 0x89 0xe5 0xb8 0x2a 0x0 0x0 0x0 0x5d) and some Python bytecode (like that one: 0x64 0x1 0x0 0x53) they look more or less like the same sort of gibberish. Along with the actual bytecode, Python's compiler emits additional fields, most of them must be coupled with the bytecode (otherwise it would be meaningless). The bytecode and these fields are lumped together in an object called a code object, our subject for this article.

You might initially confuse function objects with code objects, but shouldn't. Functions are higher level creatures that execute code by relying on a lower level primitive, the code object, but adding more functionality on top of that (in other words, every function has precisely one code object directly associated with it, this is the function's __code__ attribute, or f_code in Python 2.x). For example, among other things, a function keeps a reference to the global namespace (remember that?) in which it was originally defined, and knows the default values of arguments it receives. You can sometimes execute a code objects without a function (see eval and exec), but then you will have to provide it with a namespace or two to work in. Finally, just for accuracy's sake, please note that tp_call of a function object isn't exactly like exec or eval; the latter don't pass in arguments or provide free argument binding (more below on these). If this doesn't sit well with you yet, don't panic, it just means functions' code objects won't necessarily be executable using eval or exec. I hope we have that settled.

Let's see when code objects are created. Code objects are created whenever a block of Python code is compiled. We have mentioned blocks briefly before, the fine material defines them as "a piece of Python program text that is executed as a unit. The following are blocks: a module, a function body, and a class definition." (the fine material also lists other but less-interesting-to-us code blocks, like every command in the interactive interpreter, the string passed to Python's executable's -c switch, etc). As usual, I don't want to dig too deeply into compilation, but basically when a code block is encountered, it has to be successfully transformed into an AST (which requires mostly that its syntax will be correct), which is then passed to ./Python/compile.c: PyAST_Compile, the entry point into Python's compilation machinary. A kind comment in ./Python/compile.c explains the general execution flow of this function.

Next, let's discuss what is in a code object; I said it has stuff other than bytecode, but what? To whet our appetite about the various fields of a code object, we can look at the compiled Python sample from the first paragraph and disassemble it ourselves; it's easier if we know beforehand that both samples implement a function which simply returns the value 42. Unlike the x86 machine code sample, which is self-contained and should be ready to run (<cough>assuming I didn't botch it</cough>) the Python bytecode sample doesn't include the constant value 42 in it at all. You absolutely can't run this code meaningfully without its constants, and indeed 42 is referred to by one of the extra fields of the code object. We will best see the interaction between the actual bytecode and the accompanying fields as we do a manual disassembly.

From the interpreter (as usual, slight editing for readability):

# the opcode module has a mapping of opcode
#  byte values to their symbolic names
>>> import opcode
>>> def return42(): return 42
... 
# this is the function's code object
>>> return42.__code__
<code object return42 ... >
# this is the actual bytecode
>>> return42.__code__.co_code
b'd\x01\x00S'
# this is the field holding constants
>>> return42.__code__.co_consts
(None, 42)
# the first opcode is LOAD_CONST
>>> opcode.opname[return42.__code__.co_code[0]]
'LOAD_CONST'
# LOAD_CONST has one word as an operand
#  let's get its value
>>> return42.__code__.co_code[1] + \
... 256 * return42.__code__.co_code[2]
1
# and which constant can we find in offset 1?
>>> return42.__code__.co_consts[1]
42
# finally, the next opcode
>>> opcode.opname[return42.__code__.co_code[3]]
'RETURN_VALUE'
>>> 

I hope this was educational, albeit doing it all the time could get boring. Fortunately, we have dis to do this work for us (>>> from dis import dis, you already saw I aliased and augmented it as diss). In addition to dis, the function show_code from the same module is useful to look at code objects (I aliased and augmented a bit as ssc). So let's look at return42 with diss and ssc:

>>> diss(return42)
  1           0 LOAD_CONST               1 (42) 
              3 RETURN_VALUE         
>>> ssc(return42)
Name:              return42
Filename:          <stdin>
Argument count:    0
Kw-only arguments: 0
Number of locals:  0
Stack size:        1
Flags:             OPTIMIZED, NEWLOCALS, NOFREE
Constants:
   0: None
   1: 42
>>> 

We see diss and ssc generally agree with our disassembly, though ssc further parsed all sorts of other fields of the code object which we didn't handle so far (you can run dir on a code object to see them yourself). We have also seen that our value of 42 is indeed referred to by a field of the code object, rather than somehow be encoded in the bytecode.

Code objects are immutable and their fields don't hold any references (directly or indirectly) to mutable objects. This immutability is useful in simplifying many things, one of which is the handling of nested code blocks. An example of a nested code block is a class with two methods: the class is built using a code block, and this code block nests two inner code blocks, one for each method. This situation is recursively handled by creating the innermost code objects first and treating them as constants for the enclosing code object (much like an integer or a string literal would be treated). You may be wondering how mutable object literals (a = [1, 2, 3]) are represented in a code object, and the answer is that rather than referring to the mutable object with the code object, the 'recipe' to prepare it is kept (try >>> import dis; dis.dis(compile("a=(1,2,[3,4,{5:6}])", "string", "exec")) to make this immediately clear).

Now that we have seen the relation between the bytecode and a code object field (co_consts), let's take a look at the myriad of other fields in a code object. To be honest, I'm not sure this list would be particularly exciting. Many of these fields are just integer counters or tuples of strings representing how many or which variables of various sorts are used in a code object. But looking to the horizon where ceval.c and frame object evaluation is waiting for us, I can tell you that we need an immediate and crisp understanding of all these fields and their exact meaning, subtleties included. So I'll (tediously?) list and categorize them all, building on the rather terse description you can find in the standard type hierarchy. If this seems to boring right now, you best skim it now but keep it as a reference for later posts; trust me, it's useful.

Identity or origin (strings)
co_name
A name (a string) for this code object; for a function this would be the function's name, for a class this would be the class' name, etc. The compile builtin doesn't let you specify this, so all code objects generated with it carry the name <module>.
co_filename
The filename from which the code was compiled. Will be <stdin> for code entered in the interactive interpreter or whatever name is given as the second argument to compile for code objects created with compile.
Different types of names (string tuples)
co_varnames
A tuple containing the names of the local variables (including arguments). To parse this tuple properly you need to look at co_flags and the counter fields listed below, so you'll know which item in the tuple is what kind of variable. In the 'richest' case, co_varnames contains (in order): positional argument names (including optional ones), keyword only argument names (again, both required and optional), varargs argument name (i.e., *args), kwds argument name (i.e., **kwargs), and then any other local variable names. So you need to look at co_argcount, co_kwonlyargcount and co_flags to fully interpret this tuple.
co_cellvars
A tuple containing the names of local variables that are stored in cells (discussed in the previous article) because they are referenced by lexically nested functions.
co_freevars
A tuple containing the names of free variables. Generally, a free variable means a variable which is referenced by an expression but isn't defined in it. In our case, it means a variable that is referenced in this code object but was defined and will be dereferenced to a cell in another code object (also see co_cellvars above and, again, the previous article).
co_names
A tuple containing the names which aren't covered by any of the other fields (they are not local variables, they are not free variables, etc) used by the bytecode. This includes names deemed to be in the global or builtin namespace as well as attributes (i.e., if you do foo.bar in a function, bar will be listed in its code object's names).
Counters and indexes (integers)
co_argcount
The number of positional arguments the code object expects to receive, including those with default values. For example, def foo(a, b, c=3): pass would have a code object with this value set to three. The code object of classes accept one argument which we will explore when we discuss class creation.
co_kwonlyargcount
The number of keyword arguments the code object can receive.
co_nlocals
The number of local variables used in the code object (including arguments).
co_firstlineno
The line offset where the code object's source code began, relative to the module it was defined in, starting from one. In this (and some but not all other regards), each input line typed in the interactive interpreter is a module of its own.
co_stacksize
The maximum size required of the value stack when running this object. This size is statically computed by the compiler (./Python/compile.c: stackdepth when the code object is created, by looking at all possible flow paths searching for the one that requires the deepest value stack. To illustrate this, look at the diss and ssc outputs for a = 1 and a = [1,2,3]. The former has at most one value on the value stack at a time, the latter has three, because it needs to put all three integer literals on the stack before building the list.
Other stuff (various)
co_code
A string representing the sequence of bytecode instructions, contains a stream of opcodes and their operands (or rather, indexes which are used with other code object fields to represent their operands, as we saw above).
co_consts
A tuple containing the literals used by the bytecode. Remember everything in a code object must be immutable, running diss and ssc on the code snippets a=(1,2,3) versus [1,2,3] and yet again versus a=(1,2,3,[4,5,6]) recommended to dig this field.
co_lnotab
A string encoding the mapping from bytecode offsets to line numbers. If you happen to really care how this is encoded you can either look at ./Python/compile.c or ./Lib/dis.py: findlinestarts.
co_flags
An integer encoding a number of flags regarding the way this code object was created (which says something about how it should be evaluated). The list of possible flags is listed in ./Include/code.h, as a small example I can give CO_NESTED, which marks a code object which was compiled from a lexically nested function. Flags also have an important role in the implementation of the __future__ mechanism, which is still unused in Python 3.1 at the time of this writing, as no "future syntax" exists in Python 3.1. However, even when thinking in Python 3.x terms co_flags is still important as it facilitates the migration from the 2.x branch. In 2.x, __future__ is used when enabling Python 3.x like behaviour (i.e., from __future__ import print_function in Python 2.7 will disable the print statement and add a print function to the builtins module, just like in Python 3.x). If we come across flags from now on (in future posts), I'll try to mention their relevance in the particular scenario.
co_zombieframe
This field of the PyCodeObject struct is not exposed in the Python object; it (optionally) points to a stack frame object. This can aid performance by maintaining an association between a code object and a stack frame object, so as to avoid reallocation of frames by recycling the frame object used for a code object. There's a detailed comment in ./Objects/frameobject.c explaining zombie frames and their reanimation, we may mention this issue again when we discuss stack frames.

Phew! This is everything in a code object. In making this list I've compiled quite a few code blocks, looking how changes in the Python source changes the resulting code object. I recommend you do something similar, and I actually bothered to make it ultra-easy for you to look into how various code blocks affect these fields: in the Mercurial repository I have for this series I created a directory called code_objects, within it you can find a self-explaining little utility that can facilitate looking at a few sample code blocks I wrote alongside with their disassembly and show_code output. Not all fields are necessarily covered in the sample code blocks I provided, you should be able to add a few more samples (if anything intrigues you) yourself and see them disassembled/analyzed. Also, I'm sorry, I'm totally a *NIX bigot (and will erase all flame or even flame-ish comments about that) and this toy might not run on Windows. There's no good reason for that, I just wanted to use less for pagination, etc, and couldn't be bothered with achieving the same effect on Windows.

This pretty much sums up what I have to say about code objects at the moment. Time permitting, I sincerely hope we'll soon reach the next article, where we'll tackle the final frontier before ./Python/ceval.c: PyEval_EvalFrameEx itself: frame objects. ¡Olé!


I would like to thank Nick Coghlan for reviewing this article; any mistakes that slipped through are my own.

Where Am I?

You are currently viewing the archives for July, 2010 at NIL: .to write(1) ~ help:about.

Follow

Get every new post delivered to your Inbox.

Join 30 other followers