Why don’t I contribute to Python (often)

Oddly enough, just a week or two after I wrote the post “Contributing to Python“, Jesse Noller (of multiprocessing fame) wrote a post called “Why aren’t you contributing (to Python)?“. As a somewhat wannabe contributor, I think it’s a terrific question which deserves wordy and open answers (hey, just like this one). I realize the question has been asked specifically in the context of python.org’s development, but I think some of the answer applies to the whole Python eco-system, alternative implementations and libraries/frameworks included. Before we begin, I think that to answer the question, some distinction should be made about two rather different kinds of contributions.

The first is driven by whatever it is that you’re contributing. Suppose you happen to hack about a better GIL arbitration algorithm, or a sprawling and insanely rich asynchronous networking framework. You want that stuff to get out. I’d go so far as to say the source code itself wants to be out. It’s your baby, more often than not you think it’s great and unless something bad happens, you’ll push it. These are the kinds of things where you’re likely to find yourself obsessing over the stats of the website describing your latest gizmo or taking it personally when some loser on python-dev (like, say, that van-what’s-his-name-guy) says your implementation of goto for Python is not quite the next best thing since sliced lists.

The other, rather different kind, is that you run into something that is rather obviously a bug and wish to open-a-ticket-that-has-a-good-chance-to-be-committed for it. First of all, this is usually a far smaller patch. I doubt many people import threading, discover The Beazley Effect, rework the GIL and open a ticket with a patch. The use-case here is more like “I have a reproducible SIGSEGV” or “I wish import zipfile would support ZIP64″. Two interesting observations about this case: first, people are far less committed to their contribution, and second, more importantly, the realities of life dictate that the J. Random Hacker who ran into this either found a workaround or patched their own Python, so they sidestepped the bug. This is important. In practically all interesting cases, the reporter has already sidestepped the bug before or shortly after posting (sidestepped is a loose term, maybe they even moved to Perl…). I doubt anyone’s schedule is loose enough to allow them to wait without sidestepping a painful thorn even for the next bugfix release. This is a hundred times more true for documentation touchups – if you realized it’s wrong, you probably don’t have to fix it to keep working, you just use whatever knowledge you now know is right.

A rather pathological, tertiary case is the “I am not-Python-core, I have some free time, I wanna contribute to Python and I went bug-hunting in the tracker” one. I think its a pathological case of the second kind of contribution, and one that I suspect happens rather rarely. I’ll lump these two together.

If you agree so far, that we have a commit-driven-contribution (“damn this is so awesome I want this to be part of Python/twisted/Django/etc”) and a contribution-driven-commit (“damn Python is so awesome, it’s a shame to leave this wart unfixed, I’ll help”). As I said, I think very different reasons prevent people from doing either. I’ll start talking about the latter kind, both because it seemed to be the focus of Jesse’s original post and because it’s easiest to answer.

First, almost everything Jesse listed as reasons is true. Don’t know how, don’t know where, etc, almost all true. The best remedy here is to get as many people as possible to have, ugh, “broken their contribution cherry”, so to speak. The easier it will be to submit minor fixes for the first time, the more people will do it. The first time is important, psychologically and setup-ly. I think after a patch of yours has been committed, the fear of the technical part process is gone and the feeling of “gee, I can actually put stuff in Python!” kicks in, and you’re far more likely to start submitting more small patches. So if you want many more people to help with mundane issues, documentations touchups, etc, the community at large should make every effort to make this first time sweet.

How do we make it sweet? I don’t know for sure, but here is a short flurry of ideas which I’d be happy to discuss (and aid implementing!):

  • Easy step-by-step instructions for opening a bug report, submitting a patch, for Ubuntu, OSX and Windows, all concentrated in one place, from setup to bug tracker attachment. The “contributing to Python” post I mentioned earlier is a (small) step in what I think is the right direction. We can flesh it out a lot, but make sure it keeps the step-by-step cookbook get-it-done approach, rather than what exists today, which is good, but isn’t aimed at getting-things-done. Compare signing up to Facebook with applying for a Tourist Visa in some foreign country.
  • Small-time-Python-contribution-talks material to be made available. This is both to be consumed online in web-talks, but mainly aims to reach out and encourage such talks in LUGs and highschools/colleges (hmm, I love this idea, I should do this sometime…).
  • A bit on a limb here, but maybe even doing what’s possible to optimize the review process in favour of first-time contributors. This is quite debatable, and (intentionally) vague, but I cautiously think it will pay off rather quickly.

These means (and probably others I didn’t think of) could probably alleviate the problem of a “contribution-driven-commit”, as I called it. Which leaves us with your fabulous implementation of goto, or “commit-driven-contribution”. I think two factors come into play here, both of them nearly irrelevant for the previous type of contribution. The first is the feeling that whatever it is you’ve done, it’s not good enough (this usually breaks my balls). “Me? Send this? To python-dev? Get outta here.”. And the second, I think, is indeed the feeling of an ‘uphill battle’ against grizzled python-dev grey beards and sharp tongued lurkers that are, I suspect, more likely than not to shred your idea to bits. Let’s face it, hacker communities at large are pretty harsh, and generally for understandable reasons. However, I think at times this tough skin and high barrier for contributing anything significant hurts us.

I used to have a co-worker, a strong hacker, who made two significant open-source packages for Python. I humbly think both are exceptionally elegant and at least one of them could have been a strong addition to stdlib. Every time I hear/read the code of some poor soul who recreated the efforts of this guy with these two packages, I cringe. I wouldn’t like to disclose his name before talking to him, but when I asked him why aren’t these packages part of stdlib, he said something like: “blah, unless you’re part of the python-dev cognoscenti you’ve no chance of putting anything anywhere”. I think he might not have pushed these packages hard enough, I should raid python-dev’s archives to know, but looking at the finesse of these packages on the one hand, and the number of questions on #python at freenode which I can answer by uttering these packages’ names, I think maybe we’re missing out on something. His perception, even if downright wrong (and I suspect it isn’t accurate, but not so wrong) is the bad thing that can happen to make you not contribute that big masterpiece you’ve made in your back yard, and that’s a damn shame. Most people will not survive the School of Hard Knocks, and that’s not necessarily always a good thing.

The issue of contributing big stuff is far more delicate and complex, and I’d be the first to admit I’m probably not rad enough to even discuss it. But it feels wrong to write a post under a heading like the one this one bears without at least mentioning this hacker-subculture-centric and sensitive issue, which affects many OSS projects, and Python as well. Micro-contributions are important, they’re the water and wind which slowly erode the landscape of a project into something cleaner, stabler and more elegant, but let’s not forget them big commits from which the mountains are born, too.

So Jesse, or any other curious soul, this is my answer to you. Should the gauntlet be picked up (by you, me or both of us) regarding the list of items I suggested earler about making micro-contributions more accessible? How about taking this to python-dev?


11 responses to “Why don’t I contribute to Python (often)”

  1. Are you aggregated via planet.python.org? I didn’t see these articles there. If you’re interested, email planet at python.org

  2. Vinay Sajip Avatar
    Vinay Sajip

    [your co-worker] said something like: “blah, unless you’re part of the python-dev cognoscenti you’ve no chance of putting anything anywhere”.

    That’s not really fair. I wasn’t a committer, nor was I part of the Python-dev community when the logging package I wrote got accepted into Python. Likewise, Steven Bethard, who was not previously a committer, got commit access initially for PEP updates and recently got his argparse package accepted into Python.

    Not all packages are a good fit for the stdlib, and having a package in the stdlib introduces a heavy backward-compatibility burden which can restrict evolution of the package, more than in the case of a downloadable package from PyPI. It could be argued that only certain “infrastructure”-type packages belong in the stdlib.

    I don’t know exactly which packages your co-worker wrote, but I would say that anyone is able to propose some changes on python-dev, and if well presented and favourably received, they could draft a PEP based on the initial feedback. If accepted, this could lead to their software being brought into Python.

    It’s not a closed shop, as far as I can see, and a lot more open to contributions than some other projects (such as Django, for example).

    1. I appreciate your comment (and ‘import logging’, by the way), you obviously have personal experience which shows the opposite of what I said towards the end of this post.

      I might’ve been wrong, and though I disclaimed my words there, if you or others see them as offending or unfair, I apologize.

      1. Vinay Sajip Avatar
        Vinay Sajip

        No offence taken – not by me, anyway. And have you noticed that your first name and mine are reverses of each other’s?!

        1. Uncanny… I think my family won’t take it well if I change my surname to Pijas. How do you find the surname Ninka? 🙂

  3. Another great post, thanks dude

  4. […] reading a number of blog posts (here, here, and here) and thinking about my personal experiences with recently becoming a committer, I decided that it […]

  5. Nick Coghlan Avatar
    Nick Coghlan

    Very interesting (wandered over here by way of getting distracted before reading your Python’s Innards articles).

    python-list and python-ideas are generally good places to float the idea of new standard library additions – python-dev will often be much harsher on suggestions that weren’t discussed in those venues first.

    It definitely takes passion to push for inclusion of a module in the standard library (see the recent discussions over PEP 3148 and the argparse PEP). However, for truly useful modules, this passion will often come from existing and potential users of the module – the module author will hear from supporters as well as detractors.

    However, the ethos of “vigorous criticism is the only known antidote to error” runs deep in the open source world and Python is no exception. Something an author can do though, is give *others* blessing to push for standard library inclusion. Taking the time to champion a PEP is a significant investment, so we can definitely understand someone having other priorities. (A proposal for inclusion without at least the author’s blessing isn’t likely to get very far though)

    1. Thank you for your comment, I largely agree with what you said. It’s interesting to note I how much following I see for this post in my blog’s statistics, I seem to have struck a nerve without really aiming to; the post was mainly a knee-jerk reaction to Jesse’s post on the mater which came just weeks after I resolved and blogged about seeing what I can do to start contributing more.

      I wrote the post as I was focused on the ‘small contributions’ front, my current muddy pool and where I suspect the main barriers of entry is psychological (‘oh, it will be lots of work’ or ‘oh, it will never get committed’) and superficially technical (‘ugh, svn… I know only git!’) rather than substantial. We should work as a community to lower these barriers, I think Jesse is doing great in that direction.

      That said, it seems more seasoned readers than me felt more strongly about ‘large contributions’, some here in the comments and some privately. I’m not sure my opinions should matter that much before I contributed something more significant then the little I already have, but the large contribution field is also one I humbly feel the community should iron out and improve upon. Contributing a PEP takes not just ‘time’, it’s not just the hours spent, I strongly suspect it’s also a lot of emotional stamina.

      I watch closely as PEP 3148 is being heavily pounded upon despite being discussed previously at length off python-dev and being on PyPI for about a year. I’m not saying this pounding is useless, far from it – vigorous criticism seems indeed to be the only known antidote to error, and python-dev’s collective technical might is indeed monumental. That said, and without wanting to sound too ‘touchy’, my heart goes out for people who worked hard on providing needed functionality at an excellent level (or even at ‘just’ a very good level, God knows stdlib is not always a shrine to good design and implementation), and have to go through the occasional tarring and feathering of approval.

  6. […] both in the original comments, Hacker News, Reddit and other thoughtful responses by Tim Golden, Yaniv Aknin and Brian Curtin made me sit back and think for a long time what we – as a “python […]

  7. […] was stunned to realize that this conversation is novel in the Ruby community, but has been going on for some time in the Python […]