I just pushed an update to labour which supports forking multiple HTTP clients, alongside with some other small improvements. For the sake of this update I wrote a simple module called multicall which enables a process to distribute a callable among several subprocesses (POSIX only). More elegant solutions exist (‘import multiprocessing’ come to mind), but I think it’s hard to beat multicall’s simplicity, and it doesn’t use Python threads /at all/ (unlike multiprocessing), which lands some performance improvement.
I think the next area which requires work is probably reporting, and I’m not entirely sure what kind of reports would be useful. Labour isn’t a performance oriented benchmark, but a bit more like a cross between a benchmark and a test-suite for WSGI servers. I’m not sure if the output should be a graph (using behaviour set ‘foo’, servers X, Y and Z reach A, B and C hits per second) or a table (server X handles SIGSEGV well, server Y doesn’t, server X doesn’t handle a slow worker well, server Y does).
Also, and this is something that’s been troubling me for a while, I think I know what the results of most benchmarks will be. Let’s start with the fact that all servers that don’t fork out processes will not survive pretty much all the wedging/SIGSEGVing/memory leaking benchmarks. Second, I’m pretty sure most servers will handle somewhat different scenarios (IO/CPU/sleep) in exactly the same way, which is, not notice anything went wrong and have the worker pool reduced by one.
Assuming this is indeed the way the result matrix will look like, and assuming the response of the respective WSGI servers’ programmers’ will be as I expect (“fix the workers”) – then why am I doing this benchmark, anyway?
The main answer which comes to mind is that this is an ‘everyone fails’ exam which opens the door to a future where some will fail and some will succeed (or fail more gracefully). I’m pretty sure this is a future we should be heading to. I never liked the ‘fix the workers’ response, we’re hackers, not mathematicians – failing well is part of the job. That said, I’m not sure Labour will have enough weight in the WSGI/Python Web community to make this change.
If anyone wants to chip in, or even just send moral support, go ahead. I’ll post updates about Labour the next time I revisit this project.
Comments