TurnKey Linux Virtual Appliance Library

Python optimization principles and methodology


The basic methodology for optimization:

  1. Discover where you program is spending its time (hotspots vs coolspots)

    A good way to get an overview is to use the Python profiler. The Python profile will usually be included in Python's standard library:

4 simple software optimization tips

1) Always be experimenting!

Trying to squeeze out more performance out of your program? Don't be afraid to experiment!

In practice what that means is you setup small, simple throwaway experiments to establish how things work when you're not absolutely sure you fully understand something such (e.g., how many times a second a certain function can be invoked, how the profiler measures blocking IO or the time it takes a sub-program to complete).

Pythonic attribute magic (property, customized attribute access)

Many languages encourage programmers to use a getter/setter pattern.

Like this:

StdTrap: a magical Pythonic mechanism for intercepting console output

As a programmer I believe less is more. Good code is small, simple and elegant and many times favorable to larger, noisier code that does the same. It's not just about aesthetics either. Making code small and beautiful makes it easier to read, and easier to understand. Which is guaranteed to make it work better. Trust me on this.

Python iterators considered harmful

I just tracked down a nasty bug in my code to a gotcha with Python iterators.

Consider the following code...

Python property gotcha

If you like using a single getter/setter function for your properties, watch out if using None for the default. If you do that you won't be able to set your property to None!

Example code and workaround...

My code refactoring algorithm

You're looking at a block of Python code. It's not immediately obvious what it does. It's sort of a mess and you realize it needs to be refactored. But how? What mental algorithm do you use?

Neat trick: invoking a Python debugger at an arbitrary point in your program

Do you find yourself occasionally wishing you could freeze a misbehaving program at an arbitrary point in time and then examining what was going on interactively?

That's exactly what the debugger is for, but sometimes it's just too much of a bother to run your program inside it, you have to set breakpoints, etc.

Well there's a really simple alternative: call the debugger from an arbitrary point in your program, like this...

Python performance tests reaffirms "premature optimization is the root of all evil"

"Premature optimization is the root of all evil"

Today while programming I found myself thinking about the performance impact of various basic Python operations such as method invocation, the "in" operator, etc.

This is a mistake because you really shouldn't be thinking about this stuff while programming. The performance impact of doing things one way vs another way is usually so small that you couldn't measure it.

Practical guidelines for beautiful Python code

Every now and then Liraz and I find ourselves chatting about how much we love Python, but more so the lessons we have learned from coding, and how to apply them to create beautiful Python code. I've tried to refine some of the "lessons" into practical guidelines that I apply religiously to all new code I write, and the refactoring of old code I written.