Implementing Sliding Windows in Python

A sliding window is a type of function that accepts an iterable and sends it back in overlapping chunks. This is useful when you need to process an item in context. There's one in the old library examples for itertools (I'm talking Python v. 2.3.5)

In [1]:

from itertools import islice

def old_window(seq, n=2):
    "Returns a sliding window (of width n) over data from the iterable"
    "   s -> (s0,s1,...s[n-1]), (s1,s2,...,sn), ...                   "
    it = iter(seq)
    result = tuple(islice(it, n))
    if len(result) == n:
        yield result    
    for elem in it:
        result = result[1:] + (elem,)
        yield result

%timeit -n 1000 -r 5 list(old_window('abcdefgh', 3))
print(*list(old_window('abcdefgh', 3)), sep='\n')

1000 loops, best of 5: 3.4 µs per loop
('a', 'b', 'c')
('b', 'c', 'd')
('c', 'd', 'e')
('d', 'e', 'f')
('e', 'f', 'g')
('f', 'g', 'h')

The New Hotness

But that's old and if Node, Rust, Go and Mongo have taught me anything, it's anything more than 10 seconds old is garbage and not webscale. Sarcasm aside, I just enjoy tearing things down just to rebuild them and hopefully learn something in the process. Let's see what we can come up with...

In [2]:

from itertools import tee

def window(it, size=3):
    yield from zip(*[islice(it, s, None) for s, it in enumerate(tee(it, size))])

%timeit -n 1000 -r 5 list(window('abcedefgh', 4))
print(*list(window('abcedefgh', 4)), sep='\n')

1000 loops, best of 5: 6.7 µs per loop
('a', 'b', 'c', 'e')
('b', 'c', 'e', 'd')
('c', 'e', 'd', 'e')
('e', 'd', 'e', 'f')
('d', 'e', 'f', 'g')
('e', 'f', 'g', 'h')

Okay, so it's about half as fast (or twice as slow, I'm not sure which is the Glass Half Full).

Let's deconstruct it first:

• First the iterable is cloned a number of times.
• Those iterables are then passed to enumerate which is a generator that returns an item and it's position in an iter
• The positions and iterables are then passed into a list comphrension loop
• The loop populates the list with a series of islice objects -- each starting an iterable at the position given to it by enumerate
• The list is splatted into zip
• yield from the zip

But it's one line! One liners are cool. Right?

Benchmarks Mean Nothing

I'm of the opinion that benchmarks don't prove much by themselves -- a broader context is needed.

However... let's see how it stacks up against the old window in a variety of situations.

In [3]:

# calculate rolling average

from operator import add
from functools import reduce

f = lambda x: reduce(add, x)/len(x)

print(list(map(f, window(range(10), size=4))))
%timeit -n 1000 -r 5 list(map(f, old_window(range(10), n=4)))
%timeit -n 1000 -r 5 list(map(f, window(range(10), size=4)))

[1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5]
1000 loops, best of 5: 10.1 µs per loop
1000 loops, best of 5: 14.3 µs per loop

In [4]:

# generate pseudo text markov chains

text = "mary had a little lamb"

print({c[:-1]:c[-1] for c in window(text.split(), 3)})
%timeit -n 1000 -r 5 {c[:-1]:c[-1] for c in window(text.split(), 3)}
%timeit -n 1000 -r 5 {c[:-1]:c[-1] for c in old_window(text.split(), 3)}

{('had', 'a'): 'little', ('a', 'little'): 'lamb', ('mary', 'had'): 'a'}
1000 loops, best of 5: 14.8 µs per loop
1000 loops, best of 5: 5.14 µs per loop

Annother application for this in as a solution to Euler 8.

In [5]:

from collections import deque
from itertools import chain
from functools import partial
from operator import mul

nums = """73167176531330624919225119674426574742355349194934
96983520312774506326239578318016984801869478851843
85861560789112949495459501737958331952853208805511
12540698747158523863050715693290963295227443043557
66896648950445244523161731856403098711121722383113
62229893423380308135336276614282806444486645238749
30358907296290491560440772390713810515859307960866
70172427121883998797908792274921901699720888093776
65727333001053367881220235421809751254540594752243
52584907711670556013604839586446706324415722155397
53697817977846174064955149290862569321978468622482
83972241375657056057490261407972968652414535100474
82166370484403199890008895243450658541227588666881
16427171479924442928230863465674813919123162824586
17866458359124566529476545682848912883142607690042
24219022671055626321111109370544217506941658960408
07198403850962455444362981230987879927244284909188
84580156166097919133875499200524063689912560717606
05886116467109405077541002256983155200055935729725
71636269561882670428252483600823257530420752963450"""

nums = [int(n) for n in chain.from_iterable(nums.split("\n"))]

def euler_8_1(numbers):
    limit = 13
    run = []
    largest = 0
    
    # originally this was a while-loop that popped off items from numbers
    # I changed for the sake of not being so fucking stupid
    # inserting at index[0] on a list is still dumb though.
    
    # purpose for this change was to avoid copying the 
    # numbers list every time and thus artificially
    # bloating the run time for the solution
    
    # This was my first encounter with windowing
    # through I didn't know it yet.
    
    for n in numbers:
        run.insert(0, n)
        if len(run) > limit:
            run.pop()
        
        x = reduce(lambda x, y: x*y, run)
        if x > largest:
            largest = x
    return largest

def euler_8_2(numbers):
    run = deque(maxlen=13)
    largest = 0
    for n in nums:
        # look another window
        # still didn't really know  what
        # windowing was at this time
        run.append(n)
        x = reduce(mul, run)
        
        if x > largest:
            largest = x
    
    return largest

print("Solution to Euler #8 (as of 8/21/14): ", max(map(partial(reduce, mul), window(nums, size=13))))
%timeit -n 1000 -r 5 euler_8_1(nums)
%timeit -n 1000 -r 5 euler_8_2(nums)
%timeit -n 1000 -r 5 -p 5 max(map(partial(reduce, mul), window(nums, size=13)))    # upped the precision of these two
%timeit -n 1000 -r 5 -p 5 max(map(partial(reduce, mul), old_window(nums, n=13)))   # because they're pretty close

Solution to Euler #8 (as of 8/21/14):  23514624000
1000 loops, best of 5: 2.53 ms per loop
1000 loops, best of 5: 1.05 ms per loop
1000 loops, best of 5: 1.2112 ms per loop
1000 loops, best of 5: 1.2605 ms per loop

Not too shabby, not quite as fast as the second iteration of my solution but way faster than my first (admittedly, stupid) solution. The old windowing function falls just short of the new one.

Windowing Infinite Generators

The real differences start to be seen here: working with infinite generators. The old windowing function handles this quite well, assuming you don't attempt to consume "entire" generator at once. The new windowing function needs you to massage the infinite generator into a subset. Either way, islice comes into play, it's just a matter of where.

In [6]:

def fib():
    yield from range(2)
    a,b = range(2)
    while True:
        yield a+b
        a,b = b, a+b

size = 5
start = 0
end = 9

def display(it):
    print(*list(it), sep='\n')

display(window(islice(fib(), start, end), size))

(0, 1, 1, 2, 3)
(1, 1, 2, 3, 5)
(1, 2, 3, 5, 8)
(2, 3, 5, 8, 13)
(3, 5, 8, 13, 21)

That's pretty nifty. However, window still creates a number of iterators equal to it's window size and partially consumes them almost immediately. And we can't ask for, say the first five windows of five numbers without jumping throw some nasty hoops as opposed to the old windowing function:

In [7]:

size = 5
frames = 5

# what is this hy? -- https://github.com/hylang/hy
display(map(next,(window(islice(fib(), i, i+size), size) for i in range(frames)))) 
#islice(old_window(fib(), size), frames)

(0, 1, 1, 2, 3)
(1, 1, 2, 3, 5)
(1, 2, 3, 5, 8)
(2, 3, 5, 8, 13)
(3, 5, 8, 13, 21)

Let's break that down, because even though I wrote it just now, I have a hard time following it (and it only took about ten tries to get it right, too):

• islice(fib(), i, i+size) -- slice the number generator at an arbitrary length
• window(..., size) -- window that sliced generator to the appropriate size
• ... for i in range(frames) generate the arbitrary numbers for the original islice
• (...) wrap the whole thing in a generator expression

As opposed to the old busted, which breaks down to:

• old_window(fib(), size) -- begin windowing the generator
• islice(..., frames) -- only take the frames that we want.

Starting the generator in an arbitrary spot isn't as intuitive as it is with the old windower - which is just fast forward the generator to a specific spot with islice. With the new hotness we have to make sure we're offsetting appropriately (only took three tries and some serious mind searching to determine if I was simply retarded or not).

In [8]:

start = 3
size = 5
frames = 5

#list(map(next,(window(islice(fib(), i+start, i+size+start), size) for i in range(frames))))
display(islice(old_window(islice(fib(), start, None), n=size), frames))

(2, 3, 5, 8, 13)
(3, 5, 8, 13, 21)
(5, 8, 13, 21, 34)
(8, 13, 21, 34, 55)
(13, 21, 34, 55, 89)

Everything Old Is New

Okay, so using a single container is faster than yielding out a bunch of iterators. But why not use a fixed length iterable instead of playing surgeon on a tuple? collections.deque is one of my favorite things in the standard library and it's fixed length and I (accidentally) used it for windowing before. Why not general purpose it?

In [9]:

def dedow(it, size, start=0, stop=None, step=None):
    '''dedow -- `de`que win`dow` -- alright, that's a stupid name.
    Yields a tuple for each frame of the window.
    '''
    it = iter(it)
    if any([start, stop, step]):
        it = islice(it, start, stop, step)
    window = deque(islice(it, size), maxlen=size)
    yield tuple(window)
    for i in it:
        window.append(i)
        yield tuple(window)

display(dedow('abcdefgh', size=3))
%timeit -n 1000 -r 5 list(old_window('abcdefgh', n=3))
%timeit -n 1000 -r 5 list(window('abcdefgh', size=3))
%timeit -n 1000 -r 5 list(dedow('abcdefgh', size=3))

('a', 'b', 'c')
('b', 'c', 'd')
('c', 'd', 'e')
('d', 'e', 'f')
('e', 'f', 'g')
('f', 'g', 'h')
1000 loops, best of 5: 3.52 µs per loop
1000 loops, best of 5: 6.34 µs per loop
1000 loops, best of 5: 6.02 µs per loop

It's actually slower than window here, but I find the code much more readable than either window or old window. It handles just like the old windowing function, plus it can handle slicing an iterable for you. Though, I do kinda feel that by passing arguments to islice violates the single responsibility principle -- not enough that I'll take it out, but I do see how that could be valid criticism.

In [10]:

# Window 3 over every other fibonacci number for the first 20 numbers
display(dedow(fib(), size=3, step=2, stop=20))

(0, 1, 3)
(1, 3, 8)
(3, 8, 21)
(8, 21, 55)
(21, 55, 144)
(55, 144, 377)
(144, 377, 987)
(377, 987, 2584)

Benchmarks Still Means Nothing

Let's benchmark these against some tests:

1. Window five values from the first 100 numbers of fib()
2. 100 windows of five values from fib()
3. Solve Euler 8
4. Generate a simple n-Order Markov Chain (n = window size - 1)

In [11]:

print("1. Window of 5 for first 100 values of fib()")
consume = partial(deque, maxlen=0)
fib100 = tee(islice(fib(), 100), 3)

# we could use dedow's stop kw but, I think it's less fair to construct the slice each interval
%timeit -n 1000 -r 5 consume(window(fib100[0], size=5))
%timeit -n 1000 -r 5 consume(dedow(fib100[1], size=5)) 
%timeit -n 1000 -r 5 consume(old_window(fib100[2], n=5))

print("\n2. First 100 windows of 5 of fib()")
%timeit -n 1000 -r 5 consume((window(islice(fib(), i, i+5), 5) for i in range(100)))
%timeit -n 1000 -r 5 consume(islice(dedow(fib(), size=5), 100))
%timeit -n 1000 -r 5 consume(islice(old_window(fib(), n=5), 100))

print("\n3. Solve Euler #8")
def solve(prob):
    f = partial(reduce, mul)
    return max(map(f, prob))

%timeit -n 1000 -r 5 solve(window(nums, size=13))
%timeit -n 1000 -r 5 solve(dedow(nums, size=13))
%timeit -n 1000 -r 5 solve(old_window(nums, n=13))

print("\n4. Generate a simple n-Order Markov Chain (n = window size - 1)")
text = 'lorem ipsum dolor sit amet consectetur adipiscing elit pellentesque gravida turpis egestas nisl consectetur et'
%timeit -n 1000 -r 5 {c[:-1]:c[-1] for c in window(text.split(), 3)}
%timeit -n 1000 -r 5 {c[:-1]:c[-1] for c in dedow(text.split(), 3)}
%timeit -n 1000 -r 5 {c[:-1]:c[-1] for c in old_window(text.split(), 3)}

1. Window of 5 for first 100 values of fib()
1000 loops, best of 5: 6.07 µs per loop
1000 loops, best of 5: 4.33 µs per loop
1000 loops, best of 5: 3.22 µs per loop

2. First 100 windows of 5 of fib()
1000 loops, best of 5: 91.9 µs per loop
1000 loops, best of 5: 78.8 µs per loop
1000 loops, best of 5: 54.2 µs per loop

3. Solve Euler #8
1000 loops, best of 5: 1.22 ms per loop
1000 loops, best of 5: 1.45 ms per loop
1000 loops, best of 5: 1.27 ms per loop

4. Generate a simple n-Order Markov Chain (n = window size - 1)
1000 loops, best of 5: 13.2 µs per loop
1000 loops, best of 5: 15.5 µs per loop
1000 loops, best of 5: 12.3 µs per loop

Part of the Zen of Python is "There should be one-- and preferably only one --obvious way to do it.". However, windowing isn't something that I find obvious or easy to do as evidenced by three solutions presented here. Aside from the two I presented, I can think of at least one more solution: a class based windower that implements the iterator protocol.

In [12]:

from collections import namedtuple
from string import ascii_lowercase as letters

# islice doesn't play well with actual slice objects
_slice = namedtuple('_slice', ['start', 'stop', 'step'])
        
class Window:
    '''Implements the iter protocol as to represent a window over a sequence'''
    def __init__(self, seq, size, start=None, step=None, stop=None):
        self.slice = _slice(start or 0, stop or None, step or 1) 
        self.size = size
        self._seq = iter(seq)
        if any(self.slice):
            self._seq = islice(self._seq, *self.slice)
        self.cheat = deque(islice(self._seq, size), maxlen=size)
        self._stop = False
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self._stop:
            raise StopIteration
            
        cheat = tuple(self.cheat)
        
        try:
            self.cheat.append(next(self._seq))
        except StopIteration:
            self._stop = True
        
        return cheat

display(dedow(letters, size=4, step=2))
%timeit -n 1000 -r 5 consume(Window(letters, size=4, step=2))
%timeit -n 1000 -r 5 consume(dedow(letters, size=4, step=2))

('a', 'c', 'e', 'g')
('c', 'e', 'g', 'i')
('e', 'g', 'i', 'k')
('g', 'i', 'k', 'm')
('i', 'k', 'm', 'o')
('k', 'm', 'o', 'q')
('m', 'o', 'q', 's')
('o', 'q', 's', 'u')
('q', 's', 'u', 'w')
('s', 'u', 'w', 'y')
1000 loops, best of 5: 21.5 µs per loop
1000 loops, best of 5: 9.73 µs per loop

Of course, that's a basic example and I (feel like I) cheated by using deque to store the window itself -- I mean, it's basically dedow. But it communicates the basic idea behind using an object as a windower.

Did I Learn Anything?

Implementing the iter protocol is difficult sometimes. I tried a bunch of methods until I decided to cheat and use deque. The guys who contribute to Python know more about what's fastest than I do (who saw that coming?).

Also, windowing is fun. Why haven't I considered this before for projects? These implementations are actually viable for a Markov Chain project I've been working on occasionally (which up until now had a very awkward method that broke text into tuplekey-value pairs).

blogger.ipynb

IPython is a powerful, interactive shell for Python, considered by many to be a replacement for the standard Python shell. It's most likely in your repositories under ipython (or ipython3 or ipython2 depending on which Python version is native to your distro). There's tons of information available and it's highly customizable -- even coming with support languages other than Python.

In [1]:

%%bash
echo "Hello from $BASH"

Hello from /bin/bash

But what I'm really going to evangelize today is the Notebook portion of it. You'll probably have to install the notebook separately -- typically listed under ipython-notebook in repositories. In fact, I'm blogging from it right now.

What is a Notebook

A Notebook is a portable IPython session that runs in your browser. When you run ipython notebook it starts a local server and it'll probably attempt to launch a browser, too. If not, open one up and navigate to 127.0.0.1:8888 and you'll see the IPython Notebook dashboard, which'll list all the notebooks in the directory it was launched in and allow you create new ones, run present ones and even delete them.

In many ways, a Notebook is just like using IPython on the commandline. Code in run cells is available to cells below it. You can even use IPython's magics in them.

Sharing Notebooks

IPython stores Notebooks in ipynb files -- JSON by any other name -- in the directory that the Notebook server was launched in. From there, you can store them on Github, send them to friends, host a Notebook Viewer server yourself,

Also, you can convert the Notebooks to other formats: markdown, HTML, Latex, PDF, reStructured Text, javascript slides, and actual Python files. To do so, you simply run: ipython nbconvert --to {format} [FILE]. However, you may have to install the pacdoc package to do so (I did). I'll be covering converting them to HTML and using them with Google's Blogger platform

Blogging with IPython Notebooks

The first step is to, of course, actually produce an IPython Notebook. You already know how, so just do it.

After you have your notebook, pandoc is installed, and you're sitting with a terminal open, you'll want to run ipython nbconvert --to html mynb.ipynb. This'll generate a full HTML page complete with CSS and the necessary Javascript to display mathematical equations. You'll want to extract this CSS and Javascript and place it into your Blogger template.

Modifying Blogger Templates

To do this, I actually didn't use vim, I used the text editor that came with Mint. Simply paste this CSS into the template by going in through Template > Customize > Advanced > Add CSS. And then the Javascript by going in through Template > Edit HTML and pasting it in somewhere in the <head> tag.

Translating Notebooks to HTML

To get just the body of the Notebook in HTML form, use ipython nbconvert --to html --template basic mynb.ipynb. Then simply copy the generated HTML into the HTML portion of a new Blogger post. If you use Linux, there's actually a fairly simple way of doing this on the command line: xclip. xclip -selection c mynb.html. Alternatively, if you don't care if the actual HTML file is generated you can just run ipython nbconvert --to html --template basic --stdout mypy.ipynb | xclip -selection c. Then simply CRTL-V in the HTML editor for blogger and click Publish.

Decorator Day

An Aside: I originally wrote this as just a massive brain dump but in retrospect, it wasn't helpful at all. In the process of preparing a presentation for PyATL, I decided to rewrite it to be much more informative and less example heavy. The original post was essentially just a series of examples of how you'd use decorators or what you'd use them for rather than why you'd use decorators and addressing patterns. I'm hoping this rewrite clarifies that instead of being a "Look at all this code you probably don't understand!"

Decorators are well championed but most posts I've seen about them don't address many of the issues here. I wouldn't say they're something beginner or novice friendly since it delves into concepts like functional programming and "higher order functions" which is just scary talk for "functions that accept functions as arguments" and even currying, which basically boils down to "reducing the number of inputs a function takes." I'm grossly over simplifying here for the sake of explaination, but I stand by these simplifications.

Another note, apparently there was a lot of hemming and hawing in the Python mailing list when this construct (for lack of a better word) was introduced because of confusion with the decorator design pattern. Given that design patterns aren't my strongest suit (at least in terms of knowing exactly what each does and how to best implement it), I think there's quite of bit of one-to-one between the two on a basic level. Each take in an object and modifies it in a way that the original shouldn't be worried about. The decorator pattern seems to be more concerned with adding data or processing information and is compositional whereas Python's decorators seem to be more about removing logic that, yes, is associated with the object in question but doesn't need to be mixed directly with the actual logic of the object. That makes it sound very confusing, but hopefully that will become clear through examples and explaination.

Decorator Day

This is meant to be a "hit the ground running" sort of thing decorators. Just enough information to make you feel confident but not enough that you know all there is to them. Though, honestly, once they click, they're not so complicated.

I'm going to make the dangerous assumption that you have a decent grasp on functions and how Python handles them. I'm going to briefly cover a bit about that, but just what -- I think -- is requisite knowledge for understanding decorators. If you're confident with the concept of first class functions and closures, feel free to skip the next two sections.

And just remember, a little knowledge is a dangerous thing; I'm giving you enough rope to hang yourself here but not enough to build a bridge. That comes with practice and exploration as well as digging deeper into subject matter. Don't mistake me for a guru expert code ninja wizard (though, I do hope to attain that title some day) as I'm in the mire with everyone else.

First Class Functions

The first, and probably most important, piece of background knowledge is that everything in Python is an object. Everything. Functions, Objects, Strings, and even Classes. This is what makes decorators possible in Python. We can assign functions to new names, stuff them in data structures, pass them as arguments to other functions and even return them from functions. That's really all there is to first class functions, they are treated just the same as anything else in the language. And since methods are simply functions attached to a class or object, the same applied to them as well. str.join is just as first class as map.

In [1]:

my_func = dir

funcs = [filter, list, int]

map(str, [1, 2, 3])

def returns_func():
    return dict

Closures

The concept of closures scares a lot of people. But they're really simple: closures are just functions defined inside of another function. That's it.

But there's magic involved, too. Closures close over the available outer scope and stuff it into their __closure__ attribute -- which is just simply a tuple of cell objects that contain other things. This is essentially a snapshot of the outer function when it's created. Usually, the inner function is returned by the outer function.

In [2]:

def adds_four():
    x = 4
    def inner(y=0):
        return x+y
    return inner

o = adds_four()
print(o())
print(o.__closure__)

4
(<cell at 0x7f26c154bd68: int object at 0x9f8860>,)

The same logic applies to values passed into a function as well. When an argument is passed into a function (say outer actually looked like outer(x=4)), those values become part of the scope of the function as well and are available to closures for use if needed. Maybe I'll go deeper into closures in the future, maybe not.

An aside

If you're wondering how Python knows what cell object to use well, my understanding is that it doesn't and relies on the generated byte code to do the right thing. But I could very well be completely wrong on this matter.

So, Decorators

Now we finally get to decorators. Decorators are functions that take in a function, do something and return something -- typically a callable, but it could be anything. Usually, decorators are used to either remove extraneous logic from a function or to attach additional functionality to it.

There's two ways to use decorators: the old way, pre-2.5, and the special @decorator syntax. They do the same thing, like using list(1, 2, 3) or [1, 2, 3], one's just syntatic sugar.

In [3]:

def decorator(f):
    return f

def my_func():
    pass

my_func = decorator(my_func) # this is the old way

@decorator
def my_func():
    pass

The @decorator syntax is the one you'll see in most cases. Basically, what you're telling Python is when it's done creating my_func pass it to decorator and then assign the result back to my_func. It sounds complicated to some, but it's really not.

@decorator is just a way of shorthanding your intention to the interperter. And more importantly, it gives a heads up to people reading your code that something more is happening behind the scenes. If you manually wrap functions (which is needed sometimes), this can get lost at the bottom of your function definition unless you make glaringly obvious what's happening. This applies double to longer fuctions and objects.

If you've got that down, you'll probably get the common patterns that follow.

Patterns

There's plenty of patterns you'll encounter when working with decorators. I'm hoping to identify some of the most common ones.

Pass Through Decorators

Basically, this is a decorator that takes in an object, does something and returns the original object. It might register it somewhere, gather some information about or add an attribute to it.

In [4]:

def pass_through(f):
    f.inspected = True
    return f

@pass_through
def thing():
    return "Thing()"

assert hasattr(thing, 'inspected')
assert thing.__name__ == 'thing'

It's common enough but usually they're not doing anything too exciting or surprising. But it's definitely worth distinguishing it from decorators that use closures.

The rest of the patterns I'm going to cover are all function closure oriented, except for two which will be obvious.

Flexible Decorators

You're going to want your decorators to work with as many functions as possible. Not just the one you initially noticed and thought, "I can factor this other code out into a decorator." But what does a flexible decorator look like? It's best to compare it to a brittle decorator to understand fully.

In [5]:

def brittle(f):
    def wrapper(arg1, arg2, kwarg1=None, kwarg2=None):
        return f(arg1, arg2, kwarg1, kwarg2)
    return wrapper

def flexible(f):
    def wrapper(*args, **kwargs):
        return f(*args, **kwargs)
    return wrapper

What's so important about this that we end up using wrapper when we call a decorated function. If we used the first example, the brittle one, we'd have to write every function we decorate with it to accept four arguments. Where as the second example, the flexible one, can accept any number of positional or keyword arguments.

To give a concrete example...

In [6]:

@flexible
def dinner(food, beer): # <- becomes wrapper
    print('Eating', food, 'and drinking', beer)

dinner('bratzel', 'azazel') # <- actually wrapper

Eating bratzel and drinking azazel

I'm a little homesick for Memphis. If you're ever there, I'd recommend the Flying Saucer downtown. Anyways...

Preserving metadata

When you lock your original function up in a closure, you lose the original metadata: __name__, __doc__, etc. There's a few ways to retain it. Instead of hopping to the correct way, I'll show you why you'll want to do that instead. Just like ensuring defaults are set in a dictionary, there's an increasing level of "correctness" or "Pythonness" from what you should start with and what you should end with.

The first thing you might try is manually transfering these attributes inside the decorator.

In [7]:

def loggit(f):
    def logger(*args, **kwargs):
        print("Calling {}".format(f.__name__))
        return f(*args, **kwargs)
    logger.__name__ = f.__name__
    logger.__doc__ = f.__doc__
    logger.__dict__.update(f.__dict__)
    return logger

Other than the fact that we're missing three crucial attributes, this is a terrible way to migrate attributes. You'll have to do this in every decorator you write -- meaning it'll be no fun to maintain when a new attribute is introduced. This is very similiar to the basic idiom of checking if a key is in a dictionary before we do something with it:

if something not in my_dict:
    my_dict[something] = [value]
my_dict[something].append(value)

Well, decorators are good at removing extraneous code from a function, let's try writing a decorator decorator -- as confusing as that sounds, it's pretty common, so we're catching two pythons with one net.

In [8]:

def decorator_decorator(deco):
    def wrapper(f):
        g = deco(f)
        g.__name__ = f.__name__
        g.__doc__ = f.__doc__
        g.__dict__.update(f.__dict__)
        return g
    wrapper.__name__ = deco.__name__
    wrapper.__doc__ = deco.__doc__
    wrapper.__dict__.update(deco.__dict__)
    return wrapper

@decorator_decorator
def loggit(f):
    def logger(*args, **kwargs):
        print("Calling {}".format(f.__name__))
        return f(*args, **kwargs)
    return logger

@loggit
def double(x):
    """Doubles a number"""
    return 2 * x

assert double.__name__ == 'double'
assert double.__doc__ == 'Doubles a number'

That's almost better. Except it's really ugly. Like really ugly. And actually, I find it more confusing than helpful. When I initially wrote this code it was as an example of what not to do and it's here for that same reason. It requires you to think on multiple levels at the same time, which is great if you're just trying to anger people looking over it. This decorator_decorator is pretty benign if you're comfortable with what each piece means and works. But if you're doing this in your code, unless you're having to work with 2.4 or below, you're doing it the wrong way.

This is most akin to using dict.setdefault to ensure a value is present in a dictionary, and just like that method, I find it confusing every time I look at it (though, there are many times when it's useful):

grouping = my_dict.setdefault(something, [])
grouping.append(value)

However, Python is batteries included and comes with something that handle this for us. Enter functools.wraps and functools.update_wrapper. @wraps is just the decorator form of update_wrapper -- using a decorator to create decorators seems counter intuitive, but it's really not and @wraps is a testament to it's power and another example further down delves a little deeper. But update_wrapper is pretty simple, it merely:

• Copies needed attributes from the original function to the new
  • This includes the name, docstring, module, qualified name and module attributes
• Updates the wrapper's dict with the original function's dict
• As of 3.2, it also supplies a __wrapped__ attribute that contains the original object
  • It doesn't appear that this was backported to Python 2 at all. I'd seriously recommend monkey patching update_wrapper to handle this for reasons that'll become obvious later.

update_wrapper is also smart enough to not blow up if one of these attributes isn't present on the underlying object (such as __name__ on an instance of an object).

Here's how you should be perserving metadata through decorators:

In [9]:

from functools import wraps

def loggit(f):
    @wraps(f)
    def logger(*args, **kwargs):
        print("Calling {}".format(f.__name__))
        return f(*args, **kwargs)
    return logger

@loggit
def double(x):
    """Doubles a number"""
    return 2 * x

assert double.__name__ == 'double'
assert double.__doc__ == 'Doubles a number'
assert not double.__annotations__

Using @wraps and update_wrapper is like when you stumble across defaultdict in the collections module. The complexity and hand holding you were previously doing just melts away into a few easy to grasp lines of code:

my_dict = defaultdict(list)
my_dict[something].append(value)

Minimum Decorator Best Practices

The two best things you can do for yourself is writing flexible decorators (using *args and **kwargs) and preserving metadata through wraps and update_wrapper. 80% of the headaches you'll encounter will be fixed with these. The other 20% are...tricky. I'll touch on those a little later, however.

Decorators that accept optional kwargs

Occasionally, you need to use keyword arguments in your decorators, either to pass context, run setup or just set flags. There's a few ways to do it...

In [10]:

def print_before(phrase="Setting up!"):
    def actual_decorator(f):
        @wraps(f)
        def wrapper(*args, **kwargs):
            print(phrase)
            return f(*args, **kwargs)
        return wrapper
    return actual_decorator

@print_before(phrase="such setup")
def doubles(x):
    return 2 *x

@print_before() # parens required
def random():
    return 4

In case you're wondering, the parens are required because we need to call the outermost function, which generates the decorator which in turn generators the wrapper function we're actually using. To be 100% honest, outside of a handful of cases, I find this particular implementation to be lackluster because usually the outermost function just serves to pass information one level deeper.

However, one of the best arguments for using this particular pattern is passing positional arguments to a decorator. This has more to do with how Python handles unpacking into functions than anything else. But using an outer function like this is certainly the easiest way I've thought of to handle positional arguments to a decorator. You could fiddle with pushing the fuction to the front of *args, but I feel it would quickly become messy.

Back to optional kwargs, instead of having an outermost function that we must explictly call, why not use functools.partial and setup a little check?

In [11]:

from functools import partial

def print_before(f=None, *, phrase="Settin up!"):
    if f is None:
        return partial(print_before, phrase=phrase)
    
    @wraps(f)
    def wrapper(*args, **kwargs):
        print(phrase)
        return f(*args, **kwargs)
    
    return wrapper

@print_before # parens not needed
def double(x):
    return 2*x

@print_before(phrase="such dice roll!")
def random():
    return 4

That's actually a lot better. The most interesting thing here is how using partial allows us to for go the parens when we don't overide keyword arguments. Since we're accepting only one positional argument (via Python 3's keyword only syntax, there's not really a good way to imitate this is Python 2 other than using positional arguments sparingly when calling a function), when Python puts the function into the decorator, it does it positionally. We don't need to call the outermost function explicitly because Python will call it for us. And if we need to override a keyword argument, we can call the function explicitly to do so.

However, we've just moved our primary concern inside the decorator and it's creating noise. Moreover, if we decide that this is the best thing in the world, we'll be copying that code into every decorator we write. And six months from now, when partial is considered as terrible as GOTO, we'll have to replace all these instances (though, you can never take my partials from me!). I'm not sure if that's better or worse than before.

We're already writing a decorator, and we're getting pretty good at them, so why not a decorator that'll handle this for us? This is probably the most confusing example, so hold on:

In [12]:

def optional_kwargs(deco):
    @wraps(deco)
    def wrapper(f=None, **kwargs): # <- actual function our decorator becomes
        if f is None:
            return partial(wrapper, **kwargs)
        return deco(f, **kwargs)
    return wrapper

@optional_kwargs
def print_before(f, *, phrase="Setting up!"): # <- becomes the closure inside in optional_kwargs
    @wraps(f)
    def wrapper(*args, **kwargs):
        print(phrase)
        return f(*args, **kwargs)
    return wrapper

@print_before(phrase="much mind bend!") # <- optional_kwarg's closure
def random(): # <- becomes print_before's closure
    return 4

This looks really complex and intimidating, but it's really not when you get to understand decorators. All we've done is factor out the code print_before shouldn't be concerned with, namely if it actually recieved a function or not. What happens is that we replace print_before with the closure inside optional_kwargs. This closure handles the logic of ensuring that a function was passed or not.

The true power of this becomes apparent when you replace the print statement with something that will preprocess inputs for you: escaping characters or changing integers and floats to Decimal if you're working with financial data. Here, it's simply to illustrate how you would implement the pattern.

Like the horrible decorator_decorator example above, this sort of code forces your brain to operate on multiple levels at the same time. Once you're comfortable with it, this makes quite a bit of sense as to why you'd do it. However, before that, it merely causes a headache.

Using classes as decorators

Sometimes a function closure doesn't provide the umph needed to do something, or packing in the umph creates a twisted web of logic that causes your eyes to roll into the back of your head when you revisit it. Sometimes you just need a good 'ol object to handle things for you. There's two different ways to use objects as decorators, one is to use one instance per wrap and the other is to use one instance for every wrap.

In [13]:

from functools import update_wrapper

class wrapperobj:
    def __init__(self, f):
        self.f = f
        update_wrapper(self, f)
    
    def __call__(self, *args, **kwargs):
        return self.f(*args, **kwargs)

@wrapperobj
def random():
    return 4

When using a single instance per wrap, you need to first store the wrapped obj on the instance (self.f) and then when you need to call it, you simply pass it arguments. Fairly straight forward, but it's pretty contrived example as well. This sort of pattern would be used when your logic becomes much more complex than what a simple function should reasonably handle, things like transcieving data to an event loop or socket for example or creating individual caches for each object.

Let's look at using a single instance to wrap multiple objects.

In [14]:

class onewrapper:
    def __call__(self, f):
        @wraps(f)
        def wrapper(*args, **kwargs):
            return f(*args, **kwargs)
        return wrapper
    
inst = onewrapper()

@inst
def random():
    return 4

This is essentially the same thing as using a function closure as a decorator, except it comes with the extra umph that comes with using an object as well. This sort of pattern would be used for linking several objects together in a shared state, a rudimentary messaging queue should be easy to write with this sort of pattern. Your __init__ method in this case would set the initial state for the object rather than accepting the function there.

And of course, you don't have to use __init__ or __call__ to register and run functions, you can create other methods that would handle these as well. register and run_wrapped for example could easily do the same thing.

If you're wrapping multiple objects with the same instance, it may be helpful to also use some sort of data structure to organize them if you're doing more than simply wrapping them. In most cases the weakref module will either provide the answer you need or set you on the right path; though, there are objects that can't be weak referenced -- including most of the built in or "primitive" types in Python (str, int, float, list, dict and tuple). However, for more complex objects, weakrefs are a powerful tool that let you conserve memory among other things. It's worth balancing using something like WeakKeyDictionary versus a regular dictionary for caching.

I got sidelined, anyways...

Decorating Classes

As of Python 2.6, decorating classes is also supported. There's all the same reasons you'd want to decorate a function with the added benefit that you can apply metaclasses uniformly between Python 2 and 3. Or maybe it's just simpler than writing a metaclass in the first place. This next example comes from the six.py project and allows you to decorate a class to be applied in Python 2 and 3.

In [15]:

# from six.py
def add_metaclass(metaclass):
    """Class decorator for creating a class with a metaclass."""
    def wrapper(cls):
        orig_vars = cls.__dict__.copy()
        slots = orig_vars.get('__slots__')
        if slots is not None:
            if isinstance(slots, str):
                slots = [slots]
            for slots_var in slots:
                orig_vars.pop(slots_var)
        orig_vars.pop('__dict__', None)
        orig_vars.pop('__weakref__', None)
        return metaclass(cls.__name__, cls.__bases__, orig_vars)
    return wrapper

Applying multiple decorators

This generally works. Especially if you've used wraps or update_wrapper consistently. The important thing to note here is that decorators are evaluated from the bottom up. The first thing that happens is that Python defines the original object, then it walks up each decorator, passing the return value from each to the next decorator. That's pretty much all there is to stacking decorators, except for one thing that I'll cover in the next section.

In [16]:

@wrapperobj
@print_before
def random():
    return 4

random()

Setting up!

Out[16]:

4

One nice thing is if you remember to use wraps or update_wrapper, Python will very helpfully pass along added attributes as well (since they are really just key-value pairs sitting in __dict__). In case you forgot, pass_through is the first example in this section, it merely adds an attribute called inspected to an object.

In [17]:

@wrapperobj
@pass_through
def random():
    return 4

assert hasattr(random, 'inspected')
print(random.__dict__)

{'__annotations__': {}, 'f': <function random at 0x7f26c0039d08>, '__qualname__': 'random', '__wrapped__': <function random at 0x7f26c0039d08>, 'inspected': True, '__doc__': None, '__module__': '__main__', '__name__': 'random'}

Issues and Problems

As with anything, there are certain problems and issues you'll run into. Decorators are no exception. At some point, at least one of these will bite you, maybe hard.

Stacking wrong

This is something that will happen more often then you think. However, as long as you remember that they're evaluated from the bottom up and you've used wraps and update_wrapper through out, you'll probably be okay. Sometimes, it's important to note which decorators allow an object to pass through and which lock the object in a closure -- you shouldn't have to, but at least once you will.

However, there is something to be said for the @property, @classmethod and @staticmethod decorators. These return descriptors which are a slightly different sort of object that behave in a peculiar way. I've gone into detail about them else where, but the rule of thumb is to always apply these decorators last.

Testing

This is something of a sticky point. Pass through decorators are easy to test, those wrapped objects aren't hidden away anywhere. However, you're far more likely to encounter a closure decorator which makes testing much harder.

The easiest way is to use the __wrapped__ attribute if it's available. Python 3.4 does provide a very nice function in inspect called unwrap which continually unravels a decorated function until it's no longer wrapped up.

But if neither of these are available to you, and you're decorator does something like connect to a database or ensure a user's logged in, you might be in for a head ache.

Introspection and blowing stuff up.

More than once I've been bitten by bad introspection (looking at you, Google API wrapper), thankfully it was only me manually inspecting a function to see how to use it. Even using update_wrapper isn't enough to solve this problem.

• Argument specification
• Closely related is annotations in Python 3
• Ability to pull source code
• type and isinstance stop working

I'll admit blowing up code is one of the ways I learn the best. "What did I do that I shouldn't have done and how do I fix it?" Though, sometimes the errors are hard to track down.

Python 2's inspect.getargspec doesn't like being passed an object and complains loudly about it. Python 3's version doesn't seem to mind, though take that with a grain of salt because I'm sure there's a corner case somewhere that blows it up. Python 2 will display the argument specification and source code of the wrapper. Python 3 seems to do the right thing (at least in 3.4, and this may be because of inspect.unwrap and the __wrapped__ attribute).

Wrapping classes can seriously put a kink in your inheritance pattern if you're not careful about it. If your class simply passes through the decorator, you'll be okay. If it doesn't, you need to take care to make sure to either return the actual class (most likely using descriptors) or use something like __wrapped__ or inspect.unwrap in your inheritance line, both are which are ugly.

Similar to inheritance, isinstance isn't too happy about having a function passed to it. This behavior looks consistent between Python 2 and 3. The same fixes should apply here.

The final thing, which I'm unsure about fixing, is using type to identify explicitly what an object is. Descriptors seems like they should work and rudimentary testing with it seems to cause type to behave as expected, but I've not battle tested this -- and the real question is, "Why are you using type anyways?"

Actually fixing these

If you're much more curious about the proper way to fix these issues, I highly recommend Graham Dumpleton's blog series How you implemented your Python decorator is wrong. which goes into so much more depth than I have here and really opened my eyes to what I was actually doing when writing decorators beyond just stuffing one function inside of another. It also serves as a decent introduction into how descriptors work and practical uses of them.

No Man is an Island Unto Himself

This post wouldn't have been possible without the writings of Graham Dumpleton (sourced above), Jeff Knupp and Simeon Franklin. All three of these -- as well as countless other blog posts and StackOverflow questions -- introduced me and explained what decorators are to me. I'd also like to give thanks to both PyATL and Doug Hellman for sparking my interest in giving a presentation as well as the online learning group I'm a member of for sitting through about forty minutes of me listening to the sound of my own voice and asking for clarifications on many things I glossed over initially, thinking that people think the same way I do (which is probably a lot more twisted and convoluted than most).

I'm probably going to rewrite several other posts as well in a more frank "This is what's happening" manner instead of the usual "Here's a bunch of code, I hope you get it!" that makes sense to me more than anyone else.