Musings on Problem Solving

I just started reading a book on algorithms and data structures in Python. I've also commited myself to reading every chapter, taking notes and doing every exercise at the end of each chapter (okay, Chapter 1 is all "This is Python, this is the syntax, here's how you loop..." so I skimmed it) except for the ones I can't reasonably fit in a single IPython Notebook cell (I'm not trying shove a calculator into a notebook) or are just plain boring ("Print a statement 100 times!" -- seriously, this is an exercise).

Most of the problems are okay, made me think some. Some are ones I've done before. Others made me think about how to decompose the problem into smaller, more managable bits. And I'd like to share my process on one in particular.

Our Task

To paraphrase the exercise: We want to accept three integer inputs and determine if either a = f(b,c) or f(a,b) = c, where f is some mathematical function (maybe addition, modulo but log and pow are candidates as well). Right now, if you feel this is a complex challenge, I want you attempt to code it and come back and see how your code and my code differs. If you get stuck, don't feel bad and come along for the ride.

The Steps

The first thing I like to try to do is break a problem down into steps. If a step is ambigous, I try to decompose the step into smaller bits. Eventually, I have kinda procedural pseudo-code.

1. Collect Operations
2. Collect Integers
3. Map inputs to operations
4. Check if at least one operation returned true
5. Put the whole thing together.

To me, those are pretty clear cut, atomic steps. Sure, they're composed of smaller steps internally, but they represent a single unit of work each function of the program needs to do. Step 5 is usually implied, but I find it's nice to list anyways.

Step 1: Collect Operations

We'll just make a list of all the operations we want to check against. Since Python has first class functions (meaning functions can be treated like data), this is really easy. We'll also make a simple root function for calculating things like cubic roots as well.

In [11]:

from operator import add, sub, mul, mod, truediv
from math import log

# roots are just fractional exponentials when you get down to it
# ...or maybe exponentials are fractional roots...
def root(base, power=2):
    return base ** (1/power)

operations = [add, sub, mul, mod, truediv, log, root, pow]

Super simple. Now, let's move on.

Step 3: Map Inputs To Operations

"Alec," you say, "You skipped a step!" And I did. Because I/O is nasty and I'd like to do it last. For now, let's assume the integers we collected are 1, 2 and 3.

In [3]:

integers = [1,2,3]

For this step we need to make our operation to a potential output. Basically: f(a, b) == c. Again, this is easy. Even if we decide to check if a == f(b,c) in the same step. Like I said before, Python's functions can be treated just like data and this includes passing them as arguments to other functions.

In [4]:

def either_side_true(op, a, b, c):
    return a == op(b, c) or op(a,b) == c

Step 4: Determine if any check returned True

There's many ways we could do this. But the key here is the word any -- which happens to be a Python built-in function. What any does is accept an iterable of values and checks if at least one of them is True. There's also its sibling all which checks that none of the values are False. What's really nice about any and all is they allow collapsing a bunch of and/or groupings in an if statement into a single expression.

Say you wanted to check if any variable in a given group was equal to something. You might do (ignoring that you'd actually have a list and do if 1 in list_of_vars:)...

In [6]:

a = 2
b = 1
c = 5

if a == 1 or b == 1 or c == 1:
    print("We've got a winner!")

We've got a winner!

...but with any we can simply do this...

In [7]:

if any(v == 1 for v in [a,b,c]):
    print("I said We've got a winner!")

I said We've got a winner!

Of course, the true power of any and all comes when you combine them with generators, which will allow you to lazily compute your check. These aren't a catch all, sometimes you want to store the output of a check or you already have a list built, in which case x in my_list is superior, but these should be tools you consider from time to time.

Now, we want to check if any of our operations returns True if a == f(c,b) or c == f(a,b).

In [8]:

if any(either_side_true(op, *integers) for op in operations):
    print("Heck yeah at least one did.")

Heck yeah at least one did.

Alternatively, we could store that as a separate function if we needed to fiddle with it more, but for this case, I feel that's overkill.

Step 2: Collect Integers

Okay, now we'll worry about actually grabbing some input from the user. We need to collect three integers, probably using STDIN. But they could come from a file, a database or anywhere. But let's focus on just grabbing them from STDIN.

In [9]:

def collect_ints_from_stdin(n=3):
    ints = []
    
    while len(ints) < n:
        try:
            v = int(input("Please type an integer: "))
        except ValueError:
            continue
        else:
            ints.append(v)
    
    return ints

Normally, I'd wince at a while loop used to build a fixed length list, but we have to trouble with users doing something like trying to enter the number a (well...you could use hex...but who does that?). Since we're just passing on the error, our list of integers could end up short!

You might also notice that I've used try/else. I am of the opinion that a try block should be isolated to just the code that could raise an exception. except handles cleaning up the exception. And then there's else which allows us to run code on the condition we didn't encounter a problem. Mentally, if you replace except ValueError with if catch(ValueError) where catch is some magic framehack or something, it begins to make sense. I'll also note, for completion's sake that there's also an optional finally clause which lets us run code regardless of an exception occuring. But the full try/except/else/finally concept is for another time.

Step 5: Put it altogther

Finally, we'll take these individual components and piece them together into a cohesive answer.

In [12]:

def answer(operations, num_ints=3, collector=collect_ints_from_stdin):
    ints = collector(num_ints)
    if any(either_side_true(op, *ints) for op in operations):
        print("Heck yeah at least one did!")
    else:
        print("D: Nope!")

If you add if __name__ == "__main__": answer(operations) and count spacing and imports, the whole thing's less than forty lines!

The Take Away

1. Break a problem down into steps as small as possible
2. Small, functional pieces are more manageable

Breaking down a problem is something that seems so obvious but it's a skill that needs to be practiced and honed just like any other. After all, programming isn't so much about code as solving problems. And part of solving problems is finding the smaller problems hidden inside them.

What looked liked a complicated task on the surface, ended up being just five simple tasks on the inside. That's not to say that every problem will immediately break down to simple tasks. Sometimes you'll end up with a list of steps that's nested five levels deep! But if you tackle each step one at a time, sprinkle in some Python idioms and a touch of abstraction, your application will come together.

The other, writing smaller more functional code pieces allows your code to be modular. What if tomorrow, we want to grab the integers from a database? Well, since collect_ints_from_stdin is separated from the main function, we can easily. Just build a callable that queries the database for integers and feeds them into the operations. Or if we need to change from integers to strings, we just swap out the collector and the operations and move on with our day -- or better, pass a callback to coerce the type for us!

But consider if we had just written a thirty line function that did all this? Ugh, it'd be a nightmare to untangle the pieces. Honestly, we'd probably have two functions that did the same thing, but just grabbed data from other locations. And then two more functions because maybe we need to work on strings. Quickly, this spirals out of control as the number of functions grows at (Data Types) * (Sources). We have four sources and four data types? That's 16 functions! I don't want to write the same function 16 times and I doubt you do either.

Instead we can write four functions to grab the data, pass our coercion callback and maintain the lists of operations.

Going Further

There are problems. What happens when we input 0 and our checks make it to the division, modulo, log or root operations? We get a nasty exception. :( But handling it is easy (hint: ZeroDivisionError and ValueError would needed to be caught). Also, we should be able to tell the user which operations returned true.

Of course, I'm not going to play my full hand. I'll leave these as an exercise to you to figure out.

Self-Describing Functions

This post isn't about writing functions. It's not about what *args, **kwargs means, closures, decorators, callbacks, or def. It's about writing self-describing functions. That nebulous term that everyone knows they should be doing and has an idea of what it means, but no one really knows what it means.

Before I dive into this, I will point out that way smarter people than I have written entire books about this very thing. This is more a distillation of what I've found best works for me. Your experience and philosophies on software design may lead you down an alternative path.

Since functions are the most basic code block in Python, it's easiest to lay out my thoughts with them. From there, they're easily extrapolated to classes, modules and packages.

Describing Self-Describing

Since self-describing is like "synergy" or "webscale" where people have ideas and vague defitions of what it means to be "synergistic" or "webscale" but mostly it ends up sounding like a CutCo sales pitch.

To mean, self-describing means a number of things, but the most important bits are:

• Well Named
• Avoids "Ands"
• Documentation
• Clear, Concise Code

Well Named

There's a joke in metal circles that the late guitarist for Mayhem -- Euronymous -- chose his name because he thought it was a Greek demon, but actually it's Greek for "well named." I'm unsure of the validity of the rumor (It's all Greek to me), but I've always been amused by it.

The importance of choosing a good name for functions can't be overstated. Since functions take a snippet of code and turns it into a reusable piece, when you see that piece being used you need to know exactly what it's doing. For a minute, let's place ourselves as researchers in the field of Frobincated Mathematics. We spend all day calculating frobs. Notebooks, white boards and even our code bases are littered with the Frobincator Theorem:

In []:

frob = (a + b) * x

A simple, yet powerful formula in our every day work to find ever bigger and better frobs. However, we type it out a lot. One of our CompSci friends points out, "Hey, why don't you make that a function instead of typing it everywhere?" To which leads a debate on the differences between mathematical and programmtic functions that is neither here nor there. Eventually, we do and we end up with this:

In [1]:

def calc(a, b, x):
    return (a + b) * x

See any problems? I do. While we've taken a common piece of our code base and made it reusable, we've actually obfuscated our intent! calc? calc what? Pythag triples? Fibonnaci numbers? Our most precious frobs? If we had to get our CompSci buddy to help us debug an issue, he's gonna wonder exactly that. He'll either ask us what it means and does, or he'll run grep -ir "def calc" . to find it. Either way, we've done ourselves a great disservice. But instead, if we change it, ever so slightly, to this:

In [2]:

def calc_frob(a, b, x):
    ...

Then, everyone knows that it's calculating frobs! Sure, we may still need to track down the actual source code to see if it's causing problems, but there's no headscratching at what it does.

Currently, in one of my projects' code base, there is a function that isn't well named. It's called find. What does find find? Well...actually, it's more of a filter than a finder. Sure it uses os.walk to get file names but it doesn't really find things as it makes sure we only grab the correct file types (based on the fool-proof way of trusting the extension...). Had I named it filter_file_exts or sift_extensions, there wouldn't be an issue.

But sadly, I have learned the hard way, so you don't have to. Recently, I had to revisit some old PHP I wrote between two and four years ago -- it was a time of drunken code debauchery. Some day you'll write code that you'll need to revise two years -- or even two days -- later and you'll look at a function call and go "What in the world does \manager\extractAll even do?" Do yourself a favor now, and starting naming and renaming functions to better things. Even if naming things is hard

The same logic applies to variables -- inside and outside functions. In our calc_frob function, it makes sense what a, b and x are doing. But if you have a thirty line function, will you remember what x means by line fifteen? If you're dealing with interest on a loan, try naming it interest_percent or if that's too long percent if it's still clear.

There are short hand variables that most people will recognize, like fh for a file handle. And if you're opening (and closing!) a single file, then name it fh. It won't confuse me or you or anyone else involved. But, if you're opening two files -- maybe you're shuffling data between them -- then don't name them fh1 and fh2 but consider source_fh and target_fh.

Avoids "Ands"

By "ands" I of course don't mean things like boolean ands or if/elif/else blocks, rather I mean when you're describing the function you don't need to use the word "and" to get it's job across. "This functions gathers a list of file names AND filters them." Why not write a function that just filters the file names and pass it an iterable of file names?

This ties into a later point, but in that same project code base there's a function called "store_directory" -- which I consider well-ish named, since it makes sense in the context of it's module. This thing is 54 lines long and does way more than just stuff ID3 info into a database: outputs to the screen, times the operation, counts the number of files, massages ID3 info into a dictionary, converts the dictionary to data models, figures out if genres are being tagged, what genres to tag, who's tagging, AND stuffs the data into the database. To be a little fair to myself, some of that work (not much) is farmed out to other functions, but it's a mess of try-except-finally blocks, loops, conditionals. The actually important part of the function is so buried, when I get around to refactoring it, I'll probably write a Cliff's Notes comment about it and just :dG. Good bye horrible function!

Why? Because my application isn't a single monolothic function that does twelve different things. It's a series of cooperative pieces that work together to do something. If your coffee grinder also had a salad mixer built in, would you consider it a well designed coffee grinder? Who even has salad and coffee together? Maybe a fruit salad for breakfast, but still.

By contrast, there's a function called break_tag at the top of the file which has six lines of documentation and four examples for 1 line of code:

In [3]:

def break_tag(tag, breakers=('\\\\', '/', '&', ',', ' ',  '\\\.')):
    '''Breaks a composite tag into smaller pieces based on certain
    punctuation. Smaller tags are stripped for excess white space before
    being placed into a set.
    
    .. code-block:: python
        break_tag('viking / folk')  # -> {'viking', 'folk'}
        break_tag('dance & pop') # -> {'dance', 'pop'}
        break_tag('thrash metal') # -> {'thash', 'metal'}
        break_tag('progressive metal / black metal')
            # -> {'progressive', 'black', 'metal'}
    
    :param tag str: The tag to be analyzed and broken.
    :param breakers iterable: A group of characters to break larger tags with.
    :returns set:
    '''
    return set(t.strip() for t in re.split('|'.join(breakers), tag))

Is there a technical and in there? Sure, if you want to split hairs, it does break a larger tag down and strips white space off the smaller bits, but if you (like me) consider stripping the white space off to be breaking the bigger tag down, then there's not. Plus I didn't have to make up some obtuse phrase like "atomic tags."

Rather than writing functions that are described by "and," why not write pure functions and pass them data? Pure functions are wonderful: they don't have side effects, they do one thing, you can have deterministic results!

Think of the simple unit tests you can write! You know how I'd test break_tag? I'd give it three inputs that should do the right thing, and three inputs that should do the "wrong" thing and verify those. How would I test store_directory? ...I didn't, becuase by the time I started mocking and patching things to make it "testable" I'd written about ninety lines of code that was just setup. And then I threw it away and called it a failure.

Will you always avoid "ands?" No, because just like your application isn't a single god function, it also isn't a set of completely independent operations. You can't expect a black box with no entry point and no exit point to do anything. It won't even heat up when you turn it on because you can't turn it on. You'll have "ands" but they should appear where you're retrieving data from I/O, processing it in the pure functions and then spitting it back out to I/O. "Ands" should be used to wire pieces together.

Documentation

The first thing I do after I finishing writing def frob(a, b, x): is to immediately open a docstring. To me, docstrings are the function, they describe what a function does, what the different parameters are and might even provide example usage or doctests. Below the docstring is just a language specific implementation. (a + b) * x is obvious to the point where you might think, "Do I need a docstring?" The answer is yes.

In [4]:

def calc_frob(a, b, x):
    """Calculates a frob based on the Frobincator Theorem."""
    return (a + b) * x

In addition to giving it a concise name, we've also given more information to those that come behind us. Maybe they're unfamiliar with frobincated mathematics and don't know the theorem. Now they do.

Now, what I mean about language specific implementation is that even though we can write the same exact function with the same implementation in Haskell, someone could write it like this:

In []:

frob a b x = (*) x $ (+) a b

Which looks completely different but does the exact same thing. A simple comment or docstring above it stating, "This is the Frobincator Theorem" makes it's immediately clear what the function does.

Even better, Python makes it ridiculously easy to look at docstrings without having to open a text editor!

In [5]:

print(calc_frob.__doc__)

Calculates a frob based on the Frobincator Theorem.

You can do that in a script or in the shell. IPython does it one better with the ? and ?? line magicks. ? displays the argument signature (if available) and the docstring if it's present, along with a few more things. ?? will pull any Python source code (not C/Java/Whatever, just pure Python) available for the function (or class, object, module, etc).

Since Python makes it easy to get to the docstrings, then I can't think of a valid reason not to write them. And no, "I was lazy" isn't a valid excuse -- even if I've used it before.

Another important aspect of documentation is comments. There's people that will tell you if you have comments, then you've done something wrong. I completely disagree with that notion. Consider this field in a SQLAlchemy model:

In []:

_trackpositions = db.relationship(
    'TrackPosition',
    backref='tracklist',
    order_by='TrackPosition.position',
    collection_class=ordering_list('position'),
   )

Are you familiar with the collection_class keyword? Or ordering_list? I wasn't once and chances are, one day someone else who isn't will stumble across that field, scratch their head and let out an audible "WTF?" before either giving up or looking up the meaning. But with the original comment in place...

In []:

_trackpositions = db.relationship(
    'TrackPosition',
    backref='tracklist',
    # ordering_list will automatically update the position attribute
    # on the proxied Tracks. However, it must also be fed a correct
    # inital ordering.
    order_by='TrackPosition.position',
    collection_class=ordering_list('position'),
   )

...the basic intent is communicated. If docstrings are the definition of the function, then comments are just notes on the language specific implementation of it. When you're using someone else's code, you can't just change the name of something, nor does your intent always get cleanly communicated through the code. However, comments allow you to clarify what is meant. They're also helpful for noting what needs to be improved.

Just real quick, cd over to a project you have and run grep -ir todo . and see what pops out. I have a todo in another project that reads: "#TODO: cleanly handle multiple endpoints" in the context of JSONPath. I didn't want to spend two hours at the time writing the implementation that would muddy up an otherwise nice looking function and vim is ever so helpful and highlights it an really ugly yellow that I feel the compulsive urge to remove.

This doesn't mean you should place comments willy nilly all over the place, for example this comment isn't helpful at all:

In [6]:

def frob(a, b, x):
    #multiples x by the sum of a and b
    return (a + b) * x

But when you come across a piece of code you've written that maybe needs a little clarification on how it works, then toss a comment on there for mental health's sake.

However, documentation and comments are no reason not to write...

Clean, Concise Code

This is the final point and the least important of them to me. If you've named your function well, avoided doing "And-y" things and you've written a nice docstring and some comments, I can stomach poorly written code. I can reason about what it's supposed to do. However, I do ask you to do your best.

When you actually get around to writing code, remember Einstein's take on Occam's Razor: Everything should be made as simple as possible, but not simpler. I'm not actually sure if Einstein said that, but it's widely attributed to him. This is the approach we should take to writing code and it ties in greatly with the "Avoid And" principle and pure functions.

Consider building a basic Markov Chain with a dictionary and lists. There are three ways to ensure a list of words exists at a keyword. The most basic is this:

In [7]:

markov = {}

def add_entry(chain, state, new):
    if state not in chain:
        chain[state] = []
    chain[state].append(new)

However, Python dictionaries have a setdefault method which checks if the key exists already and if not, creates a default entry for it.

In [8]:

markov = {}

def add_entry(chain, state, new):
    chain.setdefault(state, []).append(new)

But by and afar my favorite way to handle this is with collections.defaultdict:

In [9]:

from collections import defaultdict

markov = defaultdict(list)

def add_entry(chain, state, new):
    chain[state].append(new)

I'd consider all three clean and concise, and all equally appropriate. However, as much as I love defaultdict the last version doesn't communicate the full intent that we're intending to use one. It makes things too simple, whereas the first two communicate that "Hey, this new state might not exist in the chain yet." Though in actual practice, it would depend on how self-contained the actual chains were is how you'd decide which method to us.

Parting Thoughts

Like I said, this stuff is easily extrapolated to classes all the way up to whole applications. And I will also admit that the avoiding ands is largely inspired by the Clean Architecture and Onion Architecture design patterns I've been reading about lately, and specifically from Brandon Rhodes's The Clean Architecture in Python presentation at PyOhio 2014.

If you've got some poorly named, undocumented or and-y functions in your projects, I urge you to go and look at them and see what can be changed to help your project. I know not every function name can be changed easily, especially if it shares a name with something or is a common word. But if you can change it easily and it improves your project, why not?

Resolutions

I've often been told that New Year's Resolutions are something we tell ourselves to make ourselves feel better. Or at least something along those lines. How many times have I told myself, "I'm going to start working out this year!" Too many to count.

However, something I try to live by is learning or doing something new every day. Especially if it seems small and inconsequential. Small and inconsequential things are the foundations of greater things. Today, I may finally grasp what monads are. You might learn how to make a bechamel. Someone else might learn how to thread a needle. Somewhere, a child is kicking a soccer ball for the first time.

Taken by themselves, these things are small and inconsequential. However, monads are the foundation for complex Haskell (and other functional) programs. Bechamels are used to form all sorts of sauces. Threading a needle leads to becoming a designer. That kid kicking a ball for the first time might become the next Pele.

Learning these small things lead to so much more. Giving up after not noticing any progress after a week, two weeks, a month. But if you stick with it and push yourself every day on some front, you'll be able to look back and say, "I didn't even know what a roux was back then. Now, I run a restaurant!"

Why share all this?

I've made a short list of resolutions that quite honestly, will mutate and change over the year and the years to come. These things are specific to me so you might be wondering, "Alec, why do I care?" It's fine if you don't. However, I'm sharing them in an attempt to keep myself accountable for what I set out to do. In fact, being more public is one of my goals.

1: Becoming Less Introverted

I don't find anything necessarily wrong with being introverted. It's gotten me this far in life, why not another 26 years? I'm not expecting to become the life of the party and go out every night to hang out with a large group of people. That's not me, nor do I envision that happening.

However, just a few days I gave a presentation at my local Python meetup. I knew my topic well (Decorators) but speaking publically is terrifying to me. Even though I had done everything I'd ever heard of when attempting to give a presentation, I got really nervous. Not because I was worried about the well-known guys in the room, but because I don't like being the center of attention at all -- so much to the point, I don't tell many people when my birthday is. I've done things where I'm forced to be the center of attention at least some of the time -- running a D&D campaign is kinda hard to do if the DM doesn't draw any attention to himself. But being in front of six friends is quite different than being in front of thirty-five people you don't know.

I don't make friends easy because of this. I'd rather be at home reading something or watching a movie than where ever I am currently. And because of that, I miss out on experiences and potential friends.

It's also hindered me professionally. I don't interview well. I know my stuff when it comes to Python webdev or running a retail store. But in an interview, I choke up and get nervous. I can sell you ice in a snowstorm or whip up a basic CRUD app in no time, but when it comes to selling myself...eh, I'm not so great. Part of it is being introverted and another part of it is my self-esteem. But, I keep finding myself wishing I had taken that chance or done something bold. Instead, I end up beating myself up for something "stupid" I did rather than at least looking for the learning opprotunity in it. Or completely half-assing something so I'm not overly committed when I get cold feet.

Throwing myself into stuff with reckless abandon would be the complete opposite of what I am now. While I don't want to be that, I want to be a little more like that. There's no reason why I shouldn't be able to give a presentation of something I know well to thirty-something people. There's no reason why I shouldn't be able to sell myself.

2: Linux and BASH

I use Linux and BASH every day. The only time I'm not using a Linux system is at work (which uses probably the worst POS system I've ever seen). My laptop's had some variant of Linux on it almost from day one. My tower, my RasPis. I'll even technically count my phone.

However, a few weeks ago my uncle quizzed me on the Linux filesystem. Turns out I don't know as much as I thought I did. Yeah, I got Arch up and running (third times a charm) and I can modify my .bashrc and .bash_alias files to make my life easier. I use ssh and ~/.ssh/config and do most of my work from the command line. But really, at the heart of it, I feel like a script kiddy living in a hacker's world.

Currently, I'm experience a strange networking issue that seems to be two fold:

• My tower's networking process seems to live in a quantum state of running and stopped. And it's always the opposite of what I expect it to be (even if I try tricking it).
• My bridge router selectively hinders traffic behind it. Sometimes the tower connects just fine and other times, I'm about two steps from pulling my hair out. Everything behind the bridge communicates just fine. And the bridge and main router talk just fine. But moving beyond the bridge is...difficult sometimes.

I got no idea what to do. Sure, I can ifconfig eth0 and traceroute and ping and dhcp -r eth0 all day long. But I'm just aping what I've seen other people say. Looks like it's happening right now despite the WiFi dongle.

I'm not saying that I want to be network adminstrator (though, that'd be continuing the family tradition), but I should know enough to actually figure out what the heck is going on here. I know enough to be dangerous but that's about it.

And there's still quite about BASH itself I don't know. I'm not wanting to develop complex shell scripts, but there's so much that can be done with a few lines of BASH that would take many more in Python. I'm not saying BASH is better than Python, but if I want to create sequential backups of a drive and store a log of each backup and do this every night at 3AM, I can do that in a few lines of BASH. Python's great for gluing programs together, BASH is better at this task.

3: Bass

I have a bass. It's a decent Peavy four string. I've played it some over the years, but I never really devoted a ton of time to learning how to play. In high school, I took some lessons, but like people who commit to working out, I gave up after two months of not noticing gains -- despite that they were happening. Now that I'm a little older and a little wiser, I figure it's time to give it a real shot.

Music is something that is important to me. I enjoy listening to it, working with it and sharing it. Why not produce some of my own? Not necessarily forming a band or putting videos on youtube. But just playing to play and enjoy it. Besides, since I stopped working on cars, my hands have gotten soft.

4: Enroll in School

I've always kicked myself for not enrolling in college. Even though I would have ended up an English major -- probably -- if I'd done right out of high school, having a college degree is now more important than ever. I can self-teach myself Python and SQL and Haskell all day but the truth is, I know just enough to get by and recognize that there's so much more that I don't know. The worst part of knowing there's so much more out there is that I don't even know where to start or if I have the right skills for it to make sense to me.

Endofunctors? Linked lists? Pointers? Or even something as seemingly mundane as SOLID. Like I get the idea, but I truly have no idea. It's a thing that does a thing and holds the data.

I know there's the whole circlejerk of "You don't need a degree to get a job in CompSci!" and I'm sure that's technically true: You're not going to bring the piece of paper that says, "Alec Reiter graduate from University of Derp blah blah blah blah" to an interview. But it's what the degree represents, "Alec Reiter has successfully completely a program ensuring he knows the basics of CompSci!"

I can self-teach CompSci and I might learn it better that way than through sitting in a class room two or three times a week. But in twelve years of teaching myself compsci, I've had one interview and I've produced only a handful of non-impressive projects. Just being a 100% honest here. I'm not impressed with any of my works. And that's because I feel like I'm grasping at straws with some of the things I do and write because I don't have the full picture. I feel the inner rockstar wizard ninja programming in me, but I'm not sure how to let it out.

On top of all of that, there's other stuff I'm interested in that seem completely daunting to self-teach (at least to me): Electrical Engineering (ask me about my Grand Unified Coffee Pot sometime), Physics, Mathematics and Philosphy have piqued my interests many times. I'm not saying that I expect to great at all these, but I'd like to know more.

5: Become a marketable Web Developer

This ties in with the last resolution. I can't and won't say that my heart will always lie with Web Development. It's surely a bustling and busy field that is as varied as CompSci in general. But it's what I'm pretty good at right now. I like the internet. I like programming. Why not program the internet?

However, despite the fact that I'm a passable backend programmer (or is the term engineer now?), I'm a dreadful frontend guy. It comes to make the front end and I'm just like, "Javascript...oh wait, this query's busted let me fix that first."

Time to stop that. I can make the most beautiful REST Api in the world, have a gorgeous query and it's all wrapped in a nice neat OOP app, but if I can make even a generic frontend for it...what's the point? I know not ever company is looking for a "full-stack" or "end-to-end" developer -- someone that that implements a feature from database to CSS -- but many are at least looking for someone who can do that in a pinch.

I don't know Javascript. Frankly, I don't want to learn Javascript. An acquaintance told me, "Javascript is what happens when you drink too much, smoke too much and have to design a language in too little time." However, in order to do any modern web programming, Javascript is the only answer. Which saddens me, I'd love to see Python gain a true client side implementation (not just compiling to Javascript). But until that day, I'll guess I'll be learning me a Javascript for great good.

On top of that, I'm resolving myself to using Django. Django's another thing I don't like (though, I've only used it some). I much prefer the completely indifferent Flask (and I mean that affectionately). However, there's precisely zero job opportunities for Flask here -- well, that's probably not entirely true, I just haven't found any. While I may find Flask to be superior and use it for my personal projects, if I want to be marketable I'll have to pick up Django. Besides, being a one trick pony is no-good.

Other stuff

Of course, there's always smaller stuff that I'd like to do. I have a short list of languages I'd like to be passable in:

• Python
• BASH
• Haskell
• C/C++
• Javascript
• SQL

Some of these I know alright or have a decent grasp on and other's not so much. But I feel it's a well rounded list. BASH, Python and Javascript are all scripting languages but lend themselves to wildly different tasks. SQL, to me, is a complete must to anyone interacting with a database (even if it's through an ORM, and especially in that case). C/C++ for "actual programming" or at least to familiarize myself with them, plus many extensions to the other languages are written in C/C++. And Haskell because it interests me and it's radically different to the others. Not to mention the little bit I've learned has already helped me understand things like "The Clean Architecture" (the idea of separating I/O and data transformation completely) and OOP (Haskell's typeclasses are just data holders and rules for how functions interact with the data...sounds like OOP to me!) And of course, I'd like to explore the new hotness languages like D, Clojure/Scala, Go, Rust and others just because.

I'd like to actually stop smoking. I vaped on a custom 50watt box mod for a while and enjoyed it...until it was time to wrap my own coils. Given I have an IGO-W2 with a stripped post, a piece of shit something that's drilled out poorly and a 454 Big Block which has horizontal coils, I'm not making it the easiest on myself. But you know what? I actually enjoyed it otherwise. And if a little bit of elbow grease is needed, all the better.

Working on my car again is another thing. I'd be surprised if there wasn't at least one place in Atlanta where I can rent a bay and some tools for a day. I'm not at the point where I'm gonna rebuild the motor, but there's a few things I need to do to make my ride smoother -- motor mounts, inspecting the suspension, determining if I should replace the tranny fluid (rule of thumb: if you need to ask, the answer's probably don't).

Making use of my Raspberry Pis. I have two. What the heck do I do with them? Operate a coffee pot? A quick and dirty NAS? Emulate a smart TV? Emulate vidya? Run a music server? They just kinda sit there and blink at me from time to time. I kinda feel like Richmond, "And this one: flash, flash, flash and then wait for it. Nothing for a while. Here it comes. Double flash."

I'd love to start playing D&D or at least some game on a regular (or semi-regular) basis again.

Accountability

Like I said, I'm sharing all this to hold myself more accountable. To continue to do that, I'm going to keep writing about these topics. Maybe not every day and not exclusively about these topics, but as much and as often as I can. Different things I've learned or done that have built up into something bigger and better.

I look forward to 2015 and seeing how it shapes and changes my life. Hopefully you are, too!

Iterate All The Things

So after the rousing success of making int iterable (which I now know I could have done with ForbiddenFruit), I started wondering, "Why aren't classes iterable?"

In [1]:

from itertools import repeat

class IterCls(type):
    
    def __iter__(self):
        return repeat(self)
            
class Thing(metaclass=IterCls):
    
    def __init__(self, frob):
        self.frob = frob
    
    def __iter__(self):
        return repeat(self.frob)

So, that's a thing. Works just like you'd expect. Any class that declares IterCls to be it's metaclass can be iterated over unendingly.

In [2]:

from itertools import islice

list(islice(Thing, 3))

Out[2]:

[__main__.Thing, __main__.Thing, __main__.Thing]

And yes, the __iter__ on the actual Thing class works, too.

In [3]:

t = Thing(4)

list(islice(t, 10))

Out[3]:

[4, 4, 4, 4, 4, 4, 4, 4, 4, 4]

But check this out:

In [4]:

from inspect import getsource

print(getsource(Thing.__iter__))

    def __iter__(self):
        return repeat(self.frob)

Odd, huh? I'm green around the ears with metaclasses so I'm not even 10% what's going on here other than maybe since Thing is an instance of IterCls (classes are objects), it's instance dictionary would defer to the class dictionary in IterCls when looking for methods. Hell if I know right now.

I'm not really sure what you'd with this. Maybe hook some sort of alternative initializer method on there? However, without using some sort of global or stashing class attributes (or are they instance variables in this case?), I don't think it'd be incredibly useful. And the two argument version of iter with a factory function would be much clearer and way less magical. Something like this:

In [5]:

from itertools import count

def ThingFactory(start=0, step=1):
    frobs = count(start, step)
    def maker():
        nonlocal frobs
        return Thing(frob=next(frobs))
    return maker

for f in islice(iter(ThingFactory(), None), 3):
    print(f.frob)

0
1
2

And, in case, you're wondering, yes modules themselves can be made iterable as well. Inspired by fuckit module fuckery.

In [6]:

from runnables import itermodule

print(itermodule)
print(list(islice(itermodule, 4)))

<module 'wtf'>
[4, 4, 4, 4]

It's a module that implements a __iter__. And it just spits out 4 all day long. Code here I was musing about it in ##learnpython on Freenode and one user commented it might maybe possibly be useful as a datatype. Import the module and use it to represent a CSV file for example -- but the real question being, "Why not just use a class in that case?"

Why?

I was bored. Wanted to see what Python would let me get away with in terms of making things iterable. At this point, I'd be confident that everything can be made iterable. Object method?

In [7]:

from functools import partial

class IterMethod:
    def __init__(self, f):
        self.f = f
    def __get__(self, inst, cls):
        f = self.f
        if inst:
            f = partial(f, inst)
        return repeat(f)

class Thing:
    
    @IterMethod
    def frob(self, frob):
        return frob
    
print("As instance method")
for f in islice(Thing().frob, 2):
    print(f(4), end=' ')

print("\nAs class method")    
for f in islice(Thing.frob, 2):
    print(f(None, 5), end=' ')

As instance method
4 4 
As class method
5 5 

Though, I think a straight up @property would be clearer and probably more in line with what was expected:

In [8]:

class Thing:
    
    def __init__(self):
        self.frobs = count()
    
    @property
    def frob(self):
        return iter(self.frobs.__next__, None)

            
t = Thing()
print(list(islice(t.frob, 5)))
print(list(islice(t.frob, 5)))

[0, 1, 2, 3, 4]
[5, 6, 7, 8, 9]

Iterable function? Use repeat as a decorator...actually, don't really do that. Just use repeat as normal. But in the face of boredom, clearer minds rarely prevail.

In [9]:

@repeat
def frob():
    return 4

fs = [f() for f in islice(frob, 4)]
print(fs)

[4, 4, 4, 4]

I guess don't do this at home? I can't really think of any practical applications for these sorts of things. But if you need to do them...I guess use this as a reference point? Actually, I can think of an application of an iterable function: composing a function N times. I've borrowed the compose function from here:

In [10]:

from functools import reduce

def compose(*functions):
    def compose2(f, g):
        return lambda x: f(g(x))
    return reduce(compose2, functions)

def frob(x):
    return x + 4

n_times = compose(*repeat(frob, 4))
print(n_times(0))

16

If you find any other useful applications, let me know, I'll gladly add them as examples.

Understanding foo.bar()

One of my favorite shows is "How It's Made" -- my enjoyment mostly stems from learning how stuff is made, but the narrator's cheeky puns and jokes certainly add to it. But something I enjoy more than knowing how stuff is put together, is knowing how things work. I don't know what it is, but I have this childlike fascination with opening things up and learning how it fits together, what each part. That was one of my favorite things about my brief stint (a whooping six months!) in the automative service industry: understanding, a little better, how cars work. It certainly opened my eyes to all the work that goes into even simple automotive repairs.

Sadly, I no longer work on or with cars, I do still fiddle some with mine though, and if anyone has a good link to how a transmission -- manual or automatic -- actually works, I'd be thrilled! But this has left me with a hole in my life. One I've recently begun to fill with how Python operates under the hood -- so to speak. While my skills with C -- which basically amount to printf and for loops -- leave me woefully unprepared to examine much of the source, I can examine the surface parts.

To use a car analogy, if reading the C source for Python is repairing a damaged block or transmission, examining how Python works is more similar to replacing motor mounts and broken belts (something I'm regretfully too familiar with on my CRV). Whereas reading someone else's Python is like doing your own fluid changes. Flawed analogies aside, I'd like to more fully examine how Python objects work and what it really means to call foo.bar().

As a forewarning, this knowledge is great for understanding what's happening, but it's not crucial knowledge to working with classes and objects in the regular sense. All the things I will discuss here deal with how Python 3 handles them. Python 2 is slightly different.

Building a Class

To talk about Python's data model and how it relates to classes and objects, we should first write a class. It so basic as to wonder why we're doing it. The point is, rather than examine some fictional class or object, why not have one of our own to open up and poke at?

In [1]:

class Baz:
    
    def __init__(self, thing):
        self.thing = thing
    
    def bar(self):
        print(self.thing)

That's an extremely basic object. The initalizer takes a single argument a method that prints it out. Of course, we need to instantiate it for us to get use out of it.

In [2]:

foo = Baz(1)

Already, there's some mechanisms at work for us. I don't want to get too deep into class creation, but the short take away is the implicit __new__ classes inherit from object handle object creation and __init__ simply sets the initial state of the object for us.. Delving into __new__ hooks into dealing with metaclasses, which is a topic for another time. What I want to focus on today is what happens when we call foo.bar()

Classes and Objects

You'll often hear that objects and classes in Python are simply nothing more than a pile of dictionaries with dotted access. This obtuse phrasing confused me for a long time and it wasn't until I began asking, "How the heck does self actually get passed?" that I began to understand. Asking this began me down a rabbit hole that lead me to descriptors and __getattribute__ and what they do.

The Dict

All classes in Python have an underlying __dict__ and nearly every instance does as well. The first step to foo.bar() is understanding that methods live at the class level.

In [3]:

print('bar' in Baz.__dict__)
print('bar' in foo.__dict__)

True
False

Methods are entries in the class's underlying __dict__ but not in the instance's. Because of this, most Python objects can remain relatively small, they simply store their state rather than all of their available methods as well. What does this method look like in the dictionary?

In [4]:

from inspect import isfunction, ismethod

print(isfunction(Baz.__dict__['bar']))
print(ismethod(Baz.__dict__['bar']))
print(Baz.__dict__['bar'])

True
False
<function Baz.bar at 0x7f1d05a87ea0>

We can see that in the class's dictionary, methods are stored as functions and not as methods. It's reasonable to infer that methods are actually functions that operate on class instances. From here, we can imagine that behind the scenes

In [5]:

Baz.__dict__['bar'](foo)

1

Attribute Access

The next piece of the puzzle is how Python handles attribute access. If you're not familiar with how Python attribute look up happens, in short, it looks like this:

• Call __getattribute__
• Is the attribute in the object __dict__?
• No? Is the attribute in the class's __dict__?
• No? Is the attribute in any of the parent classes' __dict__?
• No? Call __getattr__ if present.
• Else, raise an AttributeError

Python starts at the bottom, calling __getattribute__. This what actually allows the dotted access. You can think of the . in foo.bar to be implicit call to this method. This method translates dictionary look up to dotted access and invokes the rest of the chain. Since we already know that methods live in the class's __dict__ and methods are functions that act on the instance, we'll fast forward to there and extrapolate.

Since methods are functions that live in the class's dictionary and act on instances and __getattribute__ is an implicit transformation from attribute to dictionary look up, we can infer that method calls look like this behind the scenes:

In [6]:

Baz.bar(foo)

1

Methods vs Functions

So far so good. All this is pretty easy to grasp. But there's still burning question of how the heck is self (or rather foo) being passed to our methods. If we examine Baz.bar and foo.bar both, we can see there's a transformation going on somewhere.

In [7]:

print(Baz.bar)
print(foo.bar)

<function Baz.bar at 0x7f1d05a87ea0>
<bound method Baz.bar of <__main__.Baz object at 0x7f1d05a88208>>

Python is some how transforming our function that lives in Baz's dictionary into a method tied to our instance foo. The answer lies in the descriptor protocol. I've written about it else where, and it's probably time to revise it again with my recent understanding. But essentially, descriptors add another rule to our attribute look up. Just before the __getattr__ call: If we recieved a descriptor, call the __get__ method on the descriptor.

This is our missing link. When a function is declared in the class, not only is it placed in the class's dictionary it's also wrapped by a descriptor. Or more accurately, a non-data descriptor because it only defines the special __get__ method. The way descriptors work is by intercepting lookup of specific attributes.

The Descriptor likely has a passing resemblance to this (of course, implemented in C):

In [8]:

from types import MethodType

class MethodDescriptor:
    def __init__(self, method):
        self.method = method
    
    def __get__(self, instance, cls):
        if instance is None:
            return self.method
        return MethodType(self.method, instance)

So, our initial thought of what foo.bar() looks like under the covers was wrong. It more accurately resembles:

In [9]:

Baz.__dict__['bar'].__get__(foo, Baz)()
# if we inspect it we see the truth
print(Baz.__dict__['bar'].__get__(foo, Baz))

1
<bound method Baz.bar of <__main__.Baz object at 0x7f1d05a88208>>

And in fact, if we put our imitation method descriptor into action, it works similarly to how object methods do.

In [10]:

def monty(self, x):
    print(x)

class Spam:
    eggs = MethodDescriptor(monty)
    
    ##of course, it's also useable as a decorator
    @MethodDescriptor
    def bar(self):
        return 4
    
ham = Spam() # a lie if I ever saw one
print(Spam.eggs)
print(ham.eggs)
ham.eggs(1)
print(ham.bar())

<function monty at 0x7f1d045cef28>
<bound method Spam.monty of <__main__.Spam object at 0x7f1d05a780b8>>
1
4

The reason we see a function when we access the bar method when we access it through the class is because the descriptor has already run and decided that it should simply return the function itself.