30 Python Language Features and Tricks You May Not Know About

original article: http://sahandsaba.com/thirty-python-language-features-and-tricks-you-may-not-know.html#unpacking

30 Python Language Features and Tricks You May Not Know About

05 Mar 2014 By Sahand Saba

1   Introduction

Since I started learning Python, I decided to maintain an oftenvisited list of "tricks". Any time I saw a piece of code (in an example, onStack Overflow, in open source software, etc.) that made me think"Cool! I didn't know you could do that!" I experimented with it until Iunderstood it and then added it to the list. This post is part of that list,after some cleaning up. If you are an experienced Python programmer, chancesare you already know most of these, though you might still find a few that youdidn't know about. If you are a C, C++ or Java programmer who is learningPython, or just brand new to programming, then you might find quite a few ofthem surprisingly useful, like I did.

Each trick or language feature is demonstrated only through examples,with no explanation. While I tried my best to make the examples clear,some of them might still appear cryptic depending on your familiarity level. Soif something still doesn't make sense after looking at the examples, the titleshould be clear enough to allow you to use Google for more information on it.

The list is very roughly ordered by difficulty, with the easier and morecommonly known language features and tricks appearing first.

A table of contents is given at the end.

Update - March 8th, 2014

This article generated a lot of good discussion on Reddit(http://redd.it/1zv3q3), Hacker News(https://news.ycombinator.com/item?id=7365410), and in the comments below,with many readers suggesting great alternatives and improvements. I haveupdated the list below to include many of the improvements suggested, andadded a few new items based on suggestions that made mehave one of those "Cool! I didn't know you could do that!" moments. Inparticular, I did not know about itertools.chain.from_iterable, anddictionary comprehensions.

There was also a very interesting discussion about the possibility of someof the techniques below leading to harder to debug code. My say on it isthat as far as I can see, none of the items below are inherently harder todebug. But I can definitely see how they can be taken too far, resultingin hard to debug, maintain and understand code. Use your best judgment andif it feels like how short and smart your code is is outweighing howreadable and maintainable it is, then break it down and simplify it. Forexample, I think list comprehensions can be very readable and rather easydebug and maintain. But a list comprehension inside another listcomprehension that is then passed tomap and then toitertools.chain? Probably not the best idea!

1.1   Unpacking

>>> a, b, c = 1, 2, 3>>> a, b, c(1, 2, 3)>>> a, b, c = [1, 2, 3]>>> a, b, c(1, 2, 3)>>> a, b, c = (2 * i + 1 for i in range(3))>>> a, b, c(1, 3, 5)>>> a, (b, c), d = [1, (2, 3), 4]>>> a1>>> b2>>> c3>>> d4

1.2   Unpacking for swapping variables

>>> a, b = 1, 2>>> a, b = b, a>>> a, b(2, 1)

1.3   Extended unpacking (Python 3 only)

>>> a, *b, c = [1, 2, 3, 4, 5]>>> a1>>> b[2, 3, 4]>>> c5

1.4   Negative indexing

>>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]>>> a[-1]10>>> a[-3]8

1.5   List slices (a[start:end])

>>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]>>> a[2:8][2, 3, 4, 5, 6, 7]

1.6   List slices with negative indexing

>>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]>>> a[-4:-2][7, 8]

1.7   List slices with step (a[start:end:step])

>>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]>>> a[::2][0, 2, 4, 6, 8, 10]>>> a[::3][0, 3, 6, 9]>>> a[2:8:2][2, 4, 6]

1.8   List slices with negative step

>>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]>>> a[::-1][10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]>>> a[::-2][10, 8, 6, 4, 2, 0]

1.9   List slice assignment

>>> a = [1, 2, 3, 4, 5]>>> a[2:3] = [0, 0]>>> a[1, 2, 0, 0, 4, 5]>>> a[1:1] = [8, 9]>>> a[1, 8, 9, 2, 0, 0, 4, 5]>>> a[1:-1] = []>>> a[1, 5]

1.10   Naming slices (slice(start, end, step))

>>> a = [0, 1, 2, 3, 4, 5]>>> LASTTHREE = slice(-3, None)>>> LASTTHREEslice(-3, None, None)>>> a[LASTTHREE]
[3, 4, 5]

1.11   Zipping and unzipping lists and iterables

>>> a = [1, 2, 3]>>> b = ['a', 'b', 'c']>>> z = zip(a, b)>>> z[(1, 'a'), (2, 'b'), (3, 'c')]>>> zip(*z)[(1, 2, 3), ('a', 'b', 'c')]

1.12   Grouping adjacent list items using zip

>>> a = [1, 2, 3, 4, 5, 6]>>> zip(a[::2], a[1::2])[(1, 2), (3, 4), (5, 6)]>>> zip(a[::3], a[1::3], a[2::3])[(1, 2, 3), (4, 5, 6)]>>> group_adjacent = lambda a, k: zip(*(a[i::k] for i in range(k)))>>> group_adjacent(a, 3)[(1, 2, 3), (4, 5, 6)]>>> group_adjacent(a, 2)[(1, 2), (3, 4), (5, 6)]>>> group_adjacent(a, 1)[(1,), (2,), (3,), (4,), (5,), (6,)]

1.13   Inverting a dictionary using zip

>>> m = {'a': 1, 'b': 2, 'c': 3, 'd': 4}>>> m.items()[('a', 1), ('c', 3), ('b', 2), ('d', 4)]>>> zip(m.values(), m.keys())[(1, 'a'), (3, 'c'), (2, 'b'), (4, 'd')]>>> mi = dict(zip(m.values(), m.keys()))>>> mi{1: 'a', 2: 'b', 3: 'c', 4: 'd'}

1.14   Flattening lists:

>>> a = [[1, 2], [3, 4], [5, 6]]>>> b = [x for l in a for x in l]>>> b[1, 2, 3, 4, 5, 6]>>> a = [[[1, 2], [3, 4]], [[5, 6], [7, 8]]]>>> b = [x for l1 in a for l2 in l1 for x in l2]>>> b[1, 2, 3, 4, 5, 6, 7, 8]>>> a = [1, 2, [3, 4], [[5, 6], [7, 8]]]>>> flatten = lambda x: [y for l in x for y in flatten(l)] if type(x) is list else [x]>>> flatten(a)[1, 2, 3, 4, 5, 6, 7, 8]>>> a = [[1, 2], [3, 4], [5, 6]]>>> list(itertools.chain.from_iterable(a))[1, 2, 3, 4, 5, 6]

1.15   Generator expressions

>>> g = (x ** 2 for x in xrange(10))>>> next(g)0>>> next(g)1>>> next(g)4>>> next(g)9>>> sum(x ** 3 for x in xrange(10))2025>>> sum(x ** 3 for x in xrange(10) if x % 3 == 1)408

1.16   Dictionary comprehensions

>>> m = {x: x ** 2 for x in range(5)}>>> m{0: 0, 1: 1, 2: 4, 3: 9, 4: 16}>>> m = {x: 'A' + str(x) for x in range(10)}>>> m{0: 'A0', 1: 'A1', 2: 'A2', 3: 'A3', 4: 'A4', 5: 'A5', 6: 'A6', 7: 'A7', 8: 'A8', 9: 'A9'}

1.17   Invertng a dictionary using a dictionary comprehension

>>> m = {'a': 1, 'b': 2, 'c': 3, 'd': 4}>>> m{'d': 4, 'a': 1, 'b': 2, 'c': 3}>>> {v: k for k, v in m.items()}{1: 'a', 2: 'b', 3: 'c', 4: 'd'}

1.18   Named tuples (collections.namedtuple)

>>> Point = collections.namedtuple('Point', ['x', 'y'])>>> p = Point(x=1.0, y=2.0)>>> pPoint(x=1.0, y=2.0)>>> p.x1.0>>> p.y2.0

1.19   Inheriting from named tuples:

>>> class Point(collections.namedtuple('PointBase', ['x', 'y'])):...     __slots__ = ()...     def __add__(self, other):...             return Point(x=self.x + other.x, y=self.y + other.y)...>>> p = Point(x=1.0, y=2.0)>>> q = Point(x=2.0, y=3.0)>>> p + qPoint(x=3.0, y=5.0)

1.20   Sets and set operations

>>> A = {1, 2, 3, 3}>>> Aset([1, 2, 3])>>> B = {3, 4, 5, 6, 7}>>> Bset([3, 4, 5, 6, 7])>>> A | Bset([1, 2, 3, 4, 5, 6, 7])>>> A & Bset([3])>>> A - Bset([1, 2])>>> B - Aset([4, 5, 6, 7])>>> A ^ Bset([1, 2, 4, 5, 6, 7])>>> (A ^ B) == ((A - B) | (B - A))True

1.21   Multisets and multiset operations (collections.Counter)

>>> A = collections.Counter([1, 2, 2])>>> B = collections.Counter([2, 2, 3])>>> ACounter({2: 2, 1: 1})>>> BCounter({2: 2, 3: 1})>>> A | BCounter({2: 2, 1: 1, 3: 1})>>> A & BCounter({2: 2})>>> A + BCounter({2: 4, 1: 1, 3: 1})>>> A - BCounter({1: 1})>>> B - ACounter({3: 1})

1.22   Most common elements in an iterable (collections.Counter)

>>> A = collections.Counter([1, 1, 2, 2, 3, 3, 3, 3, 4, 5, 6, 7])>>> ACounter({3: 4, 1: 2, 2: 2, 4: 1, 5: 1, 6: 1, 7: 1})>>> A.most_common(1)[(3, 4)]>>> A.most_common(3)[(3, 4), (1, 2), (2, 2)]

1.23   Double-ended queue (collections.deque)

>>> Q = collections.deque()>>> Q.append(1)>>> Q.appendleft(2)>>> Q.extend([3, 4])>>> Q.extendleft([5, 6])>>> Qdeque([6, 5, 2, 1, 3, 4])>>> Q.pop()4>>> Q.popleft()6>>> Qdeque([5, 2, 1, 3])>>> Q.rotate(3)>>> Qdeque([2, 1, 3, 5])>>> Q.rotate(-3)>>> Qdeque([5, 2, 1, 3])

1.24   Double-ended queue with maximum length (collections.deque)

>>> last_three = collections.deque(maxlen=3)>>> for i in xrange(10):...     last_three.append(i)...     print ', '.join(str(x) for x in last_three)...00, 10, 1, 21, 2, 32, 3, 43, 4, 54, 5, 65, 6, 76, 7, 87, 8, 9

1.25   Ordered dictionaries (collections.OrderedDict)

>>> m = dict((str(x), x) for x in range(10))>>> print ', '.join(m.keys())1, 0, 3, 2, 5, 4, 7, 6, 9, 8>>> m = collections.OrderedDict((str(x), x) for x in range(10))>>> print ', '.join(m.keys())0, 1, 2, 3, 4, 5, 6, 7, 8, 9>>> m = collections.OrderedDict((str(x), x) for x in range(10, 0, -1))>>> print ', '.join(m.keys())10, 9, 8, 7, 6, 5, 4, 3, 2, 1

1.26   Default dictionaries (collections.defaultdict)

>>> m = dict()>>> m['a']Traceback (most recent call last):  File "<stdin>", line 1, in <module>KeyError: 'a'>>>>>> m = collections.defaultdict(int)>>> m['a']0>>> m['b']0>>> m = collections.defaultdict(str)>>> m['a']''>>> m['b'] += 'a'>>> m['b']'a'>>> m = collections.defaultdict(lambda: '[default value]')>>> m['a']'[default value]'>>> m['b']'[default value]'>>>

1.27   Using default dictionaries to represent simple trees

>>> import json>>> tree = lambda: collections.defaultdict(tree)>>> root = tree()>>> root['menu']['id'] = 'file'>>> root['menu']['value'] = 'File'>>> root['menu']['menuitems']['new']['value'] = 'New'>>> root['menu']['menuitems']['new']['onclick'] = 'new();'>>> root['menu']['menuitems']['open']['value'] = 'Open'>>> root['menu']['menuitems']['open']['onclick'] = 'open();'>>> root['menu']['menuitems']['close']['value'] = 'Close'>>> root['menu']['menuitems']['close']['onclick'] = 'close();'>>> print json.dumps(root, sort_keys=True, indent=4, separators=(',', ': ')){    "menu": {        "id": "file",        "menuitems": {            "close": {                "onclick": "close();",                "value": "Close"            },            "new": {                "onclick": "new();",                "value": "New"            },            "open": {                "onclick": "open();",                "value": "Open"            }        },        "value": "File"    }}

(See https://gist.github.com/hrldcpr/2012250 for more on this.)

1.28   Mapping objects to unique counting numbers (collections.defaultdict)

>>> import itertools, collections>>> value_to_numeric_map = collections.defaultdict(itertools.count().next)>>> value_to_numeric_map['a']0>>> value_to_numeric_map['b']1>>> value_to_numeric_map['c']2>>> value_to_numeric_map['a']0>>> value_to_numeric_map['b']1

1.29   Largest and smallest elements (heapq.nlargest andheapq.nsmallest)

>>> a = [random.randint(0, 100) for __ in xrange(100)]>>> heapq.nsmallest(5, a)[3, 3, 5, 6, 8]>>> heapq.nlargest(5, a)[100, 100, 99, 98, 98]

1.30   Cartesian products (itertools.product)

>>> for p in itertools.product([1, 2, 3], [4, 5]):(1, 4)(1, 5)(2, 4)(2, 5)(3, 4)(3, 5)>>> for p in itertools.product(*([[0, 1]] * 4)):...     print ''.join(str(x) for x in p)...0000000100100011010001010110011110001001101010111100110111101111

1.31   Combinations and combinations with replacement (itertools.combinations anditertools.combinations_with_replacement)

>>> for c in itertools.combinations([1, 2, 3, 4, 5], 3):...     print ''.join(str(x) for x in c)...123124125134135145234235245345>>> for c in itertools.combinations_with_replacement([1, 2, 3], 2):...     print ''.join(str(x) for x in c)...111213222333

1.32   Permutations (itertools.permutations)

>>> for p in itertools.permutations([1, 2, 3, 4]):...     print ''.join(str(x) for x in p)...123412431324134214231432213421432314234124132431312431423214324134123421412341324213423143124321

1.33   Chaining iterables (itertools.chain)>>> a = [1, 2, 3, 4]>>> for p in itertools.chain(itertools.combinations(a, 2), itertools.combinations(a, 3)):...     print p...(1, 2)(1, 3)(1, 4)(2, 3)(2, 4)(3, 4)(1, 2, 3)(1, 2, 4)(1, 3, 4)(2, 3, 4)>>> for subset in itertools.chain.from_iterable(itertools.combinations(a, n) for n in range(len(a) + 1))...     print subset...()(1,)(2,)(3,)(4,)(1, 2)(1, 3)(1, 4)(2, 3)(2, 4)(3, 4)(1, 2, 3)(1, 2, 4)(1, 3, 4)(2, 3, 4)(1, 2, 3, 4)

1.34   Grouping rows by a given key (itertools.groupby)

>>> import itertools>>> with open('contactlenses.csv', 'r') as infile:...     data = [line.strip().split(',') for line in infile.readlines()]...>>> data = data[1:]>>> def print_data(rows):...     print '\n'.join('\t'.join('{: <16}'.format(s) for s in row) for row in rows)...>>> print_data(data)young               myope                   no                      reduced                 noneyoung               myope                   no                      normal                  softyoung               myope                   yes                     reduced                 noneyoung               myope                   yes                     normal                  hardyoung               hypermetrope            no                      reduced                 noneyoung               hypermetrope            no                      normal                  softyoung               hypermetrope            yes                     reduced                 noneyoung               hypermetrope            yes                     normal                  hardpre-presbyopic      myope                   no                      reduced                 nonepre-presbyopic      myope                   no                      normal                  softpre-presbyopic      myope                   yes                     reduced                 nonepre-presbyopic      myope                   yes                     normal                  hardpre-presbyopic      hypermetrope            no                      reduced                 nonepre-presbyopic      hypermetrope            no                      normal                  softpre-presbyopic      hypermetrope            yes                     reduced                 nonepre-presbyopic      hypermetrope            yes                     normal                  nonepresbyopic          myope                   no                      reduced                 nonepresbyopic          myope                   no                      normal                  nonepresbyopic          myope                   yes                     reduced                 nonepresbyopic          myope                   yes                     normal                  hardpresbyopic          hypermetrope            no                      reduced                 nonepresbyopic          hypermetrope            no                      normal                  softpresbyopic          hypermetrope            yes                     reduced                 nonepresbyopic          hypermetrope            yes                     normal                  none>>> data.sort(key=lambda r: r[-1])>>> for value, group in itertools.groupby(data, lambda r: r[-1]):...     print '-----------'...     print 'Group: ' + value...     print_data(group)...-----------Group: hardyoung               myope                   yes                     normal                  hardyoung               hypermetrope            yes                     normal                  hardpre-presbyopic      myope                   yes                     normal                  hardpresbyopic          myope                   yes                     normal                  hard-----------Group: noneyoung               myope                   no                      reduced                 noneyoung               myope                   yes                     reduced                 noneyoung               hypermetrope            no                      reduced                 noneyoung               hypermetrope            yes                     reduced                 nonepre-presbyopic      myope                   no                      reduced                 nonepre-presbyopic      myope                   yes                     reduced                 nonepre-presbyopic      hypermetrope            no                      reduced                 nonepre-presbyopic      hypermetrope            yes                     reduced                 nonepre-presbyopic      hypermetrope            yes                     normal                  nonepresbyopic          myope                   no                      reduced                 nonepresbyopic          myope                   no                      normal                  nonepresbyopic          myope                   yes                     reduced                 nonepresbyopic          hypermetrope            no                      reduced                 nonepresbyopic          hypermetrope            yes                     reduced                 nonepresbyopic          hypermetrope            yes                     normal                  none-----------Group: softyoung               myope                   no                      normal                  softyoung               hypermetrope            no                      normal                  softpre-presbyopic      myope                   no                      normal                  softpre-presbyopic      hypermetrope            no                      normal                  softpresbyopic          hypermetrope            no                      normal                  soft