Python: collections, part 4/4: All about expression-generators, list generators, sets and dictionaries

Table of contents:

1. Definitions and classification

1.1 What and why

• Expression generators are designed for a compact and convenient way of generating collections of elements, as well as converting one type of collection to another.
• In the process of generation or transformation, it is possible to apply conditions and modify elements.
• Expression generators are syntactic sugar and do not solve problems that could not be solved without their use.

1.2 Benefits of Using Expression Generators

• Shorter and more convenient syntax than generation in a normal loop.
• A more understandable and readable syntax than a functional analogue that combines the simultaneous use of the map (), filter () and lambda functions.
• In general: it is faster to type, easier to read, especially when there are many similar operations in the code.

1.3 Classification and features

• generator expression — an expression in parentheses that produces a new element at each iteration according to the rules.
• collection generator — a generic name for the list comprehension generator, dictionary comprehension generator, and set comprehension generator.

2. Syntax

General principles important to understand:

• Input is an iterator — it can be a generator function, a generator expression, a collection — any object that supports iteration through it.
• A condition is a filter under which the element will go to the final expression, if the element does not satisfy it, it will be skipped.
• The final expression is the transformation of each selected element before its output, or simply the output without changes.

2.1 Basic syntax

list_a = [-2, -1, 0, 1, 2, 3, 4, 5] #       list_b = [x for x in list_a] #       print(list_b) # [-2, -1, 0, 1, 2, 3, 4, 5] print(list_a is list_b) # False -   ! 

2.2 Add filtering condition

 # if x % 2 == 0 -     2   -   list_a = [-2, -1, 0, 1, 2, 3, 4, 5] list_b = [x for x in list_a if x % 2 == 0] print(list_b) # [-2, 0, 2, 4] 

 list_a = [-2, -1, 0, 1, 2, 3, 4, 5] list_b = [x for x in list_a if x % 2 == 0 and x > 0] #   x,       print(list_b) # [2, 4] 

2.3 Add item processing in expression

 list_a = [-2, -1, 0, 1, 2, 3, 4, 5] list_b = [x**2 for x in list_a] print(list_b) # [4, 1, 0, 1, 4, 9, 16, 25] 

 list_a = ['a', 'abc', 'abcde'] list_b = [len(x) for x in list_a] print(list_b) # [1, 3, 5] 

2.4 Expression branching

• Branching conditions are written not after, but before the iterator.
• In this case, the if-else is not a filter before the expression is executed, but a branch of the expression itself, that is, the variable has already passed the filter, but depending on the condition it can be processed differently!

 list_a = [-2, -1, 0, 1, 2, 3, 4, 5] list_b = [x if x < 0 else x**2 for x in list_a] #  x- -  x,    -   x print(list_b) # [-2, -1, 0, 1, 4, 9, 16, 25] 

 list_a = [-2, -1, 0, 1, 2, 3, 4, 5] list_b = [x**3 if x < 0 else x**2 for x in list_a if x % 2 == 0] #         #          ,      print(list_b) # [-8, 0, 4, 16] 

 list_a = [-2, -1, 0, 1, 2, 3, 4, 5] list_b = [] for x in list_a: if x % 2 == 0: if x < 0: list_b.append(x ** 3) else: list_b.append(x ** 2) print(list_b) # [-8, 0, 4, 16] 

2.5 Improving readability

 numbers = range(10) # Before squared_evens = [n ** 2 for n in numbers if n % 2 == 0] # After squared_evens = [ n ** 2 for n in numbers if n % 2 == 0 ] 

3. Analogs in the form of a for loop and in the form of functions

3.1 Solution using a list generator

 numbers = range(10) squared_evens = [n ** 2 for n in numbers if n % 2 == 0] print(squared_evens) # [0, 4, 16, 36, 64] 

3.2. Solution using a for loop

 numbers = range(10) squared_evens = [] for n in numbers: if n % 2 == 0: squared_evens.append(n ** 2) print(squared_evens) # [0, 4, 16, 36, 64] 

3.3. Solution using functions.

 numbers = range(10) squared_evens = map(lambda n: n ** 2, filter(lambda n: n % 2 == 0, numbers)) print(squared_evens) # <map object at 0x7f661e5dba20> print(list(squared_evens)) # [0, 4, 16, 36, 64] # :  Python 2   squared_evens   ,   Python 3 «map object»,        list() 

4. Generator expressions

 list_a = [-2, -1, 0, 1, 2, 3, 4, 5] my_gen = (i for i in list_a) # - print(next(my_gen)) # -2 -     print(next(my_gen)) # -1 -     

Features of Generator Expressions

1. Generator cannot be written without brackets - this is a syntax error.
   

    # my_gen = i for i in list_a # SyntaxError: invalid syntax 

   
   
2. When passing to the function, additional brackets are optional.
   

    list_a = [-2, -1, 0, 1, 2, 3, 4, 5] my_sum = sum(i for i in list_a) # my_sum = sum((i for i in list_a)) #    print(my_sum) # 12 

   
   
3. Cannot get len () length
   

    # my_len = len(i for i in list_a) # TypeError: object of type 'generator' has no len() 

   
   
4. Cannot print items with the print () function.
   

    print(my_gen) # <generator object <genexpr> at 0x7f162db32af0> 

   
   
5. Please note that after passing through the expression-generator, it remains empty !
   

    list_a = [-2, -1, 0, 1, 2, 3, 4, 5] my_gen = (i for i in list_a) print(sum(my_gen)) # 12 print(sum(my_gen)) # 0 

   
   
6. Generator expression can be infinite .
   

    import itertools inf_gen = (x for x in itertools.count()) #    0 to ! 

   Be careful in working with such generators, as if not properly used, the “effect” will be like from an infinite loop.
   
   
7. The expression-generator does not apply cuts !
   

    list_a = [-2, -1, 0, 1, 2, 3, 4, 5] my_gen = (i for i in list_a) my_gen_sliced = my_gen[1:3] # TypeError: 'generator' object is not subscriptable 

   
   
8. From the generator it is easy to get the desired collection . This is discussed in detail in the next chapter.

5. Generation of standard collections

5.1 Creating Collections from a Generator Expression

1. By transferring the finished generator expression assigned to a variable to the collection creation function.
   
   

    list_a = [-2, -1, 0, 1, 2, 3, 4, 5] my_gen = (i for i in list_a) # - my_list = list(my_gen) print(my_list) # [-2, -1, 0, 1, 2, 3, 4, 5] 

   
   
2. Writing a generator expression right inside the brackets of the called collection creation function.
   
   

    list_a = [-2, -1, 0, 1, 2, 3, 4, 5] my_list = list(i for i in list_a) print(my_list) # [-2, -1, 0, 1, 2, 3, 4, 5] 

   
   
   The same for a tuple, a set and an unchanged set

    #  my_tuple = tuple(i for i in list_a) print(my_tuple) # (-2, -1, 0, 1, 2, 3, 4, 5) #  my_set = set(i for i in list_a) print(my_set) # {0, 1, 2, 3, 4, 5, -1, -2} #   my_frozenset = frozenset(i for i in list_a) print(my_frozenset) # frozenset({0, 1, 2, 3, 4, 5, -1, -2}) 

5.2 Special syntax of collection generators

1. List comprehension
   
   

    list_a = [-2, -1, 0, 1, 2, 3, 4, 5] my_list = [i for i in list_a] print(my_list) # [-2, -1, 0, 1, 2, 3, 4, 5] 

   
   Do not write parentheses in square!
   
   

    list_a = [-2, -1, 0, 1, 2, 3, 4, 5] my_list = [(i for i in list_a)] print(my_list) # [<generator object <genexpr> at 0x7fb81103bf68>] 

   
   
2. Set comprehension
   
   

    list_a = [-2, -1, 0, 1, 2, 3, 4, 5] my_set= {i for i in list_a} print(my_set) # {0, 1, 2, 3, 4, 5, -1, -2} -   

   
   
3. Dictionary comprehension
   turning over the dictionary
   
   

    dict_abc = {'a': 1, 'b': 2, 'c': 3, 'd': 3} dict_123 = {v: k for k, v in dict_abc.items()} print(dict_123) # {1: 'a', 2: 'b', 3: 'd'} #  ,   ""!     , #     ,    . 

   
   Dictionary from the list:
   

    list_a = [-2, -1, 0, 1, 2, 3, 4, 5] dict_a = {x: x**2 for x in list_a} print(dict_a) # {0: 0, 1: 1, 2: 4, 3: 9, 4: 16, -2: 4, -1: 1, 5: 25} 

   
   It is important ! Such syntax for creating a dictionary works only in curly brackets, so it is impossible to create a generator expression , for this a slightly different syntax is used (thanks to longclaps for a hint in the comments):
   

    # dict_gen = (x: x**2 for x in list_a) # SyntaxError: invalid syntax dict_gen = ((x, x ** 2) for x in list_a) #   -   # dict_a = dict(x: x**2 for x in list_a) # SyntaxError: invalid syntax dict_a = dict((x, x ** 2) for x in list_a) #     @longclaps 

   
   

5.3 String generation

 list_a = [-2, -1, 0, 1, 2, 3, 4, 5] #     .join()       my_str = ''.join(str(x) for x in list_a) print(my_str) # -2-1012345 

6. Frequency and partial search

6.1 Working with enumerate ()

 for i, x in enumerate(iterable) 

 list_a = [-2, -1, 0, 1, 2, 3, 4, 5] list_d = [(i, x) for i, x in enumerate(list_a)] print(list_d) # [(0, -2), (1, -1), (2, 0), (3, 1), (4, 2), (5, 3), (6, 4), (7, 5)] 

 list_a = [-2, -1, 0, 1, 2, 3, 4, 5] list_e = [x for i, x in enumerate(list_a, 1) if i % 3 == 0] print(list_e) # [0, 3] 

Important features of the function enumerate ():

1. There are two options for calling the enumerate () function:
   • enumerate (iterator) without the second parameter counts from 0.
   • enumerate (iterator, start) - starts counting from the start value. Conveniently, for example, if we need to count from 1, not 0.
   
   
2. enumerate () returns a tuple of the sequence number and value of the current iterator element. The tuple in the expression-generator result can be obtained in two ways:
   • (i, j) for i, j in enumerate (iterator) - brackets in the first pair are needed!
   • pair for pair in enumerate (mylist) - we work immediately with a pair
   
   
3. Indices are calculated for all processed items , without taking into account they have passed the condition in the future or not!
   
   

    first_ten_even = [(i, x) for i, x in enumerate(range(10)) if x % 2 == 0] print(first_ten_even) # [(0, 0), (2, 2), (4, 4), (6, 6), (8, 8)] 

   
   
4. The enumerate () function does not access any internal attributes of the collection, but simply implements a counter of processed elements , so nothing prevents it from being used for unordered collections that are not indexed.
   
   
5. If we limit the number of elements included in the result by the enumerate () counter (for example, if i <10), then the iterator will still be processed entirely , which in the case of a huge collection will be very resource-consuming. The solution to this problem is discussed below in the sub-section “Looping a Part of the Iterated”.

6.2 Enumeration of the part to be iterated.

 import itertools first_ten = (itertools.islice((x for x in range(1000000000) if x % 2 == 0), 10)) print(list(first_ten)) # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] 

 import time import itertools #       start_time = time.time() first_ten = (itertools.islice((x for x in range(100) if x % 2 == 0), 10)) print(list(first_ten)) # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] elapsed_time = time.time() - start_time print(elapsed_time) # 3.409385681152344e-05 #       start_time = time.time() first_ten = (itertools.islice((x for x in range(100000000) if x % 2 == 0), 10)) print(list(first_ten)) # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] elapsed_time = time.time() - start_time print(elapsed_time) # 1.1205673217773438e-05 #        range()    6 , #        

7. Nested loops and generators

7.1 Nested loops

7.1.1 Nested loops for where loops are iterated through independent iterators

 rows = 1, 2, 3 cols = 'a', 'b' my_dict = {(col, row): 0 for row in rows for col in cols} print(my_dict) # {('a', 1): 0, ('b', 2): 0, ('b', 3): 0, ('b', 1): 0, ('a', 3): 0, ('a', 2): 0} 

 my_dict['b', 2] = 10 #     - print(my_dict['b', 2]) # 10 -     - 

 rows = 1, 2, 3, -4, -5 cols = 'a', 'b', 'abc' #       my_dict = { (col, row): 0 #     -    for row in rows if row > 0 #    for col in cols if len(col) == 1 #   } print(my_dict) # {('a', 1): 0, ('b', 2): 0, ('b', 3): 0, ('b', 1): 0, ('a', 3): 0, ('a', 2): 0} 

 rows = 1, 2, 3, -4, -5 cols = 'a', 'b', 'abc' my_dict = {} for row in rows: if row > 0: for col in cols: if len(col) == 1: my_dict[col, row] = 0 print(my_dict) # {('a', 1): 0, ('b', 2): 0, ('b', 3): 0, ('b', 1): 0, ('a', 3): 0, ('a', 2): 0} 

7.1.2 Nested loops for where the inner loop follows the result of the outer loop

 matrix = [[0, 1, 2, 3], [10, 11, 12, 13], [20, 21, 22, 23]] #     : flattened = [n for row in matrix for n in row] print(flattened) # [0, 1, 2, 3, 10, 11, 12, 13, 20, 21, 22, 23] 

 flattened = [] for row in matrix: for n in row: flattened.append(n) print(flattened) 

 import itertools flattened = list(itertools.chain.from_iterable(matrix)) #  @iMrDron #      #       . flattened = sum(a, []) #  @YuriM1983 # sum(a, [])   (O(n^2)) #           

7.2 Embedded Generators

7.2.1 - Nested generator inside the generator - two-dimensional of two one-dimensional

 w, h = 5, 3 #      matrix = [[0 for x in range(w)] for y in range(h)] print(matrix) # [[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]] 

 matrix = [] for y in range(h): new_row = [] for x in range(w): new_row.append(0) matrix.append(new_row) print(matrix) # [[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]] 

 #       ( -    ) matrix[0][0] = 1 matrix[1][3] = 3 print(matrix) # [[1, 0, 0, 0, 0], [0, 0, 0, 3, 0], [0, 0, 0, 0, 0]] #      x, y = 1, 3 print(matrix[x][y]) # 3 

7.2.2 - Embedded generator inside the generator - two-dimensional of the two-dimensional

 matrix = [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]] 

 squared = [[cell**2 for cell in row] for row in matrix] print(squared) # [[1, 4, 9, 16], [25, 36, 49, 64], [81, 100, 121, 144]] 

 squared = [] for row in matrix: new_row = [] for cell in row: new_row.append(cell**2) squared.append(new_row) print(squared) # [[1, 4, 9, 16], [25, 36, 49, 64], [81, 100, 121, 144]] 

7.3 - Generator iterating over generator

 list_a = [x for x in range(-2, 4)] #      , #       range(-2, 4) list_b = [x**2 for x in list_a] 

 list_c = [x**2 for x in [x for x in range(-2, 4)]] print(list_c) # [4, 1, 0, 1, 4, 9] 

 list_c = [t + t ** 2 for t in (x ** 3 + x ** 4 for x in range(-2, 4))] 

8. Using range ()

Features of the range () function:

• Most often, the range () function is used to start the for loop for the desired number of times. For example, see matrix generation in the examples above.
  
  
• In Python 3, range () returns a generator that, each time it is accessed, returns the next element.
  
  
• The parameters used are similar to those in sections (except for the first example with one parameter):
  
  
  • range (stop) - in this case from 0 to stop-1;
  • range (start, stop) - Similar to the example above, but you can set a nonzero start, you can and negative;
  • range (start, stop, step) - Add a step parameter, which can be negative, then iterate in reverse order.
  
  
• In Python 2 had two functions:
  
  
  • range (...) which is similar to the list (range (...)) expression in Python 3 - that is, it did not produce an iterator, but a list at once. That is, all the problems of possible lack of memory described in section 4 are relevant, and it must be used very carefully in Python 2!
  • xrange (...) - which worked similarly to range (...) in Python 3 and from version 3 was excluded.

 print(list(range(5))) # [0, 1, 2, 3, 4] print(list(range(-2, 5))) # [-2, -1, 0, 1, 2, 3, 4] print(list(range(5, -2, -2))) # [5, 3, 1, -1] 

9. Appendix 1. Additional examples

9.1 Sequential walk through multiple lists

 import itertools l1 = [1,2,3] l2 = [10,20,30] result = [l*2 for l in itertools.chain(l1, l2)] print(result) # [2, 4, 6, 20, 40, 60] 

9.2 Matrix Transposition

 matrix = [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]] 

 transposed = [[row[i] for row in matrix] for i in range(len(matrix[0]))] print(transposed) # [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] 

 transposed = [] for i in range(len(matrix[0])): new_row = [] for row in matrix: new_row.append(row[i]) transposed.append(new_row) print(transposed) # [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] 

 transposed = list(map(list, zip(*matrix))) print(transposed) # [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] 

9.3 The task of choosing only working days

 #     1  31     days = [d for d in range(1, 32)] #      weeks = [days[i:i+7] for i in range(0, len(days), 7)] print(weeks) # [[1, 2, 3, 4, 5, 6, 7], [8, 9, 10, 11, 12, 13, 14], [15, 16, 17, 18, 19, 20, 21], [22, 23, 24, 25, 26, 27, 28], [29, 30, 31]] #       5  ,   work_weeks = [week[0:5] for week in weeks] print(work_weeks) # [[1, 2, 3, 4, 5], [8, 9, 10, 11, 12], [15, 16, 17, 18, 19], [22, 23, 24, 25, 26], [29, 30, 31]] #      -   wdays = [item for sublist in work_weeks for item in sublist] print(wdays) # [1, 2, 3, 4, 5, 8, 9, 10, 11, 12, 15, 16, 17, 18, 19, 22, 23, 24, 25, 26, 29, 30, 31] 

 #     1  31     days = [d for d in range(1, 32)] wdays6 = [wd for (i, wd) in enumerate(days, 1) if i % 7 != 0] #   7-  #   6      : wdays5 = [wd for (i, wd) in enumerate(wdays6, 1) if i % 6 != 0] print(wdays5) # [1, 2, 3, 4, 5, 8, 9, 10, 11, 12, 15, 16, 17, 18, 19, 22, 23, 24, 25, 26, 29, 30, 31] #  ,        if  , #   ,  12-    6,      2  ! #     @sophist: days = [d + 1 for d in range(31) if d % 7 < 5] 

10. Appendix 2. Related links

1. Good English article with a detailed explanation of what generators and iterators are
   
   
   Illustration from the article:

   
   
   
   
2. If you have difficulty understanding the logic of working with generator expressions, take a look at an interesting English-language article where the analogies between generator expressions and working with SQL and Excel tables are drawn .
   
   
   For example:

    squared_evens = [n ** 2 # SELECT for n in numbers # FROM if n % 2 == 0] # WHERE 

   
   
3. UPD from fireSparrow : There is an extension Python - PythonQL, which allows you to work with databases in the style of collection generators.
   
   
4. , .
   
   
5. ( ).
   

:

• - - — , .
• - — — , .