def generator_example1():
count = 0
while True:
yield count
count+=1
g = generator_example1()
next(g)
next(g)
next(g)
and so on...
Ok let us revisit our function 'generator_example1()'. What is happening in the below code?
Inside while loop, we have 'yield' statement. Yield breakes out of loop and gives back control to whomever called function generator_exampe1(). In statement 'g = generator_example1()', g is now a geneator as shown below.
def generator_example1():
count = 0
while True:
yield count
count+=1
g = generator_example1()
g
Once you have a generator function, you can iterate through it using next() function. Since we have a infinite 'while' loop in the genereator_example() function, we can call iterator as many times as we want it. Each time, we use next(), generator starts the execution from previous position and prints a new value.
Python generators can be used outside the function without the 'yield'. Check out the below example.
g = (x for x in range(10))
g
(x for x in range(10)) is a Python generator object. The syntax is quite similar to Python list comprehension except that instead of square brackets, generators are defined using round brackets. As usual, once we have generator object, we can call iterator next() on it to print the values as shown below.
next(g)
next(g)
Python generators will throw 'StopIteration' exception, if there is no value to return for the iterator.
Let us look at following example.
def range_one():
for x in range(0,1):
yield x
g = range_one()
next(g)
next(g)
To avoid above error, we can catch exception like this and stop the iteration.
g = range_one()
try:
print(next(g))
except StopIteration:
print('Iteration Stopped')
try:
print(next(g))
except StopIteration:
print('Iteration Stopped')
We can pass value to Python Generators using send() function.
def incrment_no():
while True:
x = yield
yield x + 1
g = incrment_no() # Create our generator
next(g) # It will go to first yield
print(g.send(7)) # value 7 is sent to generator which gets assgined to x, 2nd yield statement gets executed
Python generators can be used recursively. Check out the below code. In below function, "yield from generator_factorial(n - 1)" is recursive call to function generator_factorial().
def generator_factorial(n):
if n == 1:
f = 1
else:
a = yield from generator_factorial(n - 1)
f = n * a
yield f
return f
g = generator_factorial(3)
next(g)
next(g)
next(g)
Continuing with above example, let us say we want generator to throw error for the factorial of number greater than 100. We can add generator.throw() exception such as shown below.
n = 100
if n >= 100:
g.throw(ValueError, 'Only numbers less than 100 are allowed')
Python generators take very less memory. Let us look at following two examples. In the examples below, note the difference between byte size of memory used by 'Python list' vs 'Python generator'.
import sys
#Python List comprehension
sequence = [x for x in range(1,1000000)]
sys.getsizeof(sequence)
#Python Generators
sequence = (x for x in range(1,1000000))
sys.getsizeof(sequence)
#Python List comprehension
import cProfile
cProfile.run('sum([x for x in range(1,10000000)])')
#generators
import cProfile
cProfile.run('sum((x for x in range(1,10000000)))')
Check the number of function calls and time the 'Python generator' took to compute the sum compare to Python 'list comprehension'.
Let us wrap up this tutorial with Data Pipelines. Python generators are great for building the pipelines.
Let us open a CSV file and iterate through it using Python generator.
def generator_read_csv_file():
for entry in open('stock.csv'):
yield entry
g = generator_read_csv_file()
next(g)
next(g)
Let us say, we want to replace the commas in the CSV for each line with spaces, we can build a pipeline for this.
g1 = (entry for entry in open('stock.csv'))
g2 = (row.replace(","," ") for row in g1)
next(g2)
next(g2)
next(g2)
Related Notebooks
- Python Iterators And Generators
- How to do SQL Select and Where Using Python Pandas
- Summarising Aggregating and Grouping data in Python Pandas
- Merge and Join DataFrames with Pandas in Python
- How to Convert Python Pandas DataFrame into a List
- How to Plot a Histogram in Python
- How to Generate Random Numbers in Python
- Python Pandas String To Integer And Integer To String DataFrame
- How to Upgrade Python PIP