26.1 Introduction to Generators

A generator is a special type of iterator that produces values one at a time. Instead of creating and storing every result immediately, a generator waits until the program requests the next value. This makes generators useful for large collections, long sequences, files, and data streams.

Generators remember where they stopped. When the program requests another value, execution continues from the previous stopping point. This pause-and-resume behavior allows generators to perform work gradually instead of completing everything at once.

Example

def simple_generator():
yield 10
yield 20
yield 30


# Calling the function creates a generator object

numbers = simple_generator()

print(next(numbers))
print(next(numbers))
print(next(numbers))

Output

10


20
30

Output Explanation

Each call to next() runs the generator until it reaches the next yield statement. The generator returns that value and pauses. The following call continues from the place where execution stopped.

26.2 Generator Functions

A generator function looks similar to a normal function, but it contains at least one yield statement. Calling a normal function usually runs its code immediately and returns one final result. Calling a generator function creates a generator object without running the complete function.

The generator begins running only when a value is requested with next() or when it is used in a loop. This makes generator functions useful when values should be produced gradually or when producing every result at once would use too much memory.

Example

def count_to_five():
number = 1

while number <= 5:
    yield number
    number += 1
```

counter = count_to_five()

for value in counter:
print(value)

Output

1
```

2
3
4
5

Output Explanation

The generator starts with the number 1. Each loop iteration requests the next value. After yielding a number, the generator pauses. When resumed, it increases the number and continues until the condition becomes false.

26.3 The yield Keyword

The yield keyword returns a value from a generator without permanently ending the function. It pauses the generator and saves its local variables, current position, and execution state. When another value is requested, execution continues immediately after the previous yield.

This is different from return. A return statement ends the function completely. A yield statement temporarily pauses the function so it can continue later. A generator may yield many values during its lifetime.

Example

def lesson_steps():
print("Preparing step 1")
yield "Step 1"

print("Preparing step 2")
yield "Step 2"

print("Preparing step 3")
yield "Step 3"
```

steps = lesson_steps()

print(next(steps))
print(next(steps))
print(next(steps))

Output

Preparing step 1
```

Step 1
Preparing step 2
Step 2
Preparing step 3
Step 3

Output Explanation

The generator runs only far enough to reach the next yield. It prints the preparation message, yields the step, and pauses. The next call resumes execution after the previous yield statement.

26.4 Generator Expressions

A generator expression is a short way to create a generator. It looks similar to a list comprehension, but it uses parentheses instead of square brackets. A list comprehension creates all its values immediately, while a generator expression produces values only when requested.

Generator expressions are helpful for simple transformations and filtering operations. They are often used with functions such as sum(), max(), min(), and loops because those tools can process generator values one at a time.

Example

# Create a generator expression
```

squares = (number ** 2 for number in range(1, 6))

print(next(squares))
print(next(squares))

# Process the remaining values

for square in squares:
print(square)

Output

1
```

4
9
16
25

Output Explanation

The generator expression calculates each square only when requested. The first two values are retrieved with next(). The loop then continues from the generator's current position and processes the remaining squares.

26.5 Generators vs Lists

Lists and generators can both represent sequences, but they behave differently. A list stores all its values in memory and can be reused many times. A generator normally produces values one at a time and is usually exhausted after one complete pass.

Lists support indexing, slicing, and methods such as append(). Generators do not support direct indexing because their future values may not have been created yet. Lists are convenient for small reusable collections, while generators are better for large or gradually produced data.

Example

# A list comprehension creates all values immediately
```

square_list = [number ** 2 for number in range(1, 6)]

# A generator expression produces values when requested

square_generator = (number ** 2 for number in range(1, 6))

print("List:", square_list)
print("Generator object:", square_generator)
print("Generator values:", list(square_generator))

Example Output

List: [1, 4, 9, 16, 25]
```

Generator object:  at 0x000001A234567890>
Generator values: [1, 4, 9, 16, 25]

Output Explanation

Printing the list displays all stored values. Printing the generator directly shows information about the generator object rather than its values. Converting the generator to a list requests and collects all remaining values.

26.6 Lazy Evaluation

Lazy evaluation means delaying a calculation until its result is actually needed. Generators use lazy evaluation because they do not produce every value when they are created. They calculate one value at a time as the program requests it.

This behavior can improve efficiency when a sequence is large or when the program may stop before using every possible value. It also allows generators to represent unlimited sequences because the program never tries to store all values at once.

Example

def calculate_squares(limit):
for number in range(1, limit + 1):
    print("Calculating square of", number)
    yield number ** 2
```

squares = calculate_squares(5)

print("Generator created")
print("First value:", next(squares))
print("Second value:", next(squares))

Output

Generator created
```

Calculating square of 1
First value: 1
Calculating square of 2
Second value: 4

Output Explanation

Creating the generator does not calculate any squares. The calculation happens only when next() requests a value. Because only two values are requested, the remaining three squares are never calculated.

26.7 Memory Efficiency

Generators are memory efficient because they usually keep only their current state and the information needed to create the next value. A list must store every element at the same time, which can require a large amount of memory.

Memory efficiency is especially important when processing millions of numbers, large files, database records, or continuously arriving information. A generator allows the program to process each item and then move to the next without storing the complete dataset.

Example

import sys
```

# Create a list containing one million squared values

square_list = [number ** 2 for number in range(1000000)]

# Create a generator for the same sequence

square_generator = (number ** 2 for number in range(1000000))

print("List size:", sys.getsizeof(square_list))
print("Generator size:", sys.getsizeof(square_generator))

Example Output

List size: 8448728
```

Generator size: 200

Output Explanation

The exact memory values may differ between computers and Python versions. However, the generator normally uses much less memory because it does not store one million completed results. It creates each square only when needed.

26.8 Multiple yield Statements

A generator function can contain several yield statements. Each statement can return a different value or represent a different stage of a process. The generator pauses at each one and resumes from the next line when another value is requested.

Multiple yield statements are useful when a generator has a fixed sequence of steps or must produce values from different parts of a function. They make it possible to describe a sequence clearly without building and returning a complete list.

Example

def traffic_light():
yield "Red"
yield "Green"
yield "Yellow"
```

lights = traffic_light()

for light in lights:
print(light)

Output

Red
```

Green
Yellow

Output Explanation

The generator produces one traffic-light value at a time. After yielding Red, it pauses. It later resumes and yields Green, followed by Yellow. When the function reaches the end, iteration stops.

26.9 Sending Values to Generators

A generator can receive information while it is paused. This makes generators more than simple value producers. They can also react to values sent by the calling code. The received value becomes the result of the paused yield expression.

Before sending a non-None value, the generator must first be started. This is commonly done with next(generator). Starting the generator moves execution to its first yield statement, where it waits for a value.

Example

def receiver():
print("Generator started")

received_value = yield

print("Received:", received_value)
```

data_receiver = receiver()

# Start the generator

next(data_receiver)

# Send a value into the paused generator

try:
data_receiver.send("Hello Generator")
except StopIteration:
pass

Output

Generator started
```

Received: Hello Generator

Output Explanation

The first next() call starts the generator and pauses it at yield. The send() call passes the text into the generator. That text becomes the value assigned to received_value.

26.10 The send() Method

The send() method resumes a paused generator and provides a value to it. The value is received by the generator at the location of the paused yield expression. The method may also return the next value yielded by the generator.

Calling generator.send(None) starts a generator in the same way as next(generator). Sending other values before the generator has reached its first yield causes a TypeError.

Example

def running_total():
total = 0

while True:
    # Yield the current total and wait for a new number
    number = yield total

    if number is not None:
        total += number
```

calculator = running_total()

# Start the generator and receive the initial total

print(calculator.send(None))

# Send numbers and receive updated totals

print(calculator.send(10))
print(calculator.send(5))
print(calculator.send(20))

Output

0
```

10
15
35

Output Explanation

The first call starts the generator and returns the initial total of zero. Each later call sends a number. The generator adds the number to the total and yields the updated result.

26.11 Closing Generators

The close() method stops a generator before it naturally finishes. Python sends a special GeneratorExit exception into the generator. This gives the generator an opportunity to perform cleanup operations before it ends.

Cleanup may include closing a file, releasing a resource, saving progress, or displaying a final message. After a generator is closed, calling next() raises StopIteration.

Example

def number_stream():
number = 1

try:
    while True:
        yield number
        number += 1

finally:
    print("Generator is closing.")
```

stream = number_stream()

print(next(stream))
print(next(stream))
print(next(stream))

stream.close()

Output

1
```

2
3
Generator is closing.

Output Explanation

The generator produces three values. Calling close() causes the finally block to run, which displays the closing message. The generator then ends and cannot produce additional values.

26.12 Throwing Exceptions into Generators

The throw() method sends an exception into a paused generator. Inside the generator, the exception appears at the location of the current yield statement. The generator may catch the exception and respond to it.

This advanced feature can be useful when the calling code needs to notify a generator about an error, interruption, invalid condition, or change in processing. If the generator does not catch the exception, it leaves the generator and may stop the program.

Example

def task_generator():
try:
    while True:
        task = yield "Waiting for a task"
        print("Processing:", task)

except ValueError:
    print("Invalid task received.")
```

tasks = task_generator()

print(next(tasks))
print(tasks.send("Prepare report"))

try:
tasks.throw(ValueError)
except StopIteration:
pass

Output

Waiting for a task
```

Processing: Prepare report
Waiting for a task
Invalid task received.

Output Explanation

The generator first waits for a task. The sent task is processed, and the generator pauses again. The throw() method then sends a ValueError, which the generator catches and handles with a message.

26.13 yield from

The yield from statement allows one generator to produce all values from another iterable or generator. It replaces a loop that would otherwise yield each value individually. This makes generator code shorter and easier to read.

It is useful when combining several sequences or dividing a complex generator into smaller generator functions. The outer generator automatically passes through the values produced by the inner iterable.

Example

def first_group():
yield 1
yield 2
yield 3
```

def second_group():
yield 4
yield 5
yield 6

def all_numbers():
# Produce every value from both generators
yield from first_group()
yield from second_group()

for number in all_numbers():
print(number)

Output

1
```

2
3
4
5
6

Output Explanation

The all_numbers() generator first yields every value from first_group(). It then yields every value from second_group(). The caller receives one continuous sequence.

26.14 Generator Pipelines

A generator pipeline connects several generators so that the output of one becomes the input of another. Each stage performs one task, such as creating values, filtering records, transforming information, or calculating a result.

Generator pipelines are memory efficient because values move through the stages one at a time. The program does not need to create a complete intermediate list after every operation. Pipelines are useful for files, logs, database records, and data-processing systems.

Example

def generate_numbers(limit):
for number in range(1, limit + 1):
    yield number
```

def keep_even(numbers):
for number in numbers:
if number % 2 == 0:
yield number

def square_values(numbers):
for number in numbers:
yield number ** 2

# Build the pipeline

numbers = generate_numbers(10)
even_numbers = keep_even(numbers)
squared_even_numbers = square_values(even_numbers)

for value in squared_even_numbers:
print(value)

Output

4
```

16
36
64
100

Output Explanation

The first generator produces numbers from 1 to 10. The second keeps only even numbers. The third squares each remaining value. Every number moves through the pipeline one at a time.

26.15 Practical Generator Applications

Generators are useful for reading large files, processing database records, generating report rows, handling sensor values, creating paginated results, and transforming streamed information. They allow programs to work with one piece of data at a time.

In the following example, a generator processes sales records and yields only those that meet a minimum amount. This pattern could be used to filter large datasets without creating a separate list containing every matching record.

Example: Sales Filter Generator

def high_value_sales(sales, minimum_amount):
for sale in sales:
    if sale["amount"] >= minimum_amount:
        yield sale
```

sales_data = [
{"customer": "Ali", "amount": 150},
{"customer": "Sara", "amount": 450},
{"customer": "Michael", "amount": 275},
{"customer": "Emma", "amount": 600}
]

filtered_sales = high_value_sales(sales_data, 300)

for sale in filtered_sales:
print(sale["customer"], "-", sale["amount"])

Output

Sara - 450
```

Emma - 600

Output Explanation

The generator examines each sale one at a time. It yields only records with an amount of at least 300. The records for Sara and Emma meet the condition, so only those records are displayed.

26.16 Chapter Practice Exercises

These exercises help you practise generator functions, generator expressions, lazy evaluation, memory efficiency, communication methods, and generator pipelines. Begin with simple generators before moving to the advanced exercises.

1. Create a generator that yields the numbers from 1 to 5.
2. Create a generator that counts backward from 10 to 1.
3. Create a generator that yields even numbers from 2 to 20.
4. Create a generator that yields odd numbers from 1 to 19.
5. Create a generator that produces square numbers.
6. Create a generator expression that doubles the numbers from 1 to 10.
7. Convert a generator expression into a list.
8. Compare the memory size of a list and a generator.
9. Create a generator with three separate yield statements.
10. Create an infinite generator that produces multiples of 5.
11. Limit an infinite generator to the first ten values.
12. Create a generator that receives numbers using send().
13. Create a running-average generator.
14. Create a generator that performs cleanup when close() is called.
15. Send a custom exception into a generator using throw().
16. Combine two generators using yield from.
17. Create a generator pipeline that filters positive numbers.
18. Create a pipeline that converts words to uppercase.
19. Create a generator that reads and strips lines from a text file.
20. Create a generator that processes student scores one at a time.
21. Create a generator that yields records above a selected value.
22. Create a generator-based countdown.

Practice Example: Multiples Generator

def multiples(number, limit):
current = number

while current <= limit:
    yield current
    current += number
```

for value in multiples(5, 30):
print(value)

Output

5
```

10
15
20
25
30

Output Explanation

The generator starts with 5 and adds 5 after every yielded value. It continues until the current value becomes greater than 30.

26.17 Chapter Mini Project

Project: Generator-Based Sales Report Pipeline

In this mini project, you will build a generator pipeline that processes sales records. The first generator provides records one at a time. The second generator filters records by a minimum sale amount. The third generator calculates tax and produces a formatted report record.

This project combines generator functions, yield, lazy evaluation, pipelines, dictionaries, calculations, loops, and totals. It demonstrates how large collections can be processed efficiently without creating several intermediate lists.

Complete Program

def generate_sales(sales):
"""Yield one sale record at a time."""

for sale in sales:
    print("Reading sale for", sale["customer"])
    yield sale
```

def filter_sales(sales, minimum_amount):
"""Yield only sales that meet the minimum amount."""

```
for sale in sales:
    if sale["amount"] >= minimum_amount:
        yield sale
```

def calculate_tax(sales, tax_rate):
"""Add tax information to each sale."""

```
for sale in sales:
    tax = sale["amount"] * tax_rate
    final_total = sale["amount"] + tax

    yield {
        "customer": sale["customer"],
        "subtotal": sale["amount"],
        "tax": tax,
        "total": final_total
    }
```

def format_report(sales):
"""Create a formatted line for each sale."""

```
for sale in sales:
    report_line = (
        f'{sale["customer"]}: '
        f'Subtotal ${sale["subtotal"]:.2f}, '
        f'Tax ${sale["tax"]:.2f}, '
        f'Total ${sale["total"]:.2f}'
    )

    yield report_line
```

sales_data = [
{"customer": "Ali", "amount": 125.00},
{"customer": "Sara", "amount": 450.00},
{"customer": "Michael", "amount": 275.00},
{"customer": "Emma", "amount": 600.00},
{"customer": "David", "amount": 90.00},
{"customer": "Nora", "amount": 350.00}
]

minimum_sale = 250.00
tax_rate = 0.13

# Build the generator pipeline

sales = generate_sales(sales_data)
filtered = filter_sales(sales, minimum_sale)
taxed_sales = calculate_tax(filtered, tax_rate)
report_lines = format_report(taxed_sales)

print("Sales Report")
print("------------------------------------------")

report_count = 0

for line in report_lines:
print(line)
report_count += 1

print("------------------------------------------")
print("Records in report:", report_count)

Output

Sales Report
```

---

Reading sale for Ali
Reading sale for Sara
Sara: Subtotal $450.00, Tax $58.50, Total $508.50
Reading sale for Michael
Michael: Subtotal $275.00, Tax $35.75, Total $310.75
Reading sale for Emma
Emma: Subtotal $600.00, Tax $78.00, Total $678.00
Reading sale for David
Reading sale for Nora
Nora: Subtotal $350.00, Tax $45.50, Total $395.50
-------------------------------------------------

Records in report: 4

Project Explanation

The generate_sales() function yields one sales record at a time. The printed reading messages show that records are processed gradually. The complete collection is not copied into another list.

The filter_sales() generator receives records from the first generator and yields only sales of at least 250 dollars. Lower sales continue through the first stage but are not passed into the later stages.

The calculate_tax() generator calculates 13 percent tax and creates a new dictionary containing the subtotal, tax, and final total. Each new record is yielded immediately.

The format_report() generator converts each processed dictionary into a readable report line. The main loop requests one final line at a time, which causes the required record to move through the complete pipeline.

How to Run the Mini Project

1. Open Visual Studio Code, IDLE, PyCharm, or another Python editor.
2. Create a new file named generator_sales_report.py.
3. Copy the complete project code into the file.
4. Save the file.
5. Open a terminal in the folder containing the file.
6. Run python generator_sales_report.py.
7. On some computers, run python3 generator_sales_report.py.
8. Review which records are read and which records appear in the report.
9. Change the minimum sale amount and run the program again.
10. Change the tax rate and compare the results.

Project Challenges

• Ask the user to enter the minimum sale amount.
• Ask the user to enter the tax rate.
• Add a date to every sales record.
• Filter sales by customer name.
• Filter sales by a minimum and maximum amount.
• Add a discount stage to the generator pipeline.
• Calculate the total value of all report records.
• Read sales from a CSV file using a generator.
• Save the formatted report to a text file.
• Use yield from to combine records from several stores.
• Add error handling for missing or invalid sales values.
• Create a second report containing rejected records.