What are generators in Python, and how to use them?

Generators are an important feature of Python that allows for efficient and memory-friendly iteration over large data sequences. Unlike regular functions that return a single value and terminate, generators can produce a sequence of values over time, pausing and resuming as needed. They are built using the yield keyword instead of return, offering significant advantages in terms of memory and performance.

Generators are special functions that use the yield keyword instead of return to produce a series of values. They are similar to iterators but with simpler syntax and improved memory efficiency. When a generator function is called, it returns an iteratorAn object that can be used to loop through collections. object that can be used to iterate over the sequence of values produced by the generator.

The yield keyword is used in generator functions to create a generator object. It is a powerful feature that allows the generator function to produce a sequence of values, one at a time, without the need to store the entire sequence in memory. This makes the generators highly efficient, especially when working with large or potentially infinite data sequences.

Iterators and generators in Python
Iterators and generators in Python

How to create a generator in Python

To create a generator, you define a function with the yield keyword.

Syntax of Python generator
Syntax of Python generator

Unlike a regular function that returns a single value, a generator will produce one value at a time and pause its state between each value.

# Define a generator function named 'my_generator'
def my_generator():
    # Loop through numbers 0 to 4
    for i in range(5):
        yield i  # Yield each number, pausing the function's state after each one
# Create a generator object by calling 'my_generator'
gen = my_generator()
# Iterate over each item produced by the generator
for item in gen:
    print(item)  # Print each item produced by the generator

In the above code:

  1. Generator definition: my_generator() is a generator function that uses yield to produce numbers 0 to 4, pausing after each yield.

  2. Creating generator object: gen = my_generator() creates a generator object but doesn’t execute until values are requested.

  3. Loop to access items: The for item in gen loop fetches each yielded value from the generator, printing 0, 1, 2, 3, 4 one at a time.

The generator pauses and resumes, making it memory-efficient, ideal for large sequences.

Uses of Python generators

Generators are particularly useful for:

  1. Working with large datasets: They allow for efficient data processing without consuming large amounts of memory.

  2. Lazy evaluation: Generators compute values only when requested, which improves performance.

  3. Infinite sequences: You can generate an infinite sequence of values with generators.

Difference between generator function and normal function

Let's see the difference between a normal function and a generator function:

  • Normal function: Returns a value and exits.

  • Generator function: Uses yield to return multiple values, pausing and resuming its state after each one.

# Normal function that returns a list of numbers
def normal_func():
    return [1, 2, 3]
# Generator function that yields one number at a time
def generator_func():
    for i in range(1, 4):
        yield i  # Yield each number, pausing between each
# Calling the normal function and printing the entire list
print(normal_func())  # Output: [1, 2, 3]
# Creating a generator object and printing the first value yielded
gen = generator_func()
print(next(gen))  # Output: 1

While the normal function returns the entire list at once, the generator yields values one at a time.

What is a generator object?

When a generator function is called, it doesn't immediately execute its body. Instead, it returns a generator object that can be iterated over.

# Define a simple generator function that yields two strings
def simple_gen():
    yield "First"  # First yield statement
    yield "Second"  # Second yield statement
# Create a generator object from the function
gen_obj = simple_gen()
# Retrieve the first yielded value
print(next(gen_obj))  # Output: "First"
# Retrieve the second yielded value
print(next(gen_obj))  # Output: "Second"

The generator object holds the state of the function and resumes execution when next() is called. Once all values are yielded, a StopIteration exception is raised.

Python generator expression

You can create simple generators using generator expressions, similar to list comprehensions but with parentheses instead of brackets.

# Generator expression that yields squares of numbers from 0 to 4
gen_exp = (x * x for x in range(5))
# Iterate over the generator expression and print each square
for val in gen_exp:
    print(val)  # Outputs 0, 1, 4, 9, 16

In the above code, gen_exp is a generator expression that yields the squares of numbers 0 through 4 without storing the entire sequence in memory.

Use cases of generators

To create a generator, we define a function using the yield keyword. The yield keyword suspends the execution of the function, saves its internal state, and returns a value. The next time the generator’s iterator is called, the function resumes execution from where it left off, continuing until the next yield keyword is encountered.

Let’s understand the concept with some code examples:

Example 1: Generating Fibonacci numbers

def fibonacci_generator():
    # Initialize the first two numbers of the sequence
    a, b = 0, 1
    while True:
        yield a  # Yield the current number in the sequence
        a, b = b, a + b  # Update a and b to the next numbers in sequence
# Usage of the generator
fib_gen = fibonacci_generator()
for i in range(10):
    print(next(fib_gen))  # Print the next Fibonacci number

In this example, we define a generator function fibonacci_generator() that yields Fibonacci numbers indefinitely. We can then create an instance of the generator and use the next() function to retrieve the next value from the sequence.

Try it yourself: Modify the fibonacci_generator function so it resets once it reaches a value over 100. Observe how this changes the generator's behavior.

Note: Generators are one-time iterators, meaning that once a generator is iterated over, it cannot be restarted or reused. If you need to iterate over the sequence multiple times, you will need to create a new generator object.

Example 2: Filtering even numbers

# Generator function that yields even numbers within a given range
def even_numbers(start, end):
    for num in range(start, end + 1):
        if num % 2 == 0:  # Check if the number is even
            yield num  # Yield the even number
# Usage of the generator to print even numbers from 1 to 10
even_gen = even_numbers(1, 10)
for num in even_gen:
    print(num)  # Outputs even numbers: 2, 4, 6, 8, 10

Here, the even_numbers() generator function generates even numbers within a given range. The generator filters out odd numbers using a conditional statement, yielding only even numbers. We can then iterate over the generator to print the filtered values.

Example 3: Reading large files

main.py
large_data.txt
# Generator function to read a large file line by line
def read_large_file(file_path):
    with open(file_path, 'r') as file:
        for line in file:
            yield line.strip()  # Yield each line, stripped of trailing whitespace
# Function to process each line (example)
def process_line(line):
    print(line)  # Print the line (can be replaced with any processing)
# Usage of the generator to read and process each line of a large file
large_file_gen = read_large_file('large_data.txt')
for line in large_file_gen:
    process_line(line)  # Process each line
    # break  # Uncomment this line to stop after the first line

This example demonstrates the use of generators to read large files. Instead of loading the entire file into memory, the read_large_file() generator function reads and yields one line at a time. This approach is memory-efficient and enables processing large files without overwhelming system resources.

Note: Generators are designed to produce values on-demand and avoid storing the entire sequence in memory. If you explicitly convert a generator to a list using the list() function, it will consume memory as it generates all the values at once. Be mindful of memory usage when working with large or infinite sequences.

Exhausted generator

An exhausted generator is a generator that has yielded all of its values and cannot produce any more. Once a generator has yielded its last value, subsequent calls to next() on the generator will raise a StopIteration exception, indicating there are no more items to retrieve.

# Define a simple generator function
def simple_generator():
    yield 1  # First yield statement, yields the value 1
    yield 2  # Second yield statement, yields the value 2
# Create a generator object from the function
gen = simple_generator()
# Retrieve and print the first yielded value
print(next(gen))  # Output: 1
# Retrieve and print the second yielded value
print(next(gen))  # Output: 2
# Attempt to retrieve a third value, which does not exist
try:
    print(next(gen))  # Raises StopIteration because the generator is exhausted
except StopIteration:
    print("Generator is exhausted")  # Handles the StopIteration exception and prints a message

This code shows that once all values are yielded, calling next() again results in a StopIteration error, indicating the generator is exhausted and no further values are available.

Advantages of using generators

  • Memory efficiency: Generators produce values on-the-fly, which means they don’t require storing the entire sequence in memory. This makes them ideal for working with large or infinite sequences.

Memory saving using generator
Memory saving using generator

  • Lazy evaluation: Generators follow the principle of lazy evaluation, where values are computed only when needed. This enables efficient processing of data, especially when not all values are required.

Traditional Vs. Lazy evaluation
Traditional Vs. Lazy evaluation

  • Simplified syntax: Generators provide a simpler and more readable syntax than implementing custom iterator classes. They eliminate the need for maintaining explicit state variables and managing iteration logic.

Challenge

Before we wrap up, let’s put your knowledge of Python generators to a quick challenge!

Task: Write a generator that returns prime numbers indefinitely.

def prime_generator():
  # Write your code here
  return -1

You can refer to the given solution after pressing the "Run" button if you are stuck.

Key takeaways

  • Generators are a powerful feature in Python, allowing efficient and memory-friendly iteration over large or infinite sequences.

  • Using the yield keyword, a generator produces values one at a time without storing the entire sequence in memory, enabling lazy evaluation.

  • Generators provide simpler syntax compared to custom iterators and can handle large datasets more effectively by generating values only when needed.

  • Unlike normal functions, which return values and exit, generators can pause and resume their state.

  • Generators are particularly useful when dealing with data streams, reading large files, or processing data where not all values need to be computed at once.

Become a Python developer with our comprehensive learning path!

Ready to kickstart your career as a Python Developer? Our Become a Python Developer path is designed to take you from your first line of code to landing your first job.

This comprehensive journey offers essential knowledge, hands-on practice, interview preparation, and a mock interview, ensuring you gain practical, real-world coding skills. With our AI mentor by your side, you’ll overcome challenges with personalized support, building the confidence needed to excel in the tech industry.

Frequently asked questions

Haven’t found what you were looking for? Contact Us

What is the difference between yield and return in Python?

The return statement exits a function completely and returns a value. In contrast, yield pauses the function, saving its state for future resumption, and returns a value without terminating the function.

Can a generator return multiple values in Python?

Yes, a generator can yield multiple values over time by pausing at each yield statement and resuming later when needed.

Are Python generators faster than lists or other data structures?

Generators are more memory-efficient than lists because they don’t store values in memory. In terms of speed, they can be faster for generating large sequences since values are produced on-the-fly.

Can I convert a generator to a list in Python?

Yes, you can convert a generator to a list using the list() function. However, this will load all the generated values into memory, which negates the memory efficiency benefits of a generator.

What is the use of next() with generators in Python?

The next() function retrieves the next value from a generator. Each time next() is called, the generator resumes execution from where it left off, yielding the next value until the sequence is exhausted.

What is generator vs. iterator in Python?

In Python, both generators and iterators allow for lazy evaluation, meaning they produce items one at a time on demand rather than all at once. However, the key difference is how they are created and used:

  • Generators: These are a type of iterator, created using the yield keyword in functions or with generator expressions. Generators save their state between executions, which allows them to produce a series of values lazily without holding everything in memory. They are simpler and more memory-efficient for large sequences or streams of data.

  • Iterators: These are objects that implement the __iter__() and __next__() methods. You can create an iterator from an iterable (e.g., lists, tuples) using the iter() function. Unlike generators, you don’t need yield to create them, but they can be less convenient to write.

In short, all generators are iterators, but not all iterators are generators. Generators are particularly useful when dealing with large or infinite sequences of data, as they are more memory-efficient. Have a look at Generator vs. iterator in Python for more details.

What are some tips for debugging generator functions in Python?

Use print statements before and after each yield to understand how the generator flows. For complex logic, consider converting the generator to a list temporarily for easier inspection, but remember this may consume more memory.


Free Resources

Copyright ©2025 Educative, Inc. All rights reserved