How to use Beautiful Soup's find() method

Key takeaways:

  • Use find() to locate the first occurrence of a tag such as <h1>, <p>, or <div>.

  • Narrow the search using a dictionary of attributes like class, id, or href.

  • With recursive=False parameter of find(), we can restrict the search to only direct children of the current element.

  • Use the text parameter to find elements containing specific text or using regular expressions.

  • Apply multiple filters, such as tag, attribute, and text content, to find elements more precisely.

Imagine you're looking at a huge, messy page of text. You want to find a specific word, like "science," but it's buried under many other words. Beautiful Soup is like a smart assistant that helps you quickly locate "science" and anything you search on a web page.

The find() method

The find() method in Beautiful Soup helps you locate the first matching element within an HTML or XML document. You can specify what you're looking for by providing a tag name, class, or other attributes.

Syntax of the find() method

The basic syntax of the find() method is as follows:

find(name, attrs, recursive, text, **kwargs)

  • name: The tag name or a list of tag names to be searched.

  • attrs: A dictionary of attributes and their corresponding values to filter elements.

  • recursive: A Boolean value to specify whether to search only the direct children or the entire descendants (default is True).

  • text: A string or regular expression to find elements containing specific text.

  • **kwargs: Allows us to use CSS selectors or other filters for specific use cases.

Usage of the find() method

Here are some of the functionalities that we can perform using the find() method:

  • Finding elements by tag name, id, and class name

  • Filtering elements by attributes

  • Finding element within immediate children

  • Finding elements by text content or Regex

  • Finding elements with multiple criteria

1. Finding elements by tag name

To locate elements based on their tag names, pass the tag name as the first argument to the find() method:

element = soup.find('tag_name')

Replace the 'tag_name' with the actual HTML tag name, such as 'div', 'a', etc. Let's find the element with a tag 'h1':

main.py
sample.html
<!DOCTYPE html>

<html>

<head>

    <title>Educative - Learn, Explore, and Grow</title>

</head>

<body>

    <header>

        <h1>Welcome to Educative</h1>

        <nav>

            <ul>

                <li>Courses with Assessments</li>

                <li>Assessments</li>

                <li>Blog</li>

                <li>About Us</li>

            </ul>

        </nav>

    </header>

    <div class='description'>

      Educative provides interactive courses for software developers. We are changing how 

      developers continue their education and stay relevant by providing pre-configured 

      learning environments that adapt to match a developer's skill level.

    </div>

</body>

</html>

In the code above, the soup.find('h1') traverse from the start of soup, finds the first occurrence of h1 and returns that element. In case the element is not found, the find() method returns None. Here is an example:

main.py
sample.html
h2_element = soup.find('h2')
print("First Occurrence of h2 tag:", h2_element)

If you want to learn about how to find element by id or class name with find() method, you can visit our Answers on How to find element by ID using Beautiful Soup and How to find elements by class using Beautiful Soup.

2. Filtering elements by attributes

We can also narrow down our search by using attributes. For this, we need to provide a dictionary containing the attribute-value pairs to match:

element = soup.find('tag_name', attrs={'attribute': 'value'})

For instance, to find a specific div with the class='description', use:

main.py
sample.html
<!DOCTYPE html>

<html>

<head>

    <title>Educative - Learn, Explore, and Grow</title>

</head>

<body>

    <header>

        <h1>Welcome to Educative</h1>

        <nav>

            <ul>

                <li>Courses with Assessments</li>

                <li>Assessments</li>

                <li>Blog</li>

                <li>About Us</li>

            </ul>

        </nav>

    </header>

    <div class='description'>

      Educative provides interactive courses for software developers. We are changing how 

      developers continue their education and stay relevant by providing pre-configured 

      learning environments that adapt to match a developer's skill level.

    </div>

</body>

</html>

3. Finding element within immediate children

By default, the find() searches through the entire document. By setting the recursive=False, the find() method will limit its search scope to only the immediate children of the element you are calling it on. It won't search deeper into the document's hierarchy beyond the first level.

element = soup.find('tag_name', recursive=False)

Here is how it works:

main.py
sample.html
<!DOCTYPE html>

<html>

<head>

    <title>Educative - Learn, Explore, and Grow</title>

</head>

<body>

    <header>

        <h1>Welcome to Educative</h1>

        <nav>

            <ul>

                <li>Courses with Assessments</li>

                <li>Assessments</li>

                <li>Blog</li>

                <li>About Us</li>

            </ul>

        </nav>

    </header>

    <div class='description'>

      Educative provides interactive courses for software developers. We are changing how 

      developers continue their education and stay relevant by providing pre-configured 

      learning environments that adapt to match a developer's skill level.

    </div>

</body>

</html>

In the code above, first we find the 'body' tag in soup with the recursive set to False. This returns None, since soup has only one immediate tag html. So we find 'html' from soup and then use the find method on html_element to find the 'body' tag, which is now the immediate child.

4. Finding elements by text content or regex

We can also use the text parameter to search for elements based on their text content:

element = soup.find(text='target_text')

This will return the first element that matches the given text. You can use exact text or regular expressions in place of the 'target_text'. Here is an example:

main.py
sample.html
<!DOCTYPE html>

<html>

<head>

    <title>Educative - Learn, Explore, and Grow</title>

</head>

<body>

    <header>

        <h1>Welcome to Educative</h1>

        <nav>

            <ul>

                <li>Courses with Assessments</li>

                <li>Assessments</li>

                <li>Blog</li>

                <li>About Us</li>

            </ul>

        </nav>

    </header>

    <div class='description'>

      Educative provides interactive courses for software developers. We are changing how 

      developers continue their education and stay relevant by providing pre-configured 

      learning environments that adapt to match a developer's skill level.

    </div>

</body>

</html>

5. Finding elements with multiple criteria

For complex cases, we can combine multiple filters using the find() method:

element = soup.find('tag_name', attrs={'attribute': 'value'}, recursive=False, text='target_text')

This will find the first element that satisfies all specified conditions. Here is a comprehensive example with complete implementation:

In addition to using the text parameter for exact matches, the find() method can also be used with regular expressions (regex) to find elements that match a specific pattern in their text content.

main.py
sample.html
# import beautiful soup
from bs4 import BeautifulSoup
#import re for using regular expression
import re
# Read the HTML content from the local file
file_path = 'sample.html'
with open(file_path, 'r', encoding='utf-8') as file:
    html_content = file.read()
# Parse the HTML content using BeautifulSoup
soup = BeautifulSoup(html_content, 'html.parser')
#Pattern
pattern = re.compile(r"Educative provides interactive.*skill level\.$", re.MULTILINE | re.DOTALL)
element=soup.find(name='div', attrs={'class': 'description'}, recursive=True, text=pattern)
print("Output:", element)

In the code above, a regular expression pattern is defined using the re.compile() function. The pattern r"Educative provides interactive.*skill level\.$" is used to match a string that starts with "software developers" and ends with "skill level". The re.MULTILINE and re.DOTALL flags are used to make the pattern match across multiple lines and handle newline characters. We then used the find() method that searches for a <div> element with the class attribute 'description' that contains text matching the previously defined pattern. The recursive=True argument tells Beautiful Soup to search for the element in nested structures as well.

Note: The find() method only returns first occurrence of matched element. To get all the elements of a specific criteria, you can use find_all().

Ready to master web scraping? 🚀

Unlock the power of web scraping with our course on Mastering Web Scraping Using Python: From Beginner to Advanced! Whether you’re a beginner or looking to enhance your skills, this course will guide you through the essentials to advanced techniques in web scraping.

Conclusion

The find() method is offered by the Beautiful Soup library which enables us to navigate HTML or XML documents with ease. By understanding the syntax and various filtering options of the find(), we can efficiently extract specific elements and data from web pages, making web scraping tasks more manageable and effective.

Frequently asked questions

Haven’t found what you were looking for? Contact Us

What does the find_all() method return?

The find_all() returns a list of all matching elements in the document.

How to find elements in BeautifulSoup?

We can use find() for the first match or find_all() for all matches.

What is the primary difference between find() and find_all()?

The primary difference is find() returns only the first match, while find_all() returns all matches.

How do you check if a list has elements in Python?

We can use if statement (if my_list:) to check if the list is not empty.

What does BeautifulSoup find return?

BeautifulSoup find() returns the first matching element or None if no match is found.

Free Resources

HowDev By Educative. Copyright ©2025 Educative, Inc. All rights reserved