Selenium provides automation of several browsers, allowing the user to interact with the browser in many different ways. It has numerous use cases, such as testing, debugging, scrapping, etc.
It can be helpful, sometimes, to parse through the HTML of the page inside the browser, which is being run using Selenium. XPath is handy for this task, as it can search for an element by traversing the entire document. In this answer, we’ll see three important methods for searching elements in an HTML using XPath: and, contains(), and or.
We will be using the Firefox browser with Selenium in Python. The functionality of XPath is constant.
contains()We’ll use the contains() command to find the element in HTML containing a given text. It is used with the following syntax:
//Tag[contains (@Attribute, 'Value')]
In the syntax above:
Tag: This can be input, button, a, ul, or any other tag used in HTML syntax. Instead of a specific tag, * can be used in XPath, to search all the tags, rather than just 1.
Attribute: This can be name, class, id, or any other attribute used in HTML.
Value: This is the text we are searching for. The element that will be returned will have this value in the specific attribute, with the specified tag.
In XPath, partial text can also be given as “Value”; if that partial text is a subset of the complete text, even then the element can be found.
In the following example, XPath will be used to find a search bar at this site, after which we will input a word Random into the search bar. Then XPath will be used again to find the submit button, which we will click to search our input.
For the first 20 seconds, we can right click on the search bar, then click on “Inspect.” We can also right click on “Go” and then click on “Inspect.” This way, the page’s source will open highlighting the exact HTML for the part where we would have clicked. We can check the required attributes there.
#Importing the necessary libraries
import time as tt
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium import webdriver
from selenium.webdriver.firefox.service import Service as FirefoxService
from webdriver_manager.firefox import GeckoDriverManager
#Setting up the driver and running the url
options = Options()
driver = webdriver.Firefox(service=FirefoxService(GeckoDriverManager().install()), options=options)
driver.get("https://www.python.org/")
# Finding search bar with Contains() sending them a word 'Random'
inputelement = driver.find_element(By.XPATH, value = "//input[contains(@name, 'q')]")
tt.sleep(20)
inputelement.send_keys("Random")
## Clicking on GO, with the help of Contains().
tt.sleep(4)
myelement = driver.find_element(By.XPATH, value = "//button[contains(@name, 'submit')]")
myelement.click()
##Getting the new URL.
get_url = driver.current_url
print("The current url is: "+str(get_url))
#driver.quit() Driver will continueLine 15: The code uses XPath to find an element which is of type input (it will be taking an input from the user), and can be identified by the attribute of name, which should contain the text q—usually a name assigned to inputs which are used for searching, although it’s not necessary.
Line 21: The code uses XPath to find an element which is of the type button (it will be clickable), and can be identified by the attribute of name, which should contain the text subm. In the given URL, the actual text is “submit,” but XPath still returns the element because it can take partial text as well.
orWe’ll use the or command to find the element in HTML satisfying either of the conditions we provide. Syntax for using the or command in selenium is as follows:
//Tag[@AttributeNumber1="Value1" or AttributeNumber2 = "Value2"]
In the syntax above:
Tag: This can be input, button, a, ul, or any other tag used in HTML syntax. Instead of a specific tag, * can be used in XPath, to search all the tags, rather than just 1.
AttributeNumber1 and AttributeNumber2: These will each be defining attributes which can be assigned to an element, such as name, class, id, or any other attribute used in HTML.
Value1 and Value2: These will each be the text we are searching for in their given attributes. The element that will be returned will have at least one of these value in the specified attribute, with the specified tag.
In the following example, XPath will be used to find clickable text “Learn More,” at this site. For doing so, we’ll be using an attribute of name, which will not be matching any condition, along with an attribute of class, which will be matching the correct condition.
For the first 20 seconds, we can right click on the “Learn More,” then click on “Inspect.” This way, the page’s source will open, highlighting the exact HTML for the “Learn More” button. We can check the required attributes there.
## Importing necessary libraries
import time as tt
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium import webdriver
from selenium.webdriver.firefox.service import Service as FirefoxService
from webdriver_manager.firefox import GeckoDriverManager
# Setting up driver and calling the URL
options = Options()
driver = webdriver.Firefox(service=FirefoxService(GeckoDriverManager().install()), options=options)
driver.get("https://www.python.org/")
# Finding element by OR and clicking on it
inputelement = driver.find_element(By.XPATH, value = "//a[@name = 'button' or @class = 'readmore']")
tt.sleep(20)
inputelement.click()
##Getting the new URL
get_url = driver.current_url
print("The current url is: "+str(get_url))
#driver.quit() Driver will continueLine 15: The code uses XPath to find an element which is of type a, and can be identified either by the attribute of name, which should have the text button, or by the attribute of class, which should have the text readmore.
andWe’ll use the and command to find the element in HTML satisfying all—not just 1—of the conditions we provide. Syntax for using and in selenium is as follows:
//Tagname[@AttributeNumber1="Value1" and AttributeNumber2 = "Value2"]
In the syntax above:
Tag: This can be input, button, a, ul, or any other tag used in HTML syntax. Instead of a specific tag, * can be used in XPath, to search all the tags, rather than just 1.
AttributeNumber1 and AttributeNumber2: These will each be defining attributes which can be assigned to an element, such as name, class, id, or any other attribute used in HTML.
Value1 and Value2: These will each be the text we are searching for in their given attributes. The element that will be returned will have all of these values in their respective attribute, with the specified tag.
In the following example, XPath will be used to find clickable text “Jobs,” at this href and title, both of which will match their required values.
For the first 20 seconds, we can right click on the “Jobs,” then click on “Inspect.” This way, the page’s source will open highlighting the exact HTML for the “Jobs” button. We can check the required attributes there.
#Importing necessary libraries
import time as tt
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium import webdriver
from selenium.webdriver.firefox.service import Service as FirefoxService
from webdriver_manager.firefox import GeckoDriverManager
#Setting up driver and calling site's url
options = Options()
driver = webdriver.Firefox(service=FirefoxService(GeckoDriverManager().install()), options=options)
driver.get("https://www.python.org/")
## Finding the element to be clicked, using AND operator in XPath
inputelement = driver.find_element(By.XPATH, value = "//a[@href = '/jobs/' and @title='Python Job Board']")
tt.sleep(20)
inputelement.click()
##Finding the new url and quitting the driver
get_url = driver.current_url
print("The current url is: "+str(get_url))
#driver.quit() Driver will continueLine 15: The code uses XPath to find an element which is of type a, and satisfies the given condition. That condition is that the element should have the attribute of href, with the text /jobs/, and also the attribute of class, with the text readmore.
In this answer, we have learnt about using XPath with Selenium. We have tried all the codes on a specific URL, showing the complete processes of how the element will be searched.
Free Resources