How to create a SparkSession on PySpark

A SparkSession provides access to underlying PySpark features for programmatically creating a PySpark Resilient Distributed Dataset (RDD) and DataFrame.

In a PySpark application, we can create as many SparkSession as we like by calling SparkSession.builder() or SparkSession.newSession(). We’ll need a lot of Spark session objects if we want to keep PySpark tables (that are relational entities) logically isolated.

Create a SparkSession

To create a SparkSession in python, we'll need the following methods:

  • The builder()to create a SparkSession.
  • The getOrCreate() returns a SparkSession if it exists, otherwise, it creates a new session.
  • The appName() is used to set the application name.
  • The master() is used to set the master name as an argument to it (if run on a cluster). When running in standalone mode, we use local[x]. When we utilize RDD, DataFrame, and Dataset, x should be an integer value larger than 0. This represents the number of divisions to be created. Ideally, the x value should be the number of CPU cores.

Code example

Let's look at the code below:

from pyspark.sql import SparkSession
from dotenv import load_dotenv
def create_spark_session():
    """Create a Spark Session"""
    _ = load_dotenv()
    return (
        SparkSession
        .builder
        .appName("SparkApp")
        .master("local[5]")
        .getOrCreate()
    )
spark = create_spark_session()
print('Session Started')
print('Code Executed Successfully')
Creating a SparkSession

Code explanation

  • Lines 1–2: We import the required SparkSession library to create a PySpark session.
  • Lines 3–12: We create a function to create a PySpark session.
    • Line 3: We define the function.
    • Line 5: We load the environment.
    • Lines 6–12: We return the PySpark session.
      • Line 9: We assign the name of the session.
      • Line 10: We create 5 threads as logical cores on our machine locally.
  • Line 13: We call the function to create a PySpark session.
  • Line 14: We print that our session is started.
Free AI Mock Interviews
Coding Interview
Coding PatternsFree Interview
Gain insights and practical experience with coding patterns through targeted MCQs and coding problems, designed to match and challenge your expertise level.
System Design
You TubeFree Interview
Learn to design a video streaming platform like YouTube by tackling functional and non-functional requirements, core components, and high-level to detailed design challenges.

Free Resources

Copyright ©2025 Educative, Inc. All rights reserved