How to rename multiple columns in Pyspark

The withColumnRenamed() method is used to rename an existing column. The method returns a new DataFrame with the newly named column. Multiple columns in a DataFrame can be renamed by chaining the withColumnRenamed() method for each column.

Syntax

DataFrame.withColumnRenamed(existing, new)

Parameters

existing: This is the name of the existing column.
new: This is the new name to be given to the existing column.

Return value

A new DataFrame is generated with the renamed columns.

Code example

Let’s look at the code below:

import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('edpresso').getOrCreate()
data = [("James","Smith","USA","CA"),
    ("Michael","Rose","USA","NY"),
    ("Robert","Williams","USA","CA"),
    ("Maria","Jones","USA","FL")
  ]
columns = ["firstname","lastname","country","state"]
df = spark.createDataFrame(data = data, schema = columns)
print("Original dataframe:")
df.show(truncate=False)
new_df = df.withColumnRenamed("firstname", "First-Name") \
          .withColumnRenamed("lastname", "Last-Name") \
          .withColumnRenamed("country", "Country")
print("Renamed dataframe:")
new_df.show(truncate=False)

Code explanation

Lines 1–2: We import the pyspark and SparkSession.
Line 4: A spark session named edpresso is created.
Lines 6–10: We define data for the DataFrame.
Line 12: The names of the DataFrame’s columns are defined.
Line 13: A DataFrame is created using the createDataframe() method.
Line 15: The original DataFrame is printed.
Lines 17-19: Multiple columns of DataFrame are renamed by chaining the withColumnRenamed() method.
Line 23: The new DataFrame with new column names is printed.

Free AI Mock Interviews

Coding Interview

Coding PatternsFree Interview

Gain insights and practical experience with coding patterns through targeted MCQs and coding problems, designed to match and challenge your expertise level.

System Design

You TubeFree Interview

Learn to design a video streaming platform like YouTube by tackling functional and non-functional requirements, core components, and high-level to detailed design challenges.

Free Resources