Local storage encryption in Spark 3

svg viewer

Spark ensures encryption for RPC Connections and it allows users to encrypt data they are storing on local drive (includes shuffle files, shuffle spills, and data blocks).

Note: Spark does not support the encryption of data generated by applications with APIs, such as “saveAsHadoopFile”, or temporary files created by the user.

The following table summarizes how to enable encryption for data that is written to script:

svg viewer
New on Educative
Learn to Code
Learn any Language as a beginner
Develop a human edge in an AI powered world and learn to code with AI from our beginner friendly catalog
🏆 Leaderboard
Daily Coding Challenge
Solve a new coding challenge every day and climb the leaderboard

Free Resources

Copyright ©2025 Educative, Inc. All rights reserved