Spark provides several storage levels to help control how data is cached across memory and disk. Each level offers a different trade-off between performance, memory usage, and fault tolerance. Below are the most commonly used storage levels:
-
MEMORY_ONLY:
Stores the data in memory as deserialized Java objects. If the data doesn’t fit entirely in memory, Spark will not store the overflow on disk—instead, it will recompute the missing partitions when needed. This level offers the best performance but no fallback if memory is insufficient. -
MEMORY_ONLY_2:
Similar to MEMORY_ONLY, but each partition is replicated to a second node. This adds fault tolerance in case one node goes down. -
MEMORY_ONLY_SER:
Stores the data in memory in a serialized format instead of raw objects. This saves space and allows larger datasets to fit in memory, though accessing the data becomes a bit slower due to the need for deserialization. -
MEMORY_ONLY_SER_2:
Same as MEMORY_ONLY_SER but with replication to two nodes for fault tolerance. -
MEMORY_AND_DISK:
Tries to store the data in memory first. If it doesn’t all fit, the remaining partitions are written to disk instead of being recomputed. This is useful when you're working with large datasets that partially fit in memory. -
MEMORY_AND_DISK_2:
Works like MEMORY_AND_DISK but replicates the partitions to another node for higher availability. -
MEMORY_AND_DISK_SER:
Stores the data in a serialized format in memory. Any overflow goes to disk, also in serialized form. This helps save memory and still avoids recomputation. -
MEMORY_AND_DISK_SER_2:
Same as MEMORY_AND_DISK_SER with replication added for fault tolerance. -
DISK_ONLY:
Stores the data only on disk. This uses no memory, so it's useful when memory is limited. However, reading from disk is slower than memory. -
DISK_ONLY_2:
Like DISK_ONLY, but each partition is stored on two nodes. This makes the data more fault-tolerant in case of node failures.