With several options available for big data frameworks, it is hard to choose the right one for your project.

# Apache Spark

**Apache Spark** is an open-source, distributed, general-purpose, cluster-computing framework. It is the largest open-source project in data processing. Spark promises excellent performance and comes packaged with higher-level libraries, including support for SQL queries, streaming data, machine learning, and graph processing.


# About Hadoop MapReduce

**Hadoop MapReduce** is a software framework for conveniently writing applications that process vast amounts of data (multi-terabyte data-sets) in-parallel or on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.



# Spark vs. MapReduce

Both Spark and Hadoop are both open source projects by Apache Software Foundation and flagship products in big data analytics. Both engines support nearly the same data sources and file formats, and both are scalable and limited to 100 nodes in a single cluster.

A key difference lies in the approach to processing: Spark can do it in-memory, while MapReduce has to read from and write to a disk. This results in significant performance differences -- Spark can be up to 100 times faster than MapReduce. However, processing in-memory limits the amount of data you can process. MapReduce is able to work with far larger data sets than Spark.  

The key differences between the two are highlighted below.



|                        Spark                        |                          MapReduce                         |
|:---------------------------------------------------:|:----------------------------------------------------------:|
| For batch as well as real time data processing.    | Only for batch data processing.                            |
| Up to 100x faster in-memory and 10x faster on disk. | Slower due to disk latency.                                |
| Requires large amounts of memory.                   | Does not need large amounts of memory.                     |
| Built-in APIs for machine learning.                 | Need to integrate with Apache Mahout for machine learning. |
| Less mature so comparatively less secure.           | More mature and highly secure.                             |
| Easier to use with a set of rich APIs.              | Harder to use and comparatively more complex.              |

Spark	MapReduce
For batch as well as real time data processing.	Only for batch data processing.
Up to 100x faster in-memory and 10x faster on disk.	Slower due to disk latency.
Requires large amounts of memory.	Does not need large amounts of memory.
Built-in APIs for machine learning.	Need to integrate with Apache Mahout for machine learning.
Less mature so comparatively less secure.	More mature and highly secure.
Easier to use with a set of rich APIs.	Harder to use and comparatively more complex.