Spark

What do you like best?

A number of spark features that fits for a variety of use cases are:

In-Memory computation

Processing large quantities of data (any format),beyond what can fit on a single machine, with high level easy to use api.

It is highly configurable and exposed at higher level than other frame works.

Lazy Evaluation,data as RDDs dataframes and datasets ,Dag, lineage graph

Allows us to write data transformations and ML algorithms in parallelizable , but relatively system agnostic.

Spark not only supports ‘Map' and ‘reduce'. It also supports SQL queries, Streaming data, Machine learning (ML), and Graph algorithms.

Spark provides a collection of libraries including SQL and DataFrames, MLlib for machine learning, GraphX, and Spark Streaming.

What do you dislike?

No Support for Real-time Processing

Problem with Small File

No File Management System

Manual Optimisation

The absence of an in-house file management system

No automatic optimisation process

Expensive in-memory operations

Back pressure causing lining up of data at the input and the output channel

Apache Spark does not have the required capability to handle this build-up of data implicitly, and thus this needs to be taken care of manually.

Recommendations to others considering the product:

Spark is best suited for uses cases where :

Large files of any format

quick computation

Any advanced analysis

Avoid its use when:

small files

Data science routine tasks as is slower comparatively in performance.

What problems are you solving with the product? What benefits have you realized?

Financial Data transformation to make it easy for large scale analysis.

Data streaming for Statistical analysis of data.

A number of industries are using spark for their use cases due to the benefits offered by it.

Benefits:

fast computation speed

one platform for transformation and advanced data analytics