Spark SQL

What do you like best?

1. We can write most of the SQL scripts to read the massive data

2. All kinds of SQL features are supported

3. The best feature I like is to use the SQL in python language and store the extract the data in python data frames

4. SQL views are very comfortable way to create access the data from data lakes

5. Since Spark SQL uses the Spark clusters it is very fast to run the data extraction related queries

6. Notebooks are more easy and comfortable way to write the code

What do you dislike?

Only problem is create tables or views on top of the JSON data from data lake. And to setup the external meta store instead of upgrading the current metastore

What problems are you solving with the product? What benefits have you realized?

1. Fast running jobs

2. Easier to deal with massive data

3. Easy for new users since it is mostly basic SQL

4. Integration with multiple third party software

5. Easy integration with different reporting tools