Azure Databricks

What do you like best?

Databricks has perfect integration with Azure family products, let it be ADLS storage account or Authentication mechanism, where the team not bother about security and can concentrate more on problem-solving.

easy to mount and unmount storage account for larger data processing where we can reduce the time of lift and shift the data from given region and follows data privacy and legal rules.

Auto Scale feature helps team not to worry about performance of ETL irrespective of the data load.

Pay-as-you-go helps to reduce the cost of overall ETL considering only nodes which got spin-up during ETL.

Integration with Azure Key-vault helps Databricks to fetch sensitive information and Databricks manages very effectively without exposing to end-user.

What do you dislike?

Databricks Notebook doesn't have auto-complete feature with python or scala API. user should know function in advance before use.

For java, end users have to run with jar only, which is not an interactive way while during development.

What problems are you solving with the product? What benefits have you realized?

We are processing DICOM images for cancel patient to figure out how much cell cancer has damaged or the dose with which it can be cured.

in Medical terms, contouring takes hours to detect co-ordinate where the tumor is , with Azure Databricks we made Terabytes of data available to users to predict model and tune the accuracy.