STORM

What do you like best?

It has a good architecture to define the topology. Good with distributed processing. Topologies can be easily defined. It has good performance compared to spark. Initially nimbus was single point of failure, now with latest releases its been fixed.

With storm ui we can view the topology.

Storm provides the good way to manage parallelism and also scale without affecting / restarting existing eco system, user just need to add the worker processes.

What do you dislike?

When topologies become complex its difficult to debug. Its difficult to visualize also. When we deploy in distributed environment it has issues with performance as tuples are transferred between the worker processes over network, need to be very careful with this aspect.

When number of worker processes increase it has issues with management of worker processes. Have to correlate logs to identify what went wrong.

It is difficult to deal with the number of threads which we can assign to each bolt.

Recommendations to others considering the product:

Use storm wise, otherwise leads to lot of performance issues

What problems are you solving with the product? What benefits have you realized?

We are having data streaming from connected devices. We handle the logs and also stats. We use storm to process them. enrich and push it to elastic searh. We have a topologies to read from kafka, decode the message, enrich through the topology and last one to put it to elastic search. In between we have some alert mechanism through which the conditions or rules are evaluated and forwarded to appropriate bolts.

We also have the mechanism to do some config process, where we handle cloud based watch and collect events from aws. Based on this some alarms are generated.