Apache Flink

What do you like best?

It supports both stateful and stateless computations on streams

Supports both batch mode and real-time analytics

Has proven to be high performing, less memory-hungry compared to the storm

It has capability to do windowing, machine learning integration etc.,

It is highly scalable

It also has capability to process event may be do aggregation or windowing based on event occurrence time than the processing time

Has Exactly-once state consistency

Also supports handling late data through some threshold window

Can do in memory SQL on streams

Flink UI is very user friendly

What do you dislike?

There is not much to dislike. It has capabilities of both storm and spark. If you know storm and spark it's easy to use

Recommendations to others considering the product:

Use it wisely, tune the memory parameters and parallelism wisely. Otherwise you end up back pressure or under utilising the resources

Lot of tuning with respect to num of threads and memory allotment is required

do not overwrite the processors, which will lead to a lot of parallelism and simply data transfer between the nodes and can lead to slow down

Need to archirect cautiously

What problems are you solving with the product? What benefits have you realized?

We use flink for both online streaming and offline batch processing

Mainly to enrich the incoming data, integrated with elastic search to store it. We also do aggregation using tumbling winfdow. We use flink views

For batch processing, we do use to learn some thresholds, like cpu, memory thresholds etc.,

Deployed with 100's of nodes, highly scalable

deployed in aws using kubernetese container

We also use flink ui to debug high level issues

We dont do sql on streaming