Users and customers are sending their messages or reviews from their devices.
There are several such messages streaming from different users into the system.
We must first be able to ingest these messages in real time manner.
Further we should be able to processes every single message and take corrective action as needed.
The processes would include following;
b. find out the sentiment of the message [ positve, negative ] using IE (information extraction)
We need to train a model here for this
c. filter messages with negative sentiment and put them in separate stream for further action / processing
d. find out a definitive pattern and send such events matching the patter to another stream for further review / action.
The patter is as follows;
find this pattern in continuous sliding manner
f. set running stats for different attributes in the event such as uniquw count for users or min/max/avg/sdtdev/sum/kurt for amount spend etc.
g. set up reverse index for messages such that it can used for text search by the user
h. set up secondary indexes for several attributes that could be helpful in query and also internal stream joins/ filter etc.