Hi, I'm building a stream pre-processor in Flink that sits upstream of Vertica and does some condition checking before inserting rows. The spec requires both univariate and multivariate rules. Univariate is easy enough, but I am not sure how to handle multivariate rules. Essentially what I need to do is access the state of another previously seen key in the Flink stream: that is, if Flink finds A = 1 in the stream, then next find B = 2 in the stream, and the condition to verify is (A == 1 && B == 2), I need a way to look back at the last seen value of A from the current record.
Right now I'm working around this using Redis to store state externally but this may not scale to the ~1M records per second required SLA. Has anyone used Flink's native state storage, which is apparently much faster than even Redis?
