Hello Support Team!
New version of kafka scheduler in Vertica 10 uses epoch for sorting records in stream_microbatch_history
SELECT * FROM (SELECT *, extract(milliseconds from last_batch_duration::interval second) as ld FROM "KAFKA".stream_microbatch_history LIMIT 1 OVER (PARTITION BY microbatch_id, target_schema, target_table, source_name, source_cluster, source_partition ORDER BY epoch DESC)) as lastframe ORDER BY microbatch_id, target_schema, target_table, source_name, source_cluster, source_partition
But if this table is replicated from another DB all records has the same epoch.
select epoch,count(*) FROM KAFKA.stream_microbatch_history group by 1 order by 1; epoch | count -------+--------- 48421 | 7552259
And then kafka scheduler estimates incorrect (almost random) last offset from stream_microbatch_history
Is there are any workaround besides of manually updating last records in stream_microbatch_history after replication to change epoch?
UPDATE KAFKA.stream_microbatch_history
SET batch_start = batch_start
, end_offset = end_offset
WHERE (
microbatch_id, target_schema, target_table, source_name, source_cluster, source_partition, batch_start, start_offset, end_offset
) IN (
SELECT microbatch_id, target_schema, target_table, source_name, source_cluster, source_partition, batch_start, start_offset, end_offset
FROM KAFKA.stream_microbatch_history LIMIT 1
OVER ( PARTITION BY
microbatch_id, target_schema, target_table, source_name, source_cluster, source_partition
ORDER BY batch_start DESC
)
)
Used version: Vertica Analytic Database v10.1.0-0
