How to set KafkaSource end condition 'Load data until no new data arrives within a timeout period'?

you can query stream_microbatch_history table to see which are the offsets loaded recently for every partition if you use scheduler.

https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/KafkaIntegrationGuide/KafkaTables/stream_microbatch_history.htm

if you run it manually, once the copy completes, you can run the below query to know the offsets it loaded

select KafkaOffsets() over();

https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/KafkaIntegrationGuide/KafkaFunctions/KafkaOffsets.htm

You can use the duration parameter for KafkaSource to determine how long you specify max ingestion time see https://www.vertica.com/docs/12.0.x/HTML/Content/Authoring/KafkaIntegrationGuide/KafkaFunctions/KafkaSource.htm?#4

As SruthiA mentioned, the KakfOffsets() will give you last offsets consumed by last COPY KafkaSource command.

Should have made two questions.....

ad 1: Thanks for the hint with KafkaOffsets(). How do I set permision to use KafkaOffsets?

ad 2: Setting the max ingestion time is not 'Load data until no new data arrives within a timeout period', but rather 'Copy as much data as possible for a set duration of time'
see https://www.vertica.com/docs/11.1.x/HTML/Content/Authoring/KafkaIntegrationGuide/UsingCOPYwithKafka.htm?tocpath=Apache Kafka Integration|Consuming Data From Kafka|_____1#6

dbadmin user should be able to run kafkaoffsets().. is it failing for you?

for 2. I think setting eof-timeout-ms may help

https://www.vertica.com/docs/11.1.x/HTML/Content/Authoring/KafkaIntegrationGuide/UtilityOptions/SchedulerUtilityOptions.htm?tocpath=Apache Kafka Integration|vkconfig Script Options|_____2