Configuration and parameters related to how the program should interact with a Kafka stream (stream origin location, committing, polling, etc.)
Defaults are provided for most data, yet a target topic is required
- Value parameters:
- autoCommit
Whether to automatically commit offsets periodically (see commitInterval) or not
- commitInterval
Time to wait before performing an offset commit operation
- commitRecovery
Strategy to be followed when a commit operation fails
- commitTimeout
Time to wait for offsets to be committed before raising an error
- concurrentItems
Maximum number of items to be extracted and parsed for RDF in parallel (set it to 1 for sequential execution, bear in mind that high values won't necessarily translate into performance improvements unless you know what you are doing)
- groupId
Id of the group to which the underlying Kafka consumer belongs
- offsetReset
Strategy to be followed when there are no available offsets to consume on the consumer group
- pollInterval
Time interval between poll operations
- pollTimeout
How long the an stream polling operation is allowed to block
- port
Port from which server is broadcasting
- server
Hostname or IP address of the server broadcasting data
- sessionTimeout
Consumer timeout for connecting to Kafka to or processing incoming items, some Kafka servers may limit require certain session timeouts
- topic
Name of the topic from which data will be extracted
- Note:
Commit-behaviour settings are meaningless if autoCommit is set to false
Some restrictions are in place for configuration instances, and invalid values for the configuration data (e.g.: invalid port, instant timeouts/intervals) will not be accepted (see testConfiguration)
- Companion:
- object
- Source:
- KafkaExtractorConfiguration.scala
Value members
Concrete methods
Perform several checks to make sure that this configuration:
Perform several checks to make sure that this configuration:
- Contains valid inputs (e.g.: target port is in range)
- Contains reasonable inputs (e.g.: time intervals or timeouts durations make sense)
- Throws:
- IllegalArgumentException
On invalid configuration parameters
- Note:
Testing the configuration only available from the KafkaExtractor and related locations since it raises errors that need management
- Source:
- KafkaExtractorConfiguration.scala