KafkaExtractorConfiguration

sealed case class KafkaExtractorConfiguration(topic: String, server: String, port: Int, groupId: String, offsetReset: AutoOffsetReset, autoCommit: Boolean, sessionTimeout: FiniteDuration, commitRecovery: CommitRecovery, commitTimeout: FiniteDuration, commitInterval: FiniteDuration, pollTimeout: FiniteDuration, pollInterval: FiniteDuration, concurrentItems: Int)

Configuration and parameters related to how the program should interact with a Kafka stream (stream origin location, committing, polling, etc.)

Defaults are provided for most data, yet a target topic is required

Value parameters:
autoCommit

Whether to automatically commit offsets periodically (see commitInterval) or not

commitInterval

Time to wait before performing an offset commit operation

commitRecovery

Strategy to be followed when a commit operation fails

commitTimeout

Time to wait for offsets to be committed before raising an error

concurrentItems

Maximum number of items to be extracted and parsed for RDF in parallel (set it to 1 for sequential execution, bear in mind that high values won't necessarily translate into performance improvements unless you know what you are doing)

groupId

Id of the group to which the underlying Kafka consumer belongs

offsetReset

Strategy to be followed when there are no available offsets to consume on the consumer group

pollInterval

Time interval between poll operations

pollTimeout

How long the an stream polling operation is allowed to block

port

Port from which server is broadcasting

server

Hostname or IP address of the server broadcasting data

sessionTimeout

Consumer timeout for connecting to Kafka to or processing incoming items, some Kafka servers may limit require certain session timeouts

topic

Name of the topic from which data will be extracted

Note:

Commit-behaviour settings are meaningless if autoCommit is set to false

Some restrictions are in place for configuration instances, and invalid values for the configuration data (e.g.: invalid port, instant timeouts/intervals) will not be accepted (see testConfiguration)

Companion:
object
Source:
KafkaExtractorConfiguration.scala
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any

Value members

Concrete methods

private[kafka] def testConfiguration(): Unit

Perform several checks to make sure that this configuration:

Perform several checks to make sure that this configuration:

  • Contains valid inputs (e.g.: target port is in range)
  • Contains reasonable inputs (e.g.: time intervals or timeouts durations make sense)
Throws:
IllegalArgumentException

On invalid configuration parameters

Note:

Testing the configuration only available from the KafkaExtractor and related locations since it raises errors that need management

Source:
KafkaExtractorConfiguration.scala

Inherited methods

def productElementNames: Iterator[String]
Inherited from:
Product
def productIterator: Iterator[Any]
Inherited from:
Product

Concrete fields

val bootstrapServers: String

Kafka's target bootstrap servers, made by chaining the target host and the target port

Kafka's target bootstrap servers, made by chaining the target host and the target port

Source:
KafkaExtractorConfiguration.scala