KafkaExtractor

case class KafkaExtractor[K, V](configuration: KafkaExtractorConfiguration, format: DataFormat, inference: InferenceEngine, concurrentItems: Int, itemTimeout: Option[FiniteDuration])(implicit keyDeserializer: Deserializer[IO, K], valueDeserializer: Deserializer[IO, V], toRdfElement: V => RDFElement) extends StreamExtractor[V]

StreamExtractor capable of extracting RDF items from an Apache Kafka stream of items

Kafka streams are operated as FS2 streams thanks to the library FS2-Kafka

Type parameters:
K

Type to which kafka-stream keys are deserialized

V

Type to which kafka-stream values are deserialized

Value parameters:
concurrentItems

Maximum number of items to be parsed for RDF in parallel (to parallelize the pulling of items from Kafka, see configuration.concurrentItems)

configuration

Configuration supplied to this extractor to interact with a Kafka stream

format

Format of the RDF data arriving from the Stream, the Extractor expects all data items to share format

inference

Inference of the RDF data arriving from the Stream, the Extractor expects all data items to share inference

keyDeserializer

Deserializer used to map the incoming stream keys to instances of type K

valueDeserializer

Deserializer used to map the incoming stream keys to instances of type V

Throws:
KafkaException

When the underlying Kafka consumer cannot be built or the connection fails

See also:
Note:

Although incoming stream items can be deserialized into any format V (as long as a KafkaDeserializer is provided for type V), they'll eventually be converted to RDFElements in order to parse RDF from them

Companion:
object
Source:
KafkaExtractor.scala
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any

Type members

Types

private type K
private type V

Value members

Concrete methods

override protected def checkConfiguration(): Unit
Definition Classes
Source:
KafkaExtractor.scala

Inherited methods

def productElementNames: Iterator[String]
Inherited from:
Product
def productIterator: Iterator[Any]
Inherited from:
Product

Concrete fields

private val consumerSettings: ConsumerSettings[IO, K, V]

Settings for the KafkaConsumer pulling items from the Kafka stream, build from this extractor's configuration

Settings for the KafkaConsumer pulling items from the Kafka stream, build from this extractor's configuration

Source:
KafkaExtractor.scala
lazy override private[extractors] val inputStream: Stream[IO, V]

FS2 representation of the input Stream coming from Kafka, with its items being forcibly turned to Strings whatever their type (see class notes)

FS2 representation of the input Stream coming from Kafka, with its items being forcibly turned to Strings whatever their type (see class notes)

Subscribes to the input topic to extract records, which are processed in parallel although results are still emitted according to the order of incoming elements

Source:
KafkaExtractor.scala

Inherited fields

lazy val dataStream: Stream[IO, RDFValidationItem]

The initial inputStream, transformed through toDataItems to get a stream of RDF Items

The initial inputStream, transformed through toDataItems to get a stream of RDF Items

Inherited from:
StreamExtractor
Source:
StreamExtractor.scala

Implicits

Implicits

implicit private val keyDeserializer: Deserializer[IO, K]
implicit private val toRdfElement: V => RDFElement
implicit private val valueDeserializer: Deserializer[IO, V]