StreamExtractor capable of extracting RDF items from an Apache Kafka stream of items
Kafka streams are operated as FS2 streams thanks to the library FS2-Kafka
- Type parameters:
- K
Type to which kafka-stream keys are deserialized
- V
Type to which kafka-stream values are deserialized
- Value parameters:
- concurrentItems
Maximum number of items to be parsed for RDF in parallel (to parallelize the pulling of items from Kafka, see configuration.concurrentItems)
- configuration
Configuration supplied to this extractor to interact with a Kafka stream
- format
Format of the RDF data arriving from the Stream, the Extractor expects all data items to share format
- inference
Inference of the RDF data arriving from the Stream, the Extractor expects all data items to share inference
- keyDeserializer
Deserializer used to map the incoming stream keys to instances of type K
- valueDeserializer
Deserializer used to map the incoming stream keys to instances of type V
- Throws:
- KafkaException
When the underlying Kafka consumer cannot be built or the connection fails
- See also:
Apache Kafka: https://kafka.apache.org/
FS2-Kafka: https://fd4s.github.io/fs2-kafka/
- Note:
- Companion:
- object
- Source:
- KafkaExtractor.scala
Type members
Value members
Concrete methods
Inherited methods
Concrete fields
Settings for the KafkaConsumer pulling items from the Kafka stream, build from this extractor's configuration
Settings for the KafkaConsumer pulling items from the Kafka stream, build from this extractor's configuration
- Source:
- KafkaExtractor.scala
FS2 representation of the input Stream coming from Kafka, with its items being forcibly turned to Strings whatever their type (see class notes)
FS2 representation of the input Stream coming from Kafka, with its items being forcibly turned to Strings whatever their type (see class notes)
Subscribes to the input topic to extract records, which are processed in parallel although results are still emitted according to the order of incoming elements
- Source:
- KafkaExtractor.scala
Inherited fields
The initial inputStream, transformed through toDataItems to get a stream of RDF Items
The initial inputStream, transformed through toDataItems to get a stream of RDF Items
- Inherited from:
- StreamExtractor
- Source:
- StreamExtractor.scala