org.ragna.comet.stream.extractors

Type members

Classlikes

abstract class StreamExtractor[A](val format: DataFormat, val inference: InferenceEngine, val concurrentItems: Int, val itemTimeout: Option[FiniteDuration])(implicit toRdfElement: A => RDFElement)

Abstract class for any extractor capable of reaching a data Stream and extracting data from it, then transform this data to RDF items for post processing. Implementations shall define how the initial Stream is pulled from a data source

Abstract class for any extractor capable of reaching a data Stream and extracting data from it, then transform this data to RDF items for post processing. Implementations shall define how the initial Stream is pulled from a data source

Type parameters:
A

Type of the items expected by the Stream

Value parameters:
concurrentItems

Maximum number of items to be extracted and parsed for RDF in parallel (set it to 1 for sequential execution, bear in mind that high values won't necessarily translate into performance improvements unless you know what you are doing)

format

Format of the RDF data arriving from the Stream, the Extractor expects all data items to share format

inference

Inference of the RDF data arriving from the Stream, the Extractor expects all data items to share inference

itemTimeout

Time that this extractor should wait without being fed any item before raising a StreamTimeoutException and stopping the Stream, this is specially useful for remote data sources (i.e.: Kafka streams) that may stop sending data and cause resource starvation if kept open but also to stop streams that have become stuck processing an item, etc. Set to None for no timeout

toRdfElement

Helper function for converting the incoming items of type A into RDFElements (implicit conversions provided in RDFElementImplicits)

Throws:
StreamTimeoutException

If the time between received items exceeds the one configured in itemTimeout

Note:

In the future, this class can be implemented for any FS2-compatible data source (files, in-memory structures, etc.)

Companion:
object
Source:
StreamExtractor.scala
private[stream] object StreamExtractor