data_components

Helper components for data transformation such as embeddings and document splitting.

Functions

retriever_output_to_context_str(retriever_output)

The retrieved documents from one or multiple queries.

Classes

RetrieverOutputToContextStr([deduplicate])

Wrap on functional F.retriever_output_to_context_str

ToEmbeddings(embedder[, batch_size])

It transforms a Sequence of Chunks or Documents to a List of Embeddings.
class ToEmbeddings(embedder: Embedder, batch_size: int = 50)[source]

Bases: DataComponent

It transforms a Sequence of Chunks or Documents to a List of Embeddings.

It operates on a copy of the input data, and does not modify the input data.

class RetrieverOutputToContextStr(deduplicate: bool = False)[source]

Bases: DataComponent

Wrap on functional F.retriever_output_to_context_str

retriever_output_to_context_str(retriever_output: RetrieverOutput | List[RetrieverOutput], deduplicate: bool = False) → str[source]

The retrieved documents from one or multiple queries. Deduplicate is especially helpful when you used query expansion.