这是indexloc提供的服务,不要输入任何密码
Skip to content

[FEAT]: embedder chunk and text-chunk is the same unit (character) #4454

@kalle07

Description

@kalle07

What would you like to see?

Image ALSO embedder chunk are character

its proved by two users, can you please mention "character" also there! like the emebdder-text-chunk

and maybe add a small calculation inernal that show up the estimated token (close to that numbers)
5000chars are ~1200token (I chose only 5000chars because that is one page)
chars / 4.2 ~ token

and if we are there "LLM chunk-size ... maybe mention there, that it must be setup in the provider, at least in LM-Studio model setup
THX

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions