Documentation
¶
Overview ¶
Package llms provides unified support for interacting with different Language Models (LLMs) from various providers. Designed with an extensible architecture, the package facilitates seamless integration of LLMs with a focus on modularity, encapsulation, and easy configurability.
The package includes the following subpackages for LLM providers: 1. Hugging Face: llms/huggingface/ 2. Local LLM: llms/local/ 3. OpenAI: llms/openai/ 4. Google AI: llms/googleai/ 5. Cohere: llms/cohere/
Each subpackage includes provider-specific LLM implementations and helper files for communication with supported LLM providers. The internal directories within these subpackages contain provider-specific client and API implementations.
The `llms.go` file contains the types and interfaces for interacting with different LLMs.
The `options.go` file provides various options and functions to configure the LLMs.
Index ¶
- Variables
- func CalculateMaxTokens(model, text string) int
- func CountTokens(model, text string) int
- func GenerateFromSinglePrompt(ctx context.Context, llm Model, prompt string, options ...CallOption) (string, error)
- func GetBufferString(messages []ChatMessage, humanPrefix string, aiPrefix string) (string, error)
- func GetModelContextSize(model string) int
- func ShowMessageContents(w io.Writer, msgs []MessageContent)
- type AIChatMessage
- type AudioContent
- type BinaryContent
- type CallOption
- func WithCandidateCount(c int) CallOption
- func WithFrequencyPenalty(frequencyPenalty float64) CallOption
- func WithFunctionCallBehavior(behavior FunctionCallBehavior) CallOption
- func WithFunctions(functions []FunctionDefinition) CallOption
- func WithJSONMode() CallOption
- func WithMaxLength(maxLength int) CallOption
- func WithMaxTokens(maxTokens int) CallOption
- func WithMetadata(metadata map[string]interface{}) CallOption
- func WithMinLength(minLength int) CallOption
- func WithModel(model string) CallOption
- func WithN(n int) CallOption
- func WithOptions(options CallOptions) CallOption
- func WithPresencePenalty(presencePenalty float64) CallOption
- func WithRepetitionPenalty(repetitionPenalty float64) CallOption
- func WithResponseMIMEType(responseMIMEType string) CallOption
- func WithSeed(seed int) CallOption
- func WithStopWords(stopWords []string) CallOption
- func WithStreamingFunc(streamingFunc func(ctx context.Context, chunk []byte) error) CallOption
- func WithTemperature(temperature float64) CallOption
- func WithToolChoice(choice any) CallOption
- func WithTools(tools []Tool) CallOption
- func WithTopK(topK int) CallOption
- func WithTopP(topP float64) CallOption
- type CallOptions
- type ChatMessage
- type ChatMessageModel
- type ChatMessageModelData
- type ChatMessageType
- type ContentChoice
- type ContentPart
- type ContentResponse
- type FunctionCall
- type FunctionCallBehavior
- type FunctionChatMessage
- type FunctionDefinition
- type FunctionReference
- type GenericChatMessage
- type HumanChatMessage
- type ImageURLContent
- type LLMdeprecated
- type MessageContent
- type Model
- type Named
- type PromptValue
- type SystemChatMessage
- type TextContent
- type Tool
- type ToolCall
- type ToolCallResponse
- type ToolChatMessage
- type ToolChoice
Constants ¶
This section is empty.
Variables ¶
var ErrUnexpectedChatMessageType = errors.New("unexpected chat message type")
ErrUnexpectedChatMessageType is returned when a chat message is of an unexpected type.
Functions ¶
func CalculateMaxTokens ¶
CalculateMaxTokens calculates the max number of tokens that could be added to a text.
func CountTokens ¶
CountTokens gets the number of tokens the text contains.
func GenerateFromSinglePrompt ¶
func GenerateFromSinglePrompt(ctx context.Context, llm Model, prompt string, options ...CallOption) (string, error)
GenerateFromSinglePrompt is a convenience function for calling an LLM with a single string prompt, expecting a single string response. It's useful for simple, string-only interactions and provides a slightly more ergonomic API than the more general llms.Model.GenerateContent.
func GetBufferString ¶
func GetBufferString(messages []ChatMessage, humanPrefix string, aiPrefix string) (string, error)
GetBufferString gets the buffer string of messages.
func GetModelContextSize ¶
GetModelContextSize gets the max number of tokens for a language model. If the model name isn't recognized the default value 2048 is returned.
func ShowMessageContents ¶
func ShowMessageContents(w io.Writer, msgs []MessageContent)
ShowMessageContents is a debugging helper for MessageContent.
Types ¶
type AIChatMessage ¶
type AIChatMessage struct {
// Content is the content of the message.
Content string `json:"content,omitempty"`
// FunctionCall represents the model choosing to call a function.
FunctionCall *FunctionCall `json:"function_call,omitempty"`
// ToolCalls represents the model choosing to call tools.
ToolCalls []ToolCall `json:"tool_calls,omitempty"`
// This field is only used with the deepseek-reasoner model and represents the reasoning contents of the assistant message before the final answer.
ReasoningContent string `json:"reasoning_content,omitempty"`
}
AIChatMessage is a message sent by an AI.
func (AIChatMessage) GetContent ¶
func (m AIChatMessage) GetContent() string
func (AIChatMessage) GetFunctionCall ¶
func (m AIChatMessage) GetFunctionCall() *FunctionCall
func (AIChatMessage) GetType ¶
func (m AIChatMessage) GetType() ChatMessageType
type AudioContent ¶
AudioContent is content with a base64 audio data with format tag ("wav" || "mp3").
func AudioPart ¶
func AudioPart(data []byte, format string) AudioContent
AudioPart creates a new AudioContent from the given base64 data string and format ("wav || "mp3") string.
func (AudioContent) MarshalJSON ¶
func (ac AudioContent) MarshalJSON() ([]byte, error)
func (AudioContent) String ¶
func (ac AudioContent) String() string
func (*AudioContent) UnmarshalJSON ¶
func (ac *AudioContent) UnmarshalJSON(data []byte) error
type BinaryContent ¶
BinaryContent is content holding some binary data with a MIME type.
func BinaryPart ¶
func BinaryPart(mime string, data []byte) BinaryContent
BinaryPart creates a new BinaryContent from the given MIME type (e.g. "image/png" and binary data).
func (BinaryContent) MarshalJSON ¶
func (bc BinaryContent) MarshalJSON() ([]byte, error)
func (BinaryContent) String ¶
func (bc BinaryContent) String() string
func (*BinaryContent) UnmarshalJSON ¶
func (bc *BinaryContent) UnmarshalJSON(data []byte) error
type CallOption ¶
type CallOption func(*CallOptions)
CallOption is a function that configures a CallOptions.
func WithCandidateCount ¶
func WithCandidateCount(c int) CallOption
WithCandidateCount specifies the number of response candidates to generate.
func WithFrequencyPenalty ¶
func WithFrequencyPenalty(frequencyPenalty float64) CallOption
WithFrequencyPenalty will add an option to set the frequency penalty for sampling.
func WithFunctionCallBehavior ¶
func WithFunctionCallBehavior(behavior FunctionCallBehavior) CallOption
WithFunctionCallBehavior will add an option to set the behavior to use when calling functions. Deprecated: Use WithToolChoice instead.
func WithFunctions ¶
func WithFunctions(functions []FunctionDefinition) CallOption
WithFunctions will add an option to set the functions to include in the request. Deprecated: Use WithTools instead.
func WithJSONMode ¶
func WithJSONMode() CallOption
WithJSONMode will add an option to set the response format to JSON. This is useful for models that return structured data.
func WithMaxLength ¶
func WithMaxLength(maxLength int) CallOption
WithMaxLength will add an option to set the maximum length of the generated text.
func WithMaxTokens ¶
func WithMaxTokens(maxTokens int) CallOption
WithMaxTokens specifies the max number of tokens to generate.
func WithMetadata ¶
func WithMetadata(metadata map[string]interface{}) CallOption
WithMetadata will add an option to set metadata to include in the request. The meaning of this field is specific to the backend in use.
func WithMinLength ¶
func WithMinLength(minLength int) CallOption
WithMinLength will add an option to set the minimum length of the generated text.
func WithModel ¶
func WithModel(model string) CallOption
WithModel specifies which model name to use.
func WithN ¶
func WithN(n int) CallOption
WithN will add an option to set how many chat completion choices to generate for each input message.
func WithPresencePenalty ¶
func WithPresencePenalty(presencePenalty float64) CallOption
WithPresencePenalty will add an option to set the presence penalty for sampling.
func WithRepetitionPenalty ¶
func WithRepetitionPenalty(repetitionPenalty float64) CallOption
WithRepetitionPenalty will add an option to set the repetition penalty for sampling.
func WithResponseMIMEType ¶
func WithResponseMIMEType(responseMIMEType string) CallOption
WithResponseMIMEType will add an option to set the ResponseMIMEType Currently only supported by googleai llms.
func WithSeed ¶
func WithSeed(seed int) CallOption
WithSeed will add an option to use deterministic sampling.
func WithStopWords ¶
func WithStopWords(stopWords []string) CallOption
WithStopWords specifies a list of words to stop generation on.
func WithStreamingFunc ¶
func WithStreamingFunc(streamingFunc func(ctx context.Context, chunk []byte) error) CallOption
WithStreamingFunc specifies the streaming function to use.
func WithTemperature ¶
func WithTemperature(temperature float64) CallOption
WithTemperature specifies the model temperature, a hyperparameter that regulates the randomness, or creativity, of the AI's responses.
func WithToolChoice ¶
func WithToolChoice(choice any) CallOption
WithToolChoice will add an option to set the choice of tool to use. It can either be "none", "auto" (the default behavior), or a specific tool as described in the ToolChoice type.
func WithTools ¶
func WithTools(tools []Tool) CallOption
WithTools will add an option to set the tools to use.
func WithTopK ¶
func WithTopK(topK int) CallOption
WithTopK will add an option to use top-k sampling.
func WithTopP ¶
func WithTopP(topP float64) CallOption
WithTopP will add an option to use top-p sampling.
type CallOptions ¶
type CallOptions struct {
// Model is the model to use.
Model string `json:"model"`
// CandidateCount is the number of response candidates to generate.
CandidateCount int `json:"candidate_count"`
// MaxTokens is the maximum number of tokens to generate.
MaxTokens int `json:"max_tokens"`
// Temperature is the temperature for sampling, between 0 and 1.
Temperature float64 `json:"temperature"`
// StopWords is a list of words to stop on.
StopWords []string `json:"stop_words"`
// StreamingFunc is a function to be called for each chunk of a streaming response.
// Return an error to stop streaming early.
StreamingFunc func(ctx context.Context, chunk []byte) error `json:"-"`
// TopK is the number of tokens to consider for top-k sampling.
TopK int `json:"top_k"`
// TopP is the cumulative probability for top-p sampling.
TopP float64 `json:"top_p"`
// Seed is a seed for deterministic sampling.
Seed int `json:"seed"`
// MinLength is the minimum length of the generated text.
MinLength int `json:"min_length"`
// MaxLength is the maximum length of the generated text.
MaxLength int `json:"max_length"`
// N is how many chat completion choices to generate for each input message.
N int `json:"n"`
// RepetitionPenalty is the repetition penalty for sampling.
RepetitionPenalty float64 `json:"repetition_penalty"`
// FrequencyPenalty is the frequency penalty for sampling.
FrequencyPenalty float64 `json:"frequency_penalty"`
// PresencePenalty is the presence penalty for sampling.
PresencePenalty float64 `json:"presence_penalty"`
// JSONMode is a flag to enable JSON mode.
JSONMode bool `json:"json"`
// Tools is a list of tools to use. Each tool can be a specific tool or a function.
Tools []Tool `json:"tools,omitempty"`
// ToolChoice is the choice of tool to use, it can either be "none", "auto" (the default behavior), or a specific tool as described in the ToolChoice type.
ToolChoice any `json:"tool_choice"`
// Function defitions to include in the request.
// Deprecated: Use Tools instead.
Functions []FunctionDefinition `json:"functions,omitempty"`
// FunctionCallBehavior is the behavior to use when calling functions.
//
// If a specific function should be invoked, use the format:
// `{"name": "my_function"}`
// Deprecated: Use ToolChoice instead.
FunctionCallBehavior FunctionCallBehavior `json:"function_call,omitempty"`
// Metadata is a map of metadata to include in the request.
// The meaning of this field is specific to the backend in use.
Metadata map[string]interface{} `json:"metadata,omitempty"`
// ResponseMIMEType MIME type of the generated candidate text.
// Supported MIME types are: text/plain: (default) Text output.
// application/json: JSON response in the response candidates.
ResponseMIMEType string `json:"response_mime_type,omitempty"`
}
CallOptions is a set of options for calling models. Not all models support all options.
type ChatMessage ¶
type ChatMessage interface {
// GetType gets the type of the message.
GetType() ChatMessageType
// GetContent gets the content of the message.
GetContent() string
}
ChatMessage represents a message in a chat.
type ChatMessageModel ¶
type ChatMessageModel struct {
Type string `bson:"type" json:"type"`
Data ChatMessageModelData `bson:"data" json:"data"`
}
func ConvertChatMessageToModel ¶
func ConvertChatMessageToModel(m ChatMessage) ChatMessageModel
ConvertChatMessageToModel Convert a ChatMessage to a ChatMessageModel.
func (ChatMessageModel) ToChatMessage ¶
func (c ChatMessageModel) ToChatMessage() ChatMessage
type ChatMessageModelData ¶
type ChatMessageType ¶
type ChatMessageType string
ChatMessageType is the type of chat message.
const ( // ChatMessageTypeAI is a message sent by an AI. ChatMessageTypeAI ChatMessageType = "ai" // ChatMessageTypeHuman is a message sent by a human. ChatMessageTypeHuman ChatMessageType = "human" // ChatMessageTypeSystem is a message sent by the system. ChatMessageTypeSystem ChatMessageType = "system" // ChatMessageTypeGeneric is a message sent by a generic user. ChatMessageTypeGeneric ChatMessageType = "generic" // ChatMessageTypeFunction is a message sent by a function. ChatMessageTypeFunction ChatMessageType = "function" // ChatMessageTypeTool is a message sent by a tool. ChatMessageTypeTool ChatMessageType = "tool" )
type ContentChoice ¶
type ContentChoice struct {
// Content is the textual content of a response
Content string
// StopReason is the reason the model stopped generating output.
StopReason string
// GenerationInfo is arbitrary information the model adds to the response.
GenerationInfo map[string]any
// FuncCall is non-nil when the model asks to invoke a function/tool.
// If a model invokes more than one function/tool, this field will only
// contain the first one.
FuncCall *FunctionCall
// ToolCalls is a list of tool calls the model asks to invoke.
ToolCalls []ToolCall
// This field is only used with the deepseek-reasoner model and represents the reasoning contents of the assistant message before the final answer.
ReasoningContent string
}
ContentChoice is one of the response choices returned by GenerateContent calls.
type ContentPart ¶
type ContentPart interface {
// contains filtered or unexported methods
}
ContentPart is an interface all parts of content have to implement.
type ContentResponse ¶
type ContentResponse struct {
Choices []*ContentChoice
}
ContentResponse is the response returned by a GenerateContent call. It can potentially return multiple content choices.
type FunctionCall ¶
type FunctionCall struct {
// The name of the function to call.
Name string `json:"name"`
// The arguments to pass to the function, as a JSON string.
Arguments string `json:"arguments"`
}
FunctionCall is the name and arguments of a function call.
type FunctionCallBehavior ¶
type FunctionCallBehavior string
FunctionCallBehavior is the behavior to use when calling functions.
const ( // FunctionCallBehaviorNone will not call any functions. FunctionCallBehaviorNone FunctionCallBehavior = "none" // FunctionCallBehaviorAuto will call functions automatically. FunctionCallBehaviorAuto FunctionCallBehavior = "auto" )
type FunctionChatMessage ¶
type FunctionChatMessage struct {
// Name is the name of the function.
Name string `json:"name"`
// Content is the content of the function message.
Content string `json:"content"`
}
FunctionChatMessage is a chat message representing the result of a function call. Deprecated: Use ToolChatMessage instead.
func (FunctionChatMessage) GetContent ¶
func (m FunctionChatMessage) GetContent() string
func (FunctionChatMessage) GetName ¶
func (m FunctionChatMessage) GetName() string
func (FunctionChatMessage) GetType ¶
func (m FunctionChatMessage) GetType() ChatMessageType
type FunctionDefinition ¶
type FunctionDefinition struct {
// Name is the name of the function.
Name string `json:"name"`
// Description is a description of the function.
Description string `json:"description"`
// Parameters is a list of parameters for the function.
Parameters any `json:"parameters,omitempty"`
// Strict is a flag to indicate if the function should be called strictly. Only used for openai llm structured output.
Strict bool `json:"strict,omitempty"`
}
FunctionDefinition is a definition of a function that can be called by the model.
type FunctionReference ¶
type FunctionReference struct {
// Name is the name of the function.
Name string `json:"name"`
}
FunctionReference is a reference to a function.
type GenericChatMessage ¶
GenericChatMessage is a chat message with an arbitrary speaker.
func (GenericChatMessage) GetContent ¶
func (m GenericChatMessage) GetContent() string
func (GenericChatMessage) GetName ¶
func (m GenericChatMessage) GetName() string
func (GenericChatMessage) GetType ¶
func (m GenericChatMessage) GetType() ChatMessageType
type HumanChatMessage ¶
type HumanChatMessage struct {
Content string
}
HumanChatMessage is a message sent by a human.
func (HumanChatMessage) GetContent ¶
func (m HumanChatMessage) GetContent() string
func (HumanChatMessage) GetType ¶
func (m HumanChatMessage) GetType() ChatMessageType
type ImageURLContent ¶
type ImageURLContent struct {
URL string `json:"url"`
Detail string `json:"detail,omitempty"` // Detail is the detail of the image, e.g. "low", "high".
}
ImageURLContent is content with an URL pointing to an image.
func ImageURLPart ¶
func ImageURLPart(url string) ImageURLContent
ImageURLPart creates a new ImageURLContent from the given URL.
func ImageURLWithDetailPart ¶
func ImageURLWithDetailPart(url string, detail string) ImageURLContent
ImageURLWithDetailPart creates a new ImageURLContent from the given URL and detail.
func (ImageURLContent) MarshalJSON ¶
func (iuc ImageURLContent) MarshalJSON() ([]byte, error)
func (ImageURLContent) String ¶
func (iuc ImageURLContent) String() string
func (*ImageURLContent) UnmarshalJSON ¶
func (iuc *ImageURLContent) UnmarshalJSON(data []byte) error
type MessageContent ¶
type MessageContent struct {
Role ChatMessageType
Parts []ContentPart
}
MessageContent is the content of a message sent to a LLM. It has a role and a sequence of parts. For example, it can represent one message in a chat session sent by the user, in which case Role will be ChatMessageTypeHuman and Parts will be the sequence of items sent in this specific message.
func TextParts ¶
func TextParts(role ChatMessageType, parts ...string) MessageContent
TextParts is a helper function to create a MessageContent with a role and a list of text parts.
func (MessageContent) MarshalJSON ¶
func (mc MessageContent) MarshalJSON() ([]byte, error)
func (*MessageContent) UnmarshalJSON ¶
func (mc *MessageContent) UnmarshalJSON(data []byte) error
type Model ¶
type Model interface {
// GenerateContent asks the model to generate content from a sequence of
// messages. It's the most general interface for multi-modal LLMs that support
// chat-like interactions.
GenerateContent(ctx context.Context, messages []MessageContent, options ...CallOption) (*ContentResponse, error)
// Call is a simplified interface for a text-only Model, generating a single
// string response from a single string prompt.
//
// Deprecated: this method is retained for backwards compatibility. Use the
// more general [GenerateContent] instead. You can also use
// the [GenerateFromSinglePrompt] function which provides a similar capability
// to Call and is built on top of the new interface.
Call(ctx context.Context, prompt string, options ...CallOption) (string, error)
}
Model is an interface multi-modal models implement.
type Named ¶
type Named interface {
GetName() string
}
Named is an interface for objects that have a name.
type PromptValue ¶
type PromptValue interface {
String() string
Messages() []ChatMessage
}
PromptValue is the interface that all prompt values must implement.
type SystemChatMessage ¶
type SystemChatMessage struct {
Content string
}
SystemChatMessage is a chat message representing information that should be instructions to the AI system.
func (SystemChatMessage) GetContent ¶
func (m SystemChatMessage) GetContent() string
func (SystemChatMessage) GetType ¶
func (m SystemChatMessage) GetType() ChatMessageType
type TextContent ¶
type TextContent struct {
Text string
}
TextContent is content with some text.
func TextPart ¶
func TextPart(s string) TextContent
TextPart creates TextContent from a given string.
func (TextContent) MarshalJSON ¶
func (tc TextContent) MarshalJSON() ([]byte, error)
func (TextContent) String ¶
func (tc TextContent) String() string
func (*TextContent) UnmarshalJSON ¶
func (tc *TextContent) UnmarshalJSON(data []byte) error
type Tool ¶
type Tool struct {
// Type is the type of the tool.
Type string `json:"type"`
// Function is the function to call.
Function *FunctionDefinition `json:"function,omitempty"`
}
Tool is a tool that can be used by the model.
type ToolCall ¶
type ToolCall struct {
// ID is the unique identifier of the tool call.
ID string `json:"id"`
// Type is the type of the tool call. Typically, this would be "function".
Type string `json:"type"`
// FunctionCall is the function call to be executed.
FunctionCall *FunctionCall `json:"function,omitempty"`
}
ToolCall is a call to a tool (as requested by the model) that should be executed.
func (ToolCall) MarshalJSON ¶
func (*ToolCall) UnmarshalJSON ¶
type ToolCallResponse ¶
type ToolCallResponse struct {
// ToolCallID is the ID of the tool call this response is for.
ToolCallID string `json:"tool_call_id"`
// Name is the name of the tool that was called.
Name string `json:"name"`
// Content is the textual content of the response.
Content string `json:"content"`
}
ToolCallResponse is the response returned by a tool call.
func (ToolCallResponse) MarshalJSON ¶
func (tc ToolCallResponse) MarshalJSON() ([]byte, error)
func (*ToolCallResponse) UnmarshalJSON ¶
func (tc *ToolCallResponse) UnmarshalJSON(data []byte) error
type ToolChatMessage ¶
type ToolChatMessage struct {
// ID is the ID of the tool call.
ID string `json:"tool_call_id"`
// Content is the content of the tool message.
Content string `json:"content"`
}
ToolChatMessage is a chat message representing the result of a tool call.
func (ToolChatMessage) GetContent ¶
func (m ToolChatMessage) GetContent() string
func (ToolChatMessage) GetID ¶
func (m ToolChatMessage) GetID() string
func (ToolChatMessage) GetType ¶
func (m ToolChatMessage) GetType() ChatMessageType
type ToolChoice ¶
type ToolChoice struct {
// Type is the type of the tool.
Type string `json:"type"`
// Function is the function to call (if the tool is a function).
Function *FunctionReference `json:"function,omitempty"`
}
ToolChoice is a specific tool to use.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package cache provides a generic wrapper that adds caching to a `llms.Model`.
|
Package cache provides a generic wrapper that adds caching to a `llms.Model`. |
|
Package ernie wrapper around the Baidu Large Language Model Platform APIs.
|
Package ernie wrapper around the Baidu Large Language Model Platform APIs. |
|
package googleai implements a langchaingo provider for Google AI LLMs.
|
package googleai implements a langchaingo provider for Google AI LLMs. |
|
internal/cmd
command
Code generator for vertex.go from googleai.go nolint
|
Code generator for vertex.go from googleai.go nolint |
|
palm
package palm implements a langchaingo provider for Google Vertex AI legacy PaLM models.
|
package palm implements a langchaingo provider for Google Vertex AI legacy PaLM models. |
|
vertex
package vertex implements a langchaingo provider for Google Vertex AI LLMs, including the new Gemini models.
|
package vertex implements a langchaingo provider for Google Vertex AI LLMs, including the new Gemini models. |
|
internal/localclient
Package localclient provides a client for local LLMs.
|
Package localclient provides a client for local LLMs. |