这是indexloc提供的服务,不要输入任何密码
Skip to content

Conversation

@g-hano
Copy link
Contributor

@g-hano g-hano commented Feb 4, 2025

Overview

This PR adds Ollama support and enhances response parsing capabilities for different model outputs. Fixes #6

Adding Ollama support to ReAG by enhancing the ReagClient class to support local model inference and improving response parsing.

Changes Made

  1. Enhanced ReagClient initialization:

    • Added optional api_base parameter for Ollama support
    • Maintained backward compatibility with existing implementation
  2. Added response parsing improvements:

    • New _extract_think_content method to handle DeepSeek models' markdown responses
    • Added fallback parsing for responses using think tags
    • Enhanced error handling in the query method

Environment

  • Python: 3.11.3
  • ReAG version: 0.0.3
  • Ollama version: 0.3.1

Implementation Details

The implementation adds support for:

  • Custom API base URLs for Ollama endpoints
  • Parsing both JSON and tagged format responses
  • Maintaining existing functionality while adding Ollama support

Example Usage

from reag.client import ReagClient, Document

async with ReagClient(
    model="ollama/deepseek-r1:14b",
    api_base="http://localhost:11434") as client:
	docs = [
		Document(
			name="Superagent",
			content="Superagent is a workspace for AI-agents that learn, perform work, and collaborate.",
			metadata={
				"url": "https://superagent.sh",
				"source": "web",
			},
		),
	]
	response = await client.query("What is Superagent?", documents=docs)

Added Ollama support for DeepSeek models.
I added an example of how to use deepseek models with ollama in ReagClient
Added ollama in tool.poetry.dependencies
)
except json.JSONDecodeError:
print("Error: Could not parse response:", message_content)
content, reasoning, is_irrelevant = self._extract_think_content(message_content)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@g-hano why is this added in here? it should be in the try block?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just updated the code. Please let me know if you'd like to see any other improvements.

moved `self._extract_think_content()` inside try block
@Ualas
Copy link

Ualas commented Feb 6, 2025

@g-hano thanks for your PR. I've used your code to use DeepSeek-R1 with Ollama. It's outputting the Reasoning, but, not the Content. Does it work fine on your end? If I run it with o3-mini it outputs the Content correctly.

BTW, I had to make some changes in the pyproject.toml to make it work with my system-wide Ollama v 0.5.7
image

@Ualas
Copy link

Ualas commented Feb 6, 2025

Ok, I realized that the output of ollama/deepseek-r1* models doesn't exactly follow the structure at _extract_think_content function, by making the below edit, I was able to make it work.

def _extract_think_content(self, text: str) -> tuple[str, str, bool]:
    """
    Extract reasoning from the <think> block and use everything after the </think> tag as content.
    """
    # Extract reasoning from the <think>...</think> block.
    reasoning_match = re.search(r'<think>(.*?)</think>', text, flags=re.DOTALL)
    reasoning = reasoning_match.group(1).strip() if reasoning_match else ""
    
    # Find the position of the closing </think> tag.
    end_think_tag = "</think>"
    end_index = text.find(end_think_tag)
    
    # If found, content is everything after the </think> tag; otherwise, use the whole text.
    if end_index != -1:
        content = text[end_index + len(end_think_tag):].strip()
    else:
        content = text.strip()
    
    # You can decide what to do with is_irrelevant.
    # Here we simply set it to False, meaning the content is relevant.
    is_irrelevant = False

    return content, reasoning, is_irrelevant

@g-hano
Copy link
Contributor Author

g-hano commented Feb 6, 2025

Ok, I realized that the output of ollama/deepseek-r1* models doesn't exactly follow the structure at _extract_think_content function, by making the below edit, I was able to make it work.

def _extract_think_content(self, text: str) -> tuple[str, str, bool]:
    """
    Extract reasoning from the <think> block and use everything after the </think> tag as content.
    """
    # Extract reasoning from the <think>...</think> block.
    reasoning_match = re.search(r'<think>(.*?)</think>', text, flags=re.DOTALL)
    reasoning = reasoning_match.group(1).strip() if reasoning_match else ""
    
    # Find the position of the closing </think> tag.
    end_think_tag = "</think>"
    end_index = text.find(end_think_tag)
    
    # If found, content is everything after the </think> tag; otherwise, use the whole text.
    if end_index != -1:
        content = text[end_index + len(end_think_tag):].strip()
    else:
        content = text.strip()
    
    # You can decide what to do with is_irrelevant.
    # Here we simply set it to False, meaning the content is relevant.
    is_irrelevant = False

    return content, reasoning, is_irrelevant

But with your code the method always returns false for is_irrelevant. I had no troubles with my code actually so I copy pasted the client.py Im running to the repo. Should work now.

@homanp homanp merged commit 1a01d2a into superagent-ai:main Feb 6, 2025
@homanp
Copy link
Collaborator

homanp commented Feb 6, 2025

@g-hano thank you for your contribution

@homanp
Copy link
Collaborator

homanp commented Feb 6, 2025

This is released in v0.0.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

New Feature: Ollama support for reasoning models

3 participants