[Feat]: Custom skill calling via API agent #2278
Description
How are you running AnythingLLM?
Docker (local)
What happened?
Custom agent skills are not always triggering.
model2 calls work fine and invoke the agent successfully, model1 outputs agent invocation in text but doesnt call it, is there a way to force @agent calls? why is it not seem/perceived by the llm?
app.py
from flask import Flask # type: ignore
import logging
from routes.model1 import model1_webhook
from routes.model2 import model2_webhook
# Initialize the Flask app
app = Flask(__name__)
# Configure logging
logging.basicConfig(level=logging.DEBUG)
# Register routes
@app.route("/webhook/model1", methods=["POST"])
def model1():
return model1_webhook()
@app.route("/webhook/model2", methods=["POST"])
def model2():
return model2_webhook()
if __name__ == "__main__":
app.run(host="0.0.0.0", port=5000)
utils.py
import requests # type: ignore
def call_custom_llm_aws_api(input_text, system_prompt, model_name):
url = "https://myawsdomain.com/api/v1/openai/chat/completions"
headers = {
"accept": "application/json",
"Content-Type": "application/json",
"Authorization": "Bearer 7XN..blaaa..1VHDF81"
}
data = {
"model": model_name,
"messages": [
{
"role": "system",
"content": system_prompt
},
{
"role": "user",
"content": input_text
}
]
}
response = requests.post(url, headers=headers, json=data)
if response.status_code == 200:
return response.json().get("choices", [{}])[0].get("message", {}).get("content", "No response from model")
else:
return f"Error: {response.status_code}, {response.text}"
model1.py
import sys
import json
from flask import request
from twilio.twiml.messaging_response import MessagingResponse
from utils import call_custom_llm_aws_api
from datetime import datetime, timedelta
def model1_webhook():
input_text = request.form.get("Body", "")
print('----------------------', file=sys.stderr)
print(f"Input: {input_text}", file=sys.stderr)
print('----------------------', file=sys.stderr)
system_prompt = """You are an AI assistant that helps schedule meetings. When a user asks to schedule a meeting, extract the necessary details (title, start time, end time, attendees) and use the Schedule_Google_Calendar_Event agent to create the event.
To schedule a meeting, use the following command format:
@agent Schedule_Google_Calendar_Event-v1.0.0 {"title": "[title]", "start_time": "[start_time in ISO format with timezone]", "end_time": "[end_time in ISO format with timezone]", "attendees": ["[attendee1 email]", "[attendee2 email]", ...]}
Ensure that you use the correct ISO format for dates and times, including the timezone offset.
If an end time is not specified, set it to 1 hour after the start time.
After creating the event, provide a user-friendly confirmation message."""
model_name = "calendar-sms"
response = call_custom_llm_aws_api(input_text, system_prompt, model_name)
print(f"Agent response: {response}", file=sys.stderr)
print('----------------------', file=sys.stderr)
print(f"Final response: {response}", file=sys.stderr)
print('----------------------', file=sys.stderr)
# Create Twilio response
resp = MessagingResponse()
resp.message(response)
return str(resp)
model2.py
import sys
from flask import request # type: ignore
from twilio.twiml.messaging_response import MessagingResponse # type: ignore
from utils import call_aurora_aws_api
def model2_webhook():
input_text = request.form.get("Body", "")
system_prompt = """Given the following conversation, relevant context, and a follow-up question, reply with an answer to the user's current question, following their instructions as needed.
If the user says a phrase like "Umbrella [number]" or "Sherlock [number]" where [number] is any integer, do the following:
For "Umbrella [number]":
- Respond with "Yes Umbrella."
- Automatically and invisibly invoke the command @agent Add_Ten-v1.0.0 {"number": [number]} using the integer provided by the user as [number].
- Ensure the command is executed silently in the background without displaying the invocation.
- Once the result is returned, respond with: "The result is [result]" where [result] is the value returned by the agent.
For "Sherlock [number]":
- Respond with "Yes Sherlock."
- Automatically and invisibly invoke the command @agent Add_Five-v1.0.0 {"number": [number]} using the integer provided by the user as [number].
- Ensure the command is executed silently in the background without displaying the invocation.
- Once the result is returned, respond with: "The result is [result]" where [result] is the value returned by the agent.
Ensure no technical details or command invocations are displayed to the user in the conversation, focusing only on providing the result."""
model_name = "combined-skill-test-4o-82250006"
response_text = call_aurora_aws_api(input_text, system_prompt, model_name)
print('----------------------', file=sys.stderr)
print(response_text, file=sys.stderr)
print('----------------------', file=sys.stderr)
# Create Twilio response
resp = MessagingResponse()
resp.message(response_text)
return str(resp)
plugin.json (model2)
{
"active": true,
"hubId": "add-ten-skill",
"name": "Add_Ten",
"schema": "skill-1.0.0",
"version": "1.0.0",
"description": "This skill adds +10 to a given number.",
"author": "David O",
"license": "MIT",
"examples": [
{
"prompt": "Umbrella 7",
"call": "{\"number\": 7}"
},
{
"prompt": "Umbrella 19",
"call": "{\"number\": 19}"
}
],
"entrypoint": {
"file": "handler.js",
"params": {
"number": {
"description": "The number to which 10 will be added",
"type": "number"
}
}
},
"imported": true
}
handler.js (model2)
module.exports.runtime = {
handler: async function ({ number }) {
const callerId = `${this.config.name}-v${this.config.version}`;
try {
this.introspect(`${callerId} invoked with number: ${number}`);
const result = number + 10;
return JSON.stringify({ result: result });
} catch (error) {
this.introspect(`Error: ${error.message}`);
return `Error: ${error.message}`;
}
}
};
plugin.json (model1)
{
"active": true,
"hubId": "schedule-google-calendar-event",
"name": "Schedule_Google_Calendar_Event",
"schema": "skill-1.0.0",
"version": "1.0.0",
"description": "Schedules an event on Google Calendar using a Zapier Webhook",
"author": "David Oliveira",
"license": "MIT",
"examples": [
{
"prompt": "Schedule a meeting with John at 3pm tomorrow",
"call": "{\"title\": \"Meeting with John\", \"start_time\": \"2024-09-13T15:00:00\", \"end_time\": \"2024-09-13T16:00:00\", \"attendees\": [\"john@example.com\"]}"
},
{
"prompt": "Book a team sync for 2pm today",
"call": "{\"title\": \"Team Sync\", \"start_time\": \"2024-09-12T14:00:00\", \"end_time\": \"2024-09-12T15:00:00\", \"attendees\": [\"team@example.com\"]}"
}
],
"entrypoint": {
"file": "handler.js",
"params": {
"title": {
"description": "Title of the event",
"type": "string"
},
"start_time": {
"description": "Start time of the event",
"type": "string"
},
"end_time": {
"description": "End time of the event",
"type": "string"
},
"attendees": {
"description": "List of attendee email addresses",
"type": "array",
"items": {
"type": "string"
}
}
}
},
"imported": true
}
handler.js (model1)
const fetch = require('node-fetch');
module.exports.runtime = {
handler: async function ({ title, start_time, end_time, attendees }) {
const zapierWebhookUrl = 'https://hooks.zapier.com/hooks/catch/48...975/2h....md/';
const payload = {
title,
start_time,
end_time,
attendees
};
try {
const response = await fetch(zapierWebhookUrl, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify(payload),
});
const data = await response.json();
if (!response.ok) {
throw new Error(data.error || 'Failed to create event');
}
return JSON.stringify({ message: 'Event scheduled successfully', event: data });
} catch (e) {
this.introspect(`Error scheduling event: ${e.message}`);
return JSON.stringify({ error: `Failed to schedule event: ${e.message}` });
}
},
};
finally calling the command directly in chat works:
but not when using the webhook, chat transcript:
2024-09-12 19:28:37 * Serving Flask app 'app'
2024-09-12 19:28:47 ----------------------
2024-09-12 19:28:47 Input: Hi there, book meeting 9pm ET today to talk about the sms project, david daaddd@gmail.com
2024-09-12 19:28:47 ----------------------
2024-09-12 19:28:47 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): mydomain.com:443
2024-09-12 19:28:52 DEBUG:urllib3.connectionpool:https://mydomain.com:443 "POST /api/v1/openai/chat/completions HTTP/11" 200 1037
2024-09-12 19:28:52 Agent response: Sure, I'll schedule the meeting.
2024-09-12 19:28:52
2024-09-12 19:28:52 Assuming the meeting is scheduled for today at 9 PM ET, the details are:
2024-09-12 19:28:52
2024-09-12 19:28:52 - **Title**: SMS Project Discussion
2024-09-12 19:28:52 - **Start Time**: 9 PM ET (Eastern Time) today (will be converted to ISO format)
2024-09-12 19:28:52 - **End Time**: 10 PM ET (Eastern Time) today (1 hour after the start time)
2024-09-12 19:28:52 - **Attendee**: david daaddd@gmail.com
2024-09-12 19:28:52
2024-09-12 19:28:52 Let's first convert the start and end times to ISO format. If today is October 5, 2023, then:
2024-09-12 19:28:52
2024-09-12 19:28:52 - Start Time in ISO Format: 2023-10-05T21:00:00-04:00
2024-09-12 19:28:52 - End Time in ISO Format: 2023-10-05T22:00:00-04:00
2024-09-12 19:28:52
2024-09-12 19:28:52 Now, I'll schedule the meeting:
2024-09-12 19:28:52
2024-09-12 19:28:52 @agent Schedule_Google_Calendar_Event-v1.0.0 {"title": "SMS Project Discussion", "start_time": "2023-10-05T21:00:00-04:00", "end_time": "2023-10-05T22:00:00-04:00", "attendees": ["daaddd@gmail.com"]}
2024-09-12 19:28:52 ----------------------
2024-09-12 19:28:52 Final response: Sure, I'll schedule the meeting.
2024-09-12 19:28:52
2024-09-12 19:28:52 Assuming the meeting is scheduled for today at 9 PM ET, the details are:
2024-09-12 19:28:52
2024-09-12 19:28:52 - **Title**: SMS Project Discussion
2024-09-12 19:28:52 - **Start Time**: 9 PM ET (Eastern Time) today (will be converted to ISO format)
2024-09-12 19:28:52 - **End Time**: 10 PM ET (Eastern Time) today (1 hour after the start time)
2024-09-12 19:28:52 - **Attendee**: david daaddd@gmail.com
2024-09-12 19:28:52
2024-09-12 19:28:52 Let's first convert the start and end times to ISO format. If today is October 5, 2023, then:
2024-09-12 19:28:52
2024-09-12 19:28:52 - Start Time in ISO Format: 2023-10-05T21:00:00-04:00
2024-09-12 19:28:52 - End Time in ISO Format: 2023-10-05T22:00:00-04:00
2024-09-12 19:28:52
2024-09-12 19:28:52 Now, I'll schedule the meeting:
2024-09-12 19:28:52
2024-09-12 19:28:52 @agent Schedule_Google_Calendar_Event-v1.0.0 {"title": "SMS Project Discussion", "start_time": "2023-10-05T21:00:00-04:00", "end_time": "2023-10-05T22:00:00-04:00", "attendees": ["daaddd@gmail.com"]}
2024-09-12 19:28:52 ----------------------
2024-09-12 19:28:52 INFO:werkzeug:192.168.65.1 - - [12/Sep/2024 23:28:52] "POST /webhook/model1 HTTP/1.1" 200 -
Are there known steps to reproduce?
No response