ai: the /api/chat agent loop — SSE streaming, modes, tool execution
Tool definitions are generated in-process from the /api/ai blueprint's
own OpenAPI operations (operationId = tool name, description = tool
description, dataclass bodies + declared parameters = one flat JSON
schema) — the spec IS the catalog, nothing to drift. Manage endpoints
never become tools. A mode is a server-enforced system prompt + tool
allowlist (Help / Research / System per ai.md); the executor can only
reach allowlisted /api/ai endpoints, path parameters are encoded and
traversal-rejected so a crafted argument cannot escape an endpoint.
POST /api/chat streams SSE (status/token/tool_call/tool_result/done/
error), starts llama-server on demand, validates tool args server-side
with instructive errors the model can retry on, caps tool results with
a paging hint, and bounds the loop by ai.yaml max_tool_calls plus a
hard timeout. Caddy passes the stream unbuffered (flush_interval -1)
in both deployments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 21:47:50 -06:00
|
|
|
"""POST /api/chat — the agent loop, streamed over SSE.
|
|
|
|
|
|
|
|
|
|
Events (each `event: <name>` with a JSON `data:` line):
|
|
|
|
|
status {message} e.g. "Loading the model…"
|
|
|
|
|
token {content} one streamed answer fragment
|
|
|
|
|
tool_call {tool, args} the model is using a tool
|
|
|
|
|
tool_result {tool, success} the tool finished
|
|
|
|
|
done {content} the complete final answer
|
|
|
|
|
error {message}
|
|
|
|
|
|
|
|
|
|
The loop: build the mode's system prompt and tool definitions -> stream from llama-server ->
|
|
|
|
|
execute tool calls against /api/ai only (the mode allowlist is enforced server-side) -> repeat,
|
|
|
|
|
bounded by ai.yaml's max_tool_calls -> stream the final answer.
|
|
|
|
|
|
|
|
|
|
A streaming response outlives the request middleware, so nothing here may use
|
|
|
|
|
request.ctx.session; tools execute through the blueprint, which manages its own sessions."""
|
|
|
|
|
import dataclasses
|
|
|
|
|
import json
|
|
|
|
|
import time
|
|
|
|
|
from http import HTTPStatus
|
|
|
|
|
from typing import AsyncGenerator, List, Optional
|
|
|
|
|
|
|
|
|
|
from sanic import Blueprint, Request
|
|
|
|
|
from sanic_ext import validate
|
|
|
|
|
from sanic_ext.extensions.openapi import openapi
|
|
|
|
|
|
|
|
|
|
from wrolpi.common import aiohttp_post, logger
|
|
|
|
|
from wrolpi.errors import APIError, ValidationError
|
|
|
|
|
|
|
|
|
|
from . import agent, service
|
ai: ground answers in a live library map, budget the context
Response-quality work, all aimed at giving the model enough context to
answer accurately:
- Every chat's system prompt now carries a live library map (today's
date, counts, and the exact names of channels, domains, zims, and
inventories, capped and cached 60s). Exact spellings are in context
before the first token, and the list_collections round-trip — minutes
on slow hardware — is usually unnecessary.
- The request is budgeted against the model's real context: oldest
turns are trimmed first, the current question is never dropped, and
an over-budget turn gets a clear "start a new conversation" error
instead of a silently truncated prompt. cache_prompt reuses the KV
cache for the stable prefix, which is most of the latency on a Pi.
- context_size plumbing fixed: selecting a model adopts its catalog
default (the 4B was running at 8k instead of 16k), and llama-server
runs --parallel 1 so the single user gets the whole context.
- temperature is configurable in ai.yaml (default 0.3 — small models
call tools far more reliably near 0.2-0.3 than at server defaults).
- One worked tool-use example per mode is prepended to every
conversation; small models imitate far better than they obey.
- scripts/ai_eval.py scores the assistant against a live WROLPi on
mechanical checks (finished, called the right tool first, included a
WROLPi link, no external URLs) so prompt changes stop being validated
by anecdote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:48:19 -06:00
|
|
|
from .catalog import get_effective_context_size
|
ai: the /api/chat agent loop — SSE streaming, modes, tool execution
Tool definitions are generated in-process from the /api/ai blueprint's
own OpenAPI operations (operationId = tool name, description = tool
description, dataclass bodies + declared parameters = one flat JSON
schema) — the spec IS the catalog, nothing to drift. Manage endpoints
never become tools. A mode is a server-enforced system prompt + tool
allowlist (Help / Research / System per ai.md); the executor can only
reach allowlisted /api/ai endpoints, path parameters are encoded and
traversal-rejected so a crafted argument cannot escape an endpoint.
POST /api/chat streams SSE (status/token/tool_call/tool_result/done/
error), starts llama-server on demand, validates tool args server-side
with instructive errors the model can retry on, caps tool results with
a paging hint, and bounds the loop by ai.yaml max_tool_calls plus a
hard timeout. Caddy passes the stream unbuffered (flush_interval -1)
in both deployments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 21:47:50 -06:00
|
|
|
from .config import get_ai_config
|
ai: ground answers in a live library map, budget the context
Response-quality work, all aimed at giving the model enough context to
answer accurately:
- Every chat's system prompt now carries a live library map (today's
date, counts, and the exact names of channels, domains, zims, and
inventories, capped and cached 60s). Exact spellings are in context
before the first token, and the list_collections round-trip — minutes
on slow hardware — is usually unnecessary.
- The request is budgeted against the model's real context: oldest
turns are trimmed first, the current question is never dropped, and
an over-budget turn gets a clear "start a new conversation" error
instead of a silently truncated prompt. cache_prompt reuses the KV
cache for the stable prefix, which is most of the latency on a Pi.
- context_size plumbing fixed: selecting a model adopts its catalog
default (the 4B was running at 8k instead of 16k), and llama-server
runs --parallel 1 so the single user gets the whole context.
- temperature is configurable in ai.yaml (default 0.3 — small models
call tools far more reliably near 0.2-0.3 than at server defaults).
- One worked tool-use example per mode is prepended to every
conversation; small models imitate far better than they obey.
- scripts/ai_eval.py scores the assistant against a live WROLPi on
mechanical checks (finished, called the right tool first, included a
WROLPi link, no external URLs) so prompt changes stop being validated
by anecdote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:48:19 -06:00
|
|
|
from .context import get_library_context
|
|
|
|
|
from .modes import MODE_EXAMPLES, MODE_SUGGESTIONS, MODES
|
ai: the /api/chat agent loop — SSE streaming, modes, tool execution
Tool definitions are generated in-process from the /api/ai blueprint's
own OpenAPI operations (operationId = tool name, description = tool
description, dataclass bodies + declared parameters = one flat JSON
schema) — the spec IS the catalog, nothing to drift. Manage endpoints
never become tools. A mode is a server-enforced system prompt + tool
allowlist (Help / Research / System per ai.md); the executor can only
reach allowlisted /api/ai endpoints, path parameters are encoded and
traversal-rejected so a crafted argument cannot escape an endpoint.
POST /api/chat streams SSE (status/token/tool_call/tool_result/done/
error), starts llama-server on demand, validates tool args server-side
with instructive errors the model can retry on, caps tool results with
a paging hint, and bounds the loop by ai.yaml max_tool_calls plus a
hard timeout. Caddy passes the stream unbuffered (flush_interval -1)
in both deployments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 21:47:50 -06:00
|
|
|
|
|
|
|
|
chat_bp = Blueprint('AIChat', url_prefix='/api/chat')
|
|
|
|
|
|
|
|
|
|
logger = logger.getChild(__name__)
|
|
|
|
|
|
|
|
|
|
# The whole conversation turn (all tool calls and answers) must finish within this.
|
|
|
|
|
CHAT_TIMEOUT_SECONDS = 600
|
|
|
|
|
FALLBACK_MESSAGE = "I couldn't complete that request. Try rephrasing, or a more specific question."
|
ai: ground answers in a live library map, budget the context
Response-quality work, all aimed at giving the model enough context to
answer accurately:
- Every chat's system prompt now carries a live library map (today's
date, counts, and the exact names of channels, domains, zims, and
inventories, capped and cached 60s). Exact spellings are in context
before the first token, and the list_collections round-trip — minutes
on slow hardware — is usually unnecessary.
- The request is budgeted against the model's real context: oldest
turns are trimmed first, the current question is never dropped, and
an over-budget turn gets a clear "start a new conversation" error
instead of a silently truncated prompt. cache_prompt reuses the KV
cache for the stable prefix, which is most of the latency on a Pi.
- context_size plumbing fixed: selecting a model adopts its catalog
default (the 4B was running at 8k instead of 16k), and llama-server
runs --parallel 1 so the single user gets the whole context.
- temperature is configurable in ai.yaml (default 0.3 — small models
call tools far more reliably near 0.2-0.3 than at server defaults).
- One worked tool-use example per mode is prepended to every
conversation; small models imitate far better than they obey.
- scripts/ai_eval.py scores the assistant against a live WROLPi on
mechanical checks (finished, called the right tool first, included a
WROLPi link, no external URLs) so prompt changes stop being validated
by anecdote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:48:19 -06:00
|
|
|
# Tokens kept free for the model's reply when budgeting the prompt.
|
|
|
|
|
REPLY_RESERVE_TOKENS = 768
|
|
|
|
|
CONTEXT_FULL_MESSAGE = ('This conversation no longer fits the model\'s context.'
|
|
|
|
|
' Start a new conversation and ask a more specific question.')
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def estimate_tokens(messages: List[dict], tools: List[dict]) -> int:
|
|
|
|
|
"""A deliberate overestimate (~4 chars/token) of what one completion request costs."""
|
|
|
|
|
return (len(json.dumps(messages)) + len(json.dumps(tools, default=str))) // 4
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def trim_history(history: List[dict], fixed: List[dict], tools: List[dict], budget: int) -> List[dict]:
|
|
|
|
|
"""Drop the oldest conversation turns until fixed + history fits the budget.
|
|
|
|
|
|
|
|
|
|
`fixed` (system prompt, library map, few-shot example, and the current user message) is never
|
|
|
|
|
trimmed; old turns are the least valuable tokens in the request."""
|
|
|
|
|
history = list(history)
|
|
|
|
|
while history and estimate_tokens(fixed + history, tools) > budget:
|
|
|
|
|
history.pop(0)
|
|
|
|
|
return history
|
ai: the /api/chat agent loop — SSE streaming, modes, tool execution
Tool definitions are generated in-process from the /api/ai blueprint's
own OpenAPI operations (operationId = tool name, description = tool
description, dataclass bodies + declared parameters = one flat JSON
schema) — the spec IS the catalog, nothing to drift. Manage endpoints
never become tools. A mode is a server-enforced system prompt + tool
allowlist (Help / Research / System per ai.md); the executor can only
reach allowlisted /api/ai endpoints, path parameters are encoded and
traversal-rejected so a crafted argument cannot escape an endpoint.
POST /api/chat streams SSE (status/token/tool_call/tool_result/done/
error), starts llama-server on demand, validates tool args server-side
with instructive errors the model can retry on, caps tool results with
a paging hint, and bounds the loop by ai.yaml max_tool_calls plus a
hard timeout. Caddy passes the stream unbuffered (flush_interval -1)
in both deployments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 21:47:50 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
class AIDisabled(APIError):
|
|
|
|
|
code = 'AI_DISABLED'
|
|
|
|
|
summary = 'AI is not enabled. Enable it and select a model on the AI page.'
|
|
|
|
|
status_code = HTTPStatus.CONFLICT
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@dataclasses.dataclass
|
|
|
|
|
class ChatRequest:
|
|
|
|
|
mode: str = ''
|
|
|
|
|
messages: List[dict] = dataclasses.field(default_factory=list)
|
|
|
|
|
model: Optional[str] = None # reserved; llama-server runs the single active model
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@chat_bp.get('/modes')
|
|
|
|
|
@openapi.description('The available chat modes with their suggestion chips.')
|
|
|
|
|
async def get_modes(_: Request):
|
|
|
|
|
from wrolpi.api_utils import json_response
|
|
|
|
|
modes = [dict(name=name, suggestions=MODE_SUGGESTIONS.get(name, [])) for name in MODES]
|
|
|
|
|
return json_response(dict(modes=modes))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
async def llama_chat_stream(messages: List[dict], tools: List[dict]) -> AsyncGenerator[dict, None]:
|
|
|
|
|
"""Stream one /v1/chat/completions call from llama-server.
|
|
|
|
|
|
|
|
|
|
Yields {'type': 'token', 'content': str} for answer fragments, then exactly one
|
|
|
|
|
{'type': 'stop', 'tool_calls': [...]} where tool_calls is empty for a final answer.
|
|
|
|
|
Patched in tests."""
|
ai: ground answers in a live library map, budget the context
Response-quality work, all aimed at giving the model enough context to
answer accurately:
- Every chat's system prompt now carries a live library map (today's
date, counts, and the exact names of channels, domains, zims, and
inventories, capped and cached 60s). Exact spellings are in context
before the first token, and the list_collections round-trip — minutes
on slow hardware — is usually unnecessary.
- The request is budgeted against the model's real context: oldest
turns are trimmed first, the current question is never dropped, and
an over-budget turn gets a clear "start a new conversation" error
instead of a silently truncated prompt. cache_prompt reuses the KV
cache for the stable prefix, which is most of the latency on a Pi.
- context_size plumbing fixed: selecting a model adopts its catalog
default (the 4B was running at 8k instead of 16k), and llama-server
runs --parallel 1 so the single user gets the whole context.
- temperature is configurable in ai.yaml (default 0.3 — small models
call tools far more reliably near 0.2-0.3 than at server defaults).
- One worked tool-use example per mode is prepended to every
conversation; small models imitate far better than they obey.
- scripts/ai_eval.py scores the assistant against a live WROLPi on
mechanical checks (finished, called the right tool first, included a
WROLPi link, no external URLs) so prompt changes stop being validated
by anecdote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:48:19 -06:00
|
|
|
config = get_ai_config()
|
|
|
|
|
body = dict(
|
|
|
|
|
messages=messages,
|
|
|
|
|
stream=True,
|
|
|
|
|
# Low temperature: tool calls from small models are far more reliable near 0.2-0.3.
|
|
|
|
|
temperature=config.temperature,
|
|
|
|
|
# Reuse the KV cache for the stable prefix (system prompt + example); on slow CPUs
|
|
|
|
|
# re-processing that prefix is most of the latency.
|
|
|
|
|
cache_prompt=True,
|
|
|
|
|
)
|
ai: the /api/chat agent loop — SSE streaming, modes, tool execution
Tool definitions are generated in-process from the /api/ai blueprint's
own OpenAPI operations (operationId = tool name, description = tool
description, dataclass bodies + declared parameters = one flat JSON
schema) — the spec IS the catalog, nothing to drift. Manage endpoints
never become tools. A mode is a server-enforced system prompt + tool
allowlist (Help / Research / System per ai.md); the executor can only
reach allowlisted /api/ai endpoints, path parameters are encoded and
traversal-rejected so a crafted argument cannot escape an endpoint.
POST /api/chat streams SSE (status/token/tool_call/tool_result/done/
error), starts llama-server on demand, validates tool args server-side
with instructive errors the model can retry on, caps tool results with
a paging hint, and bounds the loop by ai.yaml max_tool_calls plus a
hard timeout. Caddy passes the stream unbuffered (flush_interval -1)
in both deployments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 21:47:50 -06:00
|
|
|
if tools:
|
|
|
|
|
body['tools'] = tools
|
|
|
|
|
|
|
|
|
|
tool_calls = {} # index -> {id, name, arguments}
|
|
|
|
|
async with aiohttp_post(f'{service.LLAMA_URL}/v1/chat/completions', json_=body,
|
|
|
|
|
timeout=CHAT_TIMEOUT_SECONDS) as response:
|
|
|
|
|
if response.status != 200:
|
|
|
|
|
raise RuntimeError(f'llama-server returned HTTP {response.status}')
|
|
|
|
|
async for raw_line in response.content:
|
|
|
|
|
line = raw_line.decode('UTF-8', errors='replace').strip()
|
|
|
|
|
if not line.startswith('data:'):
|
|
|
|
|
continue
|
|
|
|
|
data = line[5:].strip()
|
|
|
|
|
if data == '[DONE]':
|
|
|
|
|
break
|
|
|
|
|
try:
|
|
|
|
|
chunk = json.loads(data)
|
|
|
|
|
except ValueError:
|
|
|
|
|
continue
|
|
|
|
|
choice = (chunk.get('choices') or [{}])[0]
|
|
|
|
|
delta = choice.get('delta') or {}
|
|
|
|
|
if content := delta.get('content'):
|
|
|
|
|
yield dict(type='token', content=content)
|
|
|
|
|
for tool_call in delta.get('tool_calls') or []:
|
|
|
|
|
index = tool_call.get('index', 0)
|
|
|
|
|
entry = tool_calls.setdefault(index, dict(id=None, name='', arguments=''))
|
|
|
|
|
if tool_call.get('id'):
|
|
|
|
|
entry['id'] = tool_call['id']
|
|
|
|
|
function = tool_call.get('function') or {}
|
|
|
|
|
if function.get('name'):
|
|
|
|
|
entry['name'] = function['name']
|
|
|
|
|
if function.get('arguments'):
|
|
|
|
|
entry['arguments'] += function['arguments']
|
|
|
|
|
|
|
|
|
|
yield dict(type='stop', tool_calls=[tool_calls[i] for i in sorted(tool_calls)])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
async def _send_event(response, event: str, data: dict):
|
|
|
|
|
await response.send(f'event: {event}\ndata: {json.dumps(data, default=str)}\n\n')
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@chat_bp.post('/')
|
|
|
|
|
@openapi.definition(
|
|
|
|
|
summary='Chat with the local AI assistant (SSE stream)',
|
|
|
|
|
body=ChatRequest,
|
|
|
|
|
)
|
|
|
|
|
@validate(ChatRequest)
|
|
|
|
|
async def post_chat(request: Request, body: ChatRequest):
|
|
|
|
|
if body.mode not in MODES:
|
|
|
|
|
raise ValidationError(f'Unknown mode: {body.mode!r}. Modes: {sorted(MODES)}')
|
|
|
|
|
if not body.messages:
|
|
|
|
|
raise ValidationError('messages must not be empty')
|
|
|
|
|
config = get_ai_config()
|
|
|
|
|
if not config.enabled or not config.active_model:
|
|
|
|
|
raise AIDisabled()
|
|
|
|
|
|
|
|
|
|
mode = MODES[body.mode]
|
|
|
|
|
tools = agent.get_mode_tools(mode['tools'])
|
|
|
|
|
max_tool_calls = config.max_tool_calls
|
|
|
|
|
deadline = time.time() + CHAT_TIMEOUT_SECONDS
|
|
|
|
|
|
|
|
|
|
response = await request.respond(content_type='text/event-stream', headers={
|
|
|
|
|
'Cache-Control': 'no-cache',
|
|
|
|
|
# Ask proxies not to buffer the stream.
|
|
|
|
|
'X-Accel-Buffering': 'no',
|
|
|
|
|
})
|
|
|
|
|
|
|
|
|
|
try:
|
|
|
|
|
if not await service.llama_healthy():
|
|
|
|
|
await _send_event(response, 'status', dict(message='Loading the model…'))
|
|
|
|
|
if not await service.ensure_llama_running():
|
|
|
|
|
await _send_event(response, 'error', dict(message='The AI service could not be started.'))
|
|
|
|
|
return
|
|
|
|
|
|
ai: ground answers in a live library map, budget the context
Response-quality work, all aimed at giving the model enough context to
answer accurately:
- Every chat's system prompt now carries a live library map (today's
date, counts, and the exact names of channels, domains, zims, and
inventories, capped and cached 60s). Exact spellings are in context
before the first token, and the list_collections round-trip — minutes
on slow hardware — is usually unnecessary.
- The request is budgeted against the model's real context: oldest
turns are trimmed first, the current question is never dropped, and
an over-budget turn gets a clear "start a new conversation" error
instead of a silently truncated prompt. cache_prompt reuses the KV
cache for the stable prefix, which is most of the latency on a Pi.
- context_size plumbing fixed: selecting a model adopts its catalog
default (the 4B was running at 8k instead of 16k), and llama-server
runs --parallel 1 so the single user gets the whole context.
- temperature is configurable in ai.yaml (default 0.3 — small models
call tools far more reliably near 0.2-0.3 than at server defaults).
- One worked tool-use example per mode is prepended to every
conversation; small models imitate far better than they obey.
- scripts/ai_eval.py scores the assistant against a live WROLPi on
mechanical checks (finished, called the right tool first, included a
WROLPi link, no external URLs) so prompt changes stop being validated
by anecdote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:48:19 -06:00
|
|
|
# System prompt + the live library map, so exact names, counts, and today's date are in
|
|
|
|
|
# context before the first token; then the mode's worked example, then the conversation.
|
|
|
|
|
system_content = mode['system_prompt']
|
|
|
|
|
if library_context := get_library_context():
|
|
|
|
|
system_content += f'\n\n{library_context}'
|
|
|
|
|
fixed = [dict(role='system', content=system_content)]
|
|
|
|
|
fixed += MODE_EXAMPLES.get(body.mode, [])
|
|
|
|
|
|
|
|
|
|
history = [dict(role=m.get('role', 'user'), content=m.get('content', '')) for m in body.messages]
|
|
|
|
|
# The current user message is never trimmed away.
|
|
|
|
|
current, history = history[-1], history[:-1]
|
|
|
|
|
fixed.append(current)
|
|
|
|
|
|
|
|
|
|
budget = get_effective_context_size() - REPLY_RESERVE_TOKENS
|
|
|
|
|
history = trim_history(history, fixed, tools, budget)
|
|
|
|
|
if estimate_tokens(fixed + history, tools) > budget:
|
|
|
|
|
await _send_event(response, 'error', dict(message=CONTEXT_FULL_MESSAGE))
|
|
|
|
|
return
|
|
|
|
|
messages = fixed[:-1] + history + [current]
|
ai: the /api/chat agent loop — SSE streaming, modes, tool execution
Tool definitions are generated in-process from the /api/ai blueprint's
own OpenAPI operations (operationId = tool name, description = tool
description, dataclass bodies + declared parameters = one flat JSON
schema) — the spec IS the catalog, nothing to drift. Manage endpoints
never become tools. A mode is a server-enforced system prompt + tool
allowlist (Help / Research / System per ai.md); the executor can only
reach allowlisted /api/ai endpoints, path parameters are encoded and
traversal-rejected so a crafted argument cannot escape an endpoint.
POST /api/chat streams SSE (status/token/tool_call/tool_result/done/
error), starts llama-server on demand, validates tool args server-side
with instructive errors the model can retry on, caps tool results with
a paging hint, and bounds the loop by ai.yaml max_tool_calls plus a
hard timeout. Caddy passes the stream unbuffered (flush_interval -1)
in both deployments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 21:47:50 -06:00
|
|
|
|
|
|
|
|
final_answer = []
|
|
|
|
|
for _ in range(max_tool_calls + 1):
|
|
|
|
|
if time.time() > deadline:
|
|
|
|
|
await _send_event(response, 'error', dict(message=FALLBACK_MESSAGE))
|
|
|
|
|
return
|
|
|
|
|
|
ai: ground answers in a live library map, budget the context
Response-quality work, all aimed at giving the model enough context to
answer accurately:
- Every chat's system prompt now carries a live library map (today's
date, counts, and the exact names of channels, domains, zims, and
inventories, capped and cached 60s). Exact spellings are in context
before the first token, and the list_collections round-trip — minutes
on slow hardware — is usually unnecessary.
- The request is budgeted against the model's real context: oldest
turns are trimmed first, the current question is never dropped, and
an over-budget turn gets a clear "start a new conversation" error
instead of a silently truncated prompt. cache_prompt reuses the KV
cache for the stable prefix, which is most of the latency on a Pi.
- context_size plumbing fixed: selecting a model adopts its catalog
default (the 4B was running at 8k instead of 16k), and llama-server
runs --parallel 1 so the single user gets the whole context.
- temperature is configurable in ai.yaml (default 0.3 — small models
call tools far more reliably near 0.2-0.3 than at server defaults).
- One worked tool-use example per mode is prepended to every
conversation; small models imitate far better than they obey.
- scripts/ai_eval.py scores the assistant against a live WROLPi on
mechanical checks (finished, called the right tool first, included a
WROLPi link, no external URLs) so prompt changes stop being validated
by anecdote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:48:19 -06:00
|
|
|
# Tool results grow the request; refuse to overflow the model's context mid-loop.
|
|
|
|
|
if estimate_tokens(messages, tools) > budget:
|
|
|
|
|
await _send_event(response, 'error', dict(message=CONTEXT_FULL_MESSAGE))
|
|
|
|
|
return
|
|
|
|
|
|
ai: the /api/chat agent loop — SSE streaming, modes, tool execution
Tool definitions are generated in-process from the /api/ai blueprint's
own OpenAPI operations (operationId = tool name, description = tool
description, dataclass bodies + declared parameters = one flat JSON
schema) — the spec IS the catalog, nothing to drift. Manage endpoints
never become tools. A mode is a server-enforced system prompt + tool
allowlist (Help / Research / System per ai.md); the executor can only
reach allowlisted /api/ai endpoints, path parameters are encoded and
traversal-rejected so a crafted argument cannot escape an endpoint.
POST /api/chat streams SSE (status/token/tool_call/tool_result/done/
error), starts llama-server on demand, validates tool args server-side
with instructive errors the model can retry on, caps tool results with
a paging hint, and bounds the loop by ai.yaml max_tool_calls plus a
hard timeout. Caddy passes the stream unbuffered (flush_interval -1)
in both deployments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 21:47:50 -06:00
|
|
|
service.record_llm_activity()
|
2026-08-27 16:28:04 -06:00
|
|
|
# Always send something before the model call: on slow CPUs the first token can be
|
|
|
|
|
# minutes away, and a silent stream looks dead to the UI and to proxies.
|
|
|
|
|
await _send_event(response, 'status', dict(message='Thinking…'))
|
ai: the /api/chat agent loop — SSE streaming, modes, tool execution
Tool definitions are generated in-process from the /api/ai blueprint's
own OpenAPI operations (operationId = tool name, description = tool
description, dataclass bodies + declared parameters = one flat JSON
schema) — the spec IS the catalog, nothing to drift. Manage endpoints
never become tools. A mode is a server-enforced system prompt + tool
allowlist (Help / Research / System per ai.md); the executor can only
reach allowlisted /api/ai endpoints, path parameters are encoded and
traversal-rejected so a crafted argument cannot escape an endpoint.
POST /api/chat streams SSE (status/token/tool_call/tool_result/done/
error), starts llama-server on demand, validates tool args server-side
with instructive errors the model can retry on, caps tool results with
a paging hint, and bounds the loop by ai.yaml max_tool_calls plus a
hard timeout. Caddy passes the stream unbuffered (flush_interval -1)
in both deployments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 21:47:50 -06:00
|
|
|
tool_calls = []
|
|
|
|
|
final_answer = []
|
|
|
|
|
async for event in llama_chat_stream(messages, tools):
|
|
|
|
|
if event['type'] == 'token':
|
|
|
|
|
final_answer.append(event['content'])
|
|
|
|
|
await _send_event(response, 'token', dict(content=event['content']))
|
|
|
|
|
elif event['type'] == 'stop':
|
|
|
|
|
tool_calls = event['tool_calls']
|
|
|
|
|
|
|
|
|
|
if not tool_calls:
|
|
|
|
|
# A final answer.
|
|
|
|
|
content = ''.join(final_answer).strip()
|
|
|
|
|
await _send_event(response, 'done', dict(content=content or FALLBACK_MESSAGE))
|
|
|
|
|
return
|
|
|
|
|
|
|
|
|
|
# Execute the tool calls, then loop for the model's next step.
|
|
|
|
|
messages.append(dict(role='assistant', content=''.join(final_answer) or None, tool_calls=[
|
|
|
|
|
dict(id=i['id'] or f'call_{n}', type='function',
|
|
|
|
|
function=dict(name=i['name'], arguments=i['arguments']))
|
|
|
|
|
for n, i in enumerate(tool_calls)
|
|
|
|
|
]))
|
|
|
|
|
for n, tool_call in enumerate(tool_calls):
|
|
|
|
|
name = tool_call['name']
|
|
|
|
|
try:
|
|
|
|
|
args = json.loads(tool_call['arguments']) if tool_call['arguments'] else {}
|
|
|
|
|
except ValueError:
|
|
|
|
|
args = None
|
|
|
|
|
await _send_event(response, 'tool_call', dict(tool=name, args=args))
|
|
|
|
|
if args is None:
|
|
|
|
|
success, result = False, 'The tool arguments were not valid JSON. Try again.'
|
|
|
|
|
else:
|
|
|
|
|
success, result = await agent.execute_tool(mode['tools'], name, args)
|
|
|
|
|
await _send_event(response, 'tool_result', dict(tool=name, success=success))
|
|
|
|
|
messages.append(dict(role='tool', tool_call_id=tool_call['id'] or f'call_{n}',
|
|
|
|
|
content=result))
|
|
|
|
|
|
|
|
|
|
# Tool-call budget exhausted without a final answer.
|
|
|
|
|
await _send_event(response, 'error', dict(message=FALLBACK_MESSAGE))
|
|
|
|
|
except Exception as e:
|
|
|
|
|
logger.error('Chat failed', exc_info=e)
|
|
|
|
|
try:
|
|
|
|
|
await _send_event(response, 'error', dict(message='The AI failed to answer. Check the logs.'))
|
|
|
|
|
except Exception:
|
|
|
|
|
pass
|
|
|
|
|
finally:
|
|
|
|
|
await response.eof()
|