ai: ground answers in a live library map, budget the context
Response-quality work, all aimed at giving the model enough context to
answer accurately:
- Every chat's system prompt now carries a live library map (today's
date, counts, and the exact names of channels, domains, zims, and
inventories, capped and cached 60s). Exact spellings are in context
before the first token, and the list_collections round-trip — minutes
on slow hardware — is usually unnecessary.
- The request is budgeted against the model's real context: oldest
turns are trimmed first, the current question is never dropped, and
an over-budget turn gets a clear "start a new conversation" error
instead of a silently truncated prompt. cache_prompt reuses the KV
cache for the stable prefix, which is most of the latency on a Pi.
- context_size plumbing fixed: selecting a model adopts its catalog
default (the 4B was running at 8k instead of 16k), and llama-server
runs --parallel 1 so the single user gets the whole context.
- temperature is configurable in ai.yaml (default 0.3 — small models
call tools far more reliably near 0.2-0.3 than at server defaults).
- One worked tool-use example per mode is prepended to every
conversation; small models imitate far better than they obey.
- scripts/ai_eval.py scores the assistant against a live WROLPi on
mechanical checks (finished, called the right tool first, included a
WROLPi link, no external URLs) so prompt changes stop being validated
by anecdote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:48:19 -06:00
|
|
|
"""Tests for the library map injected into every chat."""
|
|
|
|
|
import time
|
|
|
|
|
from unittest import mock
|
|
|
|
|
|
|
|
|
|
import pytest
|
|
|
|
|
|
|
|
|
|
from modules.ai import context
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_build_library_context(async_client, test_session, video_factory, archive_factory,
|
ai: kind-generic search_files / get_file / read_content
Small models carry every tool definition on every request, and the
per-kind trios (four searches, three getters, three readers) cost a
quarter of an 8k context while giving the model near-duplicate choices.
Three generic endpoints replace them:
- POST files/search: one search over videos, archives, and docs with an
optional kind; channel (name or id), domain, author, subject each
imply their kind and are ignored for other kinds. When the term also
names channels/domains/authors, they are returned as `matches` so the
model can narrow without a second tool. Channels resolve by name, so
the library map already in the prompt is enough: no list call first.
- GET files/{id}: detail by id whatever the kind; videos now report
has_captions/has_comments and the full description, archives keep
their snapshot history.
- GET files/{id}/content?part=text|comments: captions, page text, or
comments, paged; a wrong part for the kind says what the file is.
Inventories take an optional slug on GET /inventories (one tool lists
and fetches). Listings are slimmed: no derivable captions/text links,
no size/mimetype, duration as H:MM:SS, published as a date. The
library map gains the tag names.
wrolpi_mcp keeps every existing tool name; the wrappers now call the
generic routes, and search_files/get_file/read_content are exposed too.
The old per-kind endpoints remain until the modes switch over.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-10 16:09:50 -06:00
|
|
|
test_zim, food_inventory_factory, tag_factory):
|
ai: ground answers in a live library map, budget the context
Response-quality work, all aimed at giving the model enough context to
answer accurately:
- Every chat's system prompt now carries a live library map (today's
date, counts, and the exact names of channels, domains, zims, and
inventories, capped and cached 60s). Exact spellings are in context
before the first token, and the list_collections round-trip — minutes
on slow hardware — is usually unnecessary.
- The request is budgeted against the model's real context: oldest
turns are trimmed first, the current question is never dropped, and
an over-budget turn gets a clear "start a new conversation" error
instead of a silently truncated prompt. cache_prompt reuses the KV
cache for the stable prefix, which is most of the latency on a Pi.
- context_size plumbing fixed: selecting a model adopts its catalog
default (the 4B was running at 8k instead of 16k), and llama-server
runs --parallel 1 so the single user gets the whole context.
- temperature is configurable in ai.yaml (default 0.3 — small models
call tools far more reliably near 0.2-0.3 than at server defaults).
- One worked tool-use example per mode is prepended to every
conversation; small models imitate far better than they obey.
- scripts/ai_eval.py scores the assistant against a live WROLPi on
mechanical checks (finished, called the right tool first, included a
WROLPi link, no external URLs) so prompt changes stop being validated
by anecdote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:48:19 -06:00
|
|
|
"""The map carries today's date, counts, and exact names."""
|
|
|
|
|
from wrolpi.collections.models import Collection
|
|
|
|
|
test_session.add(Collection(name='Talking Sasquach', kind='channel'))
|
ai: kind-generic search_files / get_file / read_content
Small models carry every tool definition on every request, and the
per-kind trios (four searches, three getters, three readers) cost a
quarter of an 8k context while giving the model near-duplicate choices.
Three generic endpoints replace them:
- POST files/search: one search over videos, archives, and docs with an
optional kind; channel (name or id), domain, author, subject each
imply their kind and are ignored for other kinds. When the term also
names channels/domains/authors, they are returned as `matches` so the
model can narrow without a second tool. Channels resolve by name, so
the library map already in the prompt is enough: no list call first.
- GET files/{id}: detail by id whatever the kind; videos now report
has_captions/has_comments and the full description, archives keep
their snapshot history.
- GET files/{id}/content?part=text|comments: captions, page text, or
comments, paged; a wrong part for the kind says what the file is.
Inventories take an optional slug on GET /inventories (one tool lists
and fetches). Listings are slimmed: no derivable captions/text links,
no size/mimetype, duration as H:MM:SS, published as a date. The
library map gains the tag names.
wrolpi_mcp keeps every existing tool name; the wrappers now call the
generic routes, and search_files/get_file/read_content are exposed too.
The old per-kind endpoints remain until the modes switch over.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-10 16:09:50 -06:00
|
|
|
await tag_factory('Food Preservation')
|
ai: ground answers in a live library map, budget the context
Response-quality work, all aimed at giving the model enough context to
answer accurately:
- Every chat's system prompt now carries a live library map (today's
date, counts, and the exact names of channels, domains, zims, and
inventories, capped and cached 60s). Exact spellings are in context
before the first token, and the list_collections round-trip — minutes
on slow hardware — is usually unnecessary.
- The request is budgeted against the model's real context: oldest
turns are trimmed first, the current question is never dropped, and
an over-budget turn gets a clear "start a new conversation" error
instead of a silently truncated prompt. cache_prompt reuses the KV
cache for the stable prefix, which is most of the latency on a Pi.
- context_size plumbing fixed: selecting a model adopts its catalog
default (the 4B was running at 8k instead of 16k), and llama-server
runs --parallel 1 so the single user gets the whole context.
- temperature is configurable in ai.yaml (default 0.3 — small models
call tools far more reliably near 0.2-0.3 than at server defaults).
- One worked tool-use example per mode is prepended to every
conversation; small models imitate far better than they obey.
- scripts/ai_eval.py scores the assistant against a live WROLPi on
mechanical checks (finished, called the right tool first, included a
WROLPi link, no external URLs) so prompt changes stop being validated
by anecdote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:48:19 -06:00
|
|
|
video_factory(title='a video')
|
|
|
|
|
archive_factory(domain='example.com', title='a page', contents='contents')
|
|
|
|
|
food_inventory_factory(name='Pantry')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
text = context.build_library_context()
|
|
|
|
|
assert 'Today is' in text
|
|
|
|
|
assert '1 videos, 1 archived web pages' in text
|
|
|
|
|
assert 'Talking Sasquach' in text
|
|
|
|
|
assert 'example.com' in text
|
|
|
|
|
assert 'WROLPi example ZIM' in text
|
|
|
|
|
assert 'Pantry' in text
|
ai: kind-generic search_files / get_file / read_content
Small models carry every tool definition on every request, and the
per-kind trios (four searches, three getters, three readers) cost a
quarter of an 8k context while giving the model near-duplicate choices.
Three generic endpoints replace them:
- POST files/search: one search over videos, archives, and docs with an
optional kind; channel (name or id), domain, author, subject each
imply their kind and are ignored for other kinds. When the term also
names channels/domains/authors, they are returned as `matches` so the
model can narrow without a second tool. Channels resolve by name, so
the library map already in the prompt is enough: no list call first.
- GET files/{id}: detail by id whatever the kind; videos now report
has_captions/has_comments and the full description, archives keep
their snapshot history.
- GET files/{id}/content?part=text|comments: captions, page text, or
comments, paged; a wrong part for the kind says what the file is.
Inventories take an optional slug on GET /inventories (one tool lists
and fetches). Listings are slimmed: no derivable captions/text links,
no size/mimetype, duration as H:MM:SS, published as a date. The
library map gains the tag names.
wrolpi_mcp keeps every existing tool name; the wrappers now call the
generic routes, and search_files/get_file/read_content are exposed too.
The old per-kind endpoints remain until the modes switch over.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-10 16:09:50 -06:00
|
|
|
assert 'Tags (1): Food Preservation' in text
|
ai: ground answers in a live library map, budget the context
Response-quality work, all aimed at giving the model enough context to
answer accurately:
- Every chat's system prompt now carries a live library map (today's
date, counts, and the exact names of channels, domains, zims, and
inventories, capped and cached 60s). Exact spellings are in context
before the first token, and the list_collections round-trip — minutes
on slow hardware — is usually unnecessary.
- The request is budgeted against the model's real context: oldest
turns are trimmed first, the current question is never dropped, and
an over-budget turn gets a clear "start a new conversation" error
instead of a silently truncated prompt. cache_prompt reuses the KV
cache for the stable prefix, which is most of the latency on a Pi.
- context_size plumbing fixed: selecting a model adopts its catalog
default (the 4B was running at 8k instead of 16k), and llama-server
runs --parallel 1 so the single user gets the whole context.
- temperature is configurable in ai.yaml (default 0.3 — small models
call tools far more reliably near 0.2-0.3 than at server defaults).
- One worked tool-use example per mode is prepended to every
conversation; small models imitate far better than they obey.
- scripts/ai_eval.py scores the assistant against a live WROLPi on
mechanical checks (finished, called the right tool first, included a
WROLPi link, no external URLs) so prompt changes stop being validated
by anecdote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:48:19 -06:00
|
|
|
assert 'user spellings may differ' in text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_library_context_caps_names(async_client, test_session, archive_factory):
|
|
|
|
|
"""A huge library lists at most MAX_NAMES names per category, plus a count of the rest."""
|
|
|
|
|
for i in range(context.MAX_NAMES + 5):
|
|
|
|
|
archive_factory(domain=f'site{i:02}.com', title=f'page {i}', contents='x')
|
|
|
|
|
test_session.commit()
|
|
|
|
|
|
|
|
|
|
text = context.build_library_context()
|
|
|
|
|
assert f'Archived websites ({context.MAX_NAMES + 5}):' in text
|
|
|
|
|
assert 'and 5 more' in text
|
|
|
|
|
listed = [i for i in range(context.MAX_NAMES + 5) if f'site{i:02}.com' in text]
|
|
|
|
|
assert len(listed) == context.MAX_NAMES
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
|
|
|
async def test_library_context_cache_and_degradation(async_client, test_session):
|
|
|
|
|
"""The map is cached briefly; a build failure serves the stale copy rather than crashing chat."""
|
|
|
|
|
context._cache.update(text=None, expires=0.0)
|
|
|
|
|
first = context.get_library_context()
|
|
|
|
|
assert first and 'Today is' in first
|
|
|
|
|
|
|
|
|
|
with mock.patch('modules.ai.context.build_library_context', side_effect=RuntimeError('db down')) as build:
|
|
|
|
|
# Cached: the builder is not called.
|
|
|
|
|
assert context.get_library_context() == first
|
|
|
|
|
build.assert_not_called()
|
|
|
|
|
|
|
|
|
|
# Expired + failing builder: the stale copy is served.
|
|
|
|
|
context._cache['expires'] = time.time() - 1
|
|
|
|
|
assert context.get_library_context() == first
|
|
|
|
|
build.assert_called_once()
|