Commit graph

9 commits

Author SHA1 Message Date
Roland
1a28a5900a
repos: git inherits only an allow-listed environment
git_env copied the whole process environment, and git consults its
environment ahead of `-c`: GIT_ALLOW_PROTOCOL=file re-enabled file://
despite protocol.allow=never, and GIT_CONFIG_COUNT/KEY_*/VALUE_* could
add an `insteadOf` that rewrites an approved URL to another host.  Only
PATH, HOME, TMPDIR, TZ and the SSL certificate paths are inherited now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 17:32:59 -06:00
Roland
2dd8e908c2
repos: check submodule URLs where git resolves them
A relative submodule URL is resolved by git against the repo's remote,
and can climb to another host: with the remote
https://git.example.com/owner/repo, `../../../10.0.0.5/x` is
https://10.0.0.5/x.  Relative URLs were allowed unchecked.

git now resolves the URLs first (`submodule init` + `sync` write them to
the repo's config), and those resolved URLs, the ones git fetches, are
checked.  A relative URL is never allowed by the check itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 16:56:10 -06:00
Roland
4810b8bc50
repos: close the remaining submodule SSRF gaps
* .gitmodules is read NUL-delimited, and an unreadable record refuses
  the submodules.  A submodule name with a space (`[submodule "a b"]`)
  made `--get-regexp`'s "key value" output misfile its URL as a path, so
  the URL was never checked.
* Submodule URLs must have a strict shape every parser reads the same
  way (no userinfo, backslash, query, fragment, or whitespace), so a URL
  cannot be checked as one host and fetched from another.
* Each checked host is pinned (http.curloptResolve) to the addresses
  that were checked, so DNS cannot change between the check and git's
  fetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 16:56:10 -06:00
Roland
3b6e962dfc
repos: never fetch submodules from WROLPi's own network
A repo's .gitmodules is chosen by whoever controls the repo, so a
submodule URL could make WROLPi fetch https://localhost:8443/...,
https://10.0.0.1/..., etc. (SSRF), and show the response's error.

Every submodule URL is now checked before git fetches it: relative URLs,
the repo's own host, and hosts whose addresses are all public are
allowed; anything else defers the download with the reason.  Nested
submodules are checked out one level at a time (instead of git's
--recursive) so each level is checked, at most 5 levels deep.  Submodule
fetches do not follow HTTP redirects, so a checked host cannot redirect
git into the network.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 16:56:09 -06:00
Roland
03e3476631
repos: history, Download ZIP, and submodules
* GET /api/repos/<id>/log: the checked out branch's commits, newest
  first, a page at a time, with the total.  The repo page has a
  Files | History toggle (the tree is only listed for Files).
* GET /api/repos/<id>/archive.zip: `git archive HEAD` streamed to the
  browser; the file name is reduced to safe characters.  Everything is
  read from the database before the response starts, because the
  request's session closes once a streamed response begins.
* repository.submodules (default off): submodules are checked out with
  the repo's mode and the same git restrictions, and removed when the
  option is turned off.  A submodule that cannot be downloaded (e.g. an
  ssh URL) does not stop the repo from updating; the download is
  retried with the submodule's error.
* The README speed test bound is loose enough for a busy machine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 16:56:09 -06:00
Roland
5506dc0a88
repos: harden README search against untrusted READMEs
A README comes from an untrusted remote.

* Escape the README headline: snippet() marks matches with control
  characters, the text is HTML-escaped, then the markers become <b>.
  Headlines are rendered as HTML (HeadlineText).
* Never follow a symlinked README (O_NOFOLLOW, regular files only).
* Convert markdown line by line with linear-time patterns.  The old
  multiline patterns backtracked polynomially; a README of blank lines
  never finished, and runs of `<!--`, `![` or `<` were quadratic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 16:56:09 -06:00
Roland
5c80ad7ba4
repos: search Repos by name and README
repository_fts (FTS5, external content) indexes each Repo's search_name
(name and owner) and readme_text; a name match outranks a README
mention.  The downloader stores the README as plain text (markdown
syntax removed so snippets read well), writing it only when it changed.

* POST /api/repos/search: by search and/or Tag, with README headlines.
* /api/search_other_estimates takes an optional search_str and returns
  repo_count, so Repos count toward the global search's Other tab.
* /api/search_suggestions suggests Repos by name.

The repository FTS DDL is installed only once the repository table
exists: the baseline migration also runs install_raw_ddl, before that
table is created.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 16:56:09 -06:00
Roland
f49288b8ec
repos: validate branch names and repo directories
Branch names are passed to git as arguments and come from the API, the
config, a `/tree/<branch>` URL, or the remote itself (its default
branch).  A name starting with `-` would be read as an option, so every
branch must pass git's ref-name rules, including in the downloader just
before use.

A repo URL with a `.` or `..` path segment (or a `..` name) could format
a directory outside its template (e.g. `repos/<tag>/..` is the whole
repos directory).  Such URLs and names are rejected, and every repo
directory, including one from repos.yaml, must be strictly inside the
media directory with no `..`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 16:56:09 -06:00
Roland
b3255e715c
Add a Repos module that mirrors git repositories (backend)
A Repo is a Collection (kind='repo') plus a `repository` row, kept up to
date by a recurring Download handled by the new `git` downloader.

* Full history by default (all branches and tags, no --prune; a force
  push saves the old HEAD under refs/wrolpi/backup/); snapshot mode
  (--depth 1) and a pinned branch are optional.  `/tree/<branch>` URLs
  select a branch.
* git runs HTTPS-only, never prompts, never runs hooks, and checks out
  symlinks as plain files.  Upstream disappearing defers the download
  with git's error; the local copy is kept.
* Repos are cloned to `repos_destination` (repos/%(repo_tag)s/%(repo_name)s)
  and the `repos` directory is always ignored, so repo files are never
  indexed.
* repos.yaml is the source of truth (import, dump, backup restore).
* /api/repos: list, create, get, edit, delete (files kept unless
  delete_files), update now, and a per-directory file tree.  Tagging uses
  the collections tag endpoint.

Also:
* run_command/process_runner can merge stderr into stdout (git writes
  progress to stderr).
* move_collection deletes an old directory that only holds empty
  directories (the move left an emptied .git behind).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 16:56:09 -06:00