Contextator
ENTR

Sources

Git Repository Source

Indexing documentation straight from a git repository — private repos, provider tokens, how syncing and push-triggered re-indexing work.

Updated:

Index documentation straight from a git repository. Contextator keeps a shallow, single-branch checkout on the server and fetches the branch tip at the start of every index run — or the moment someone pushes, if you add a webhook.

Works with any HTTPS git server: GitHub, GitLab, Bitbucket, Gitea, Forgejo, Codeberg and self-hosted instances.

Adding one

Add source → Git repository

Field Example Notes
Name api-docs The mount prefix. Immutable
Repository URL https://github.com/org/repo.git HTTP(S) only, HTTPS strongly recommended — SSH remotes are not supported
Branch main The single branch that is checked out
Subdirectory docs Optional. Index only this folder of the repository
Access token – Only for private repositories. Stored encrypted, never shown again
Username – Only needed for a GitLab deploy token (gitlab+deploy-token-N) or a Bitbucket Cloud API token (your Bitbucket username or x-bitbucket-api-token-auth)
File types .md, .mdx .txt, .html/.htm, .csv, .docx and .pdf can be selected too

Once the source is saved, press Test on its row (or Test connection in its Edit dialog): it lists the remote’s refs without cloning and answers with the branch’s current commit, so a wrong URL, branch or token is obvious. It tests the source as saved, not the unsaved form: after changing the token or branch in Edit, save first, then test.

With a subdirectory set, a file at docs/guides/install.md in the repository is indexed as api-docs/guides/install.md — the subdirectory itself does not appear in the path.

A git server on the internal network

The repository URL is typed by an editor and the server connects to it, so where a git source may connect is bounded. Every request the source makes — the ref list that Test reads, the clone, each update and every redirect — goes through the same address check as a Confluence source, run on the address the connection actually uses, after DNS:

  • Always refused: loopback and link-local addresses (the cloud metadata address 169.254.169.254 included), unspecified and multicast addresses, and the reserved 240.0.0.0/4 block. No setting lifts these. An IPv6 address that carries an IPv4 one (IPv4-mapped, translated, compatible and NAT64 forms) is judged as that IPv4 address.
  • Refused unless allowed: private addresses — 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fc00::/7, the deprecated site-local fec0::/10, 100.64.0.0/10 and 198.18.0.0/15.

At most 5 redirects are followed, and the access token is dropped when a redirect leads to another origin (scheme, host or port).

Upgrading to 0.3.1. Before 0.3.1 a git source could clone from any address. A git server on an internal address — a self-hosted Gitea, for example — stops working after the upgrade until its host is listed in GIT_ALLOWED_HOSTS (host names, comma-separated, no scheme, port or path), set in .env, and the instance is restarted:

GIT_ALLOWED_HOSTS=git.corp.example

A refusal names the rule on the source’s row and in the Test message, for example:

refused: `git.corp.example` resolves to a private address; list `git.corp.example` in GIT_ALLOWED_HOSTS to allow it

No request is sent in that case, so no token leaves. The list lifts only the private-address rule, and only for git sources; SOURCE_ALLOWED_HOSTS and WEB_ALLOWED_HOSTS are separate lists and do not admit a host here. On Helm the key is config.gitAllowedHosts (see Running on Kubernetes).

How large a response may be

Each response body the git client reads is limited, so a remote that answers with far more than expected cannot exhaust the instance’s memory. The limit applies to the ref list and to the pack of a clone or update, each on its own, not to a whole sync together. It is set by GIT_MAX_PACK_BYTES, an integer number of bytes:

  • Default: 268435456 (256 MiB). The lowest accepted value is 1048576 (1 MiB); a smaller one is a configuration error at start-up.
  • Memory: the git library holds a pack in memory and copies it twice, so a fetch can use about twice the limit (roughly 512 MiB at the default) on top of what the instance already uses. The limit caps the response, not the memory.
  • When it is exceeded: the response is cut off, the sync fails, and the error on the source’s row names the setting:
git remote response is larger than GIT_MAX_PACK_BYTES (268435456 bytes); raise GIT_MAX_PACK_BYTES or point the source at a smaller repository

A large repository needs a higher value, and the memory limit of the container has to go up with it: keep it well above twice the new value plus the instance’s own footprint — twice is a lower bound, because parsing the pack adds to it. Set it in .env and restart the instance:

GIT_MAX_PACK_BYTES=536870912

On Helm the key is config.gitMaxPackBytes; leaving it null keeps the default (see Running on Kubernetes). Available from 0.3.2.

Private repositories

Paste an access token into Access token. It is encrypted with SECRET_KEY (AES-256-GCM) before it is stored, is never returned by the API and is never shown again — the dialog only says that a token exists. To replace it, paste a new one; to remove it, tick Remove the stored token.

SECRET_KEY must be set (32+ characters, e.g. openssl rand -hex 32) before a source can store a token. See Configuration.

The username sent with the token depends on the provider and is detected from the URL:

Provider Username used Token types
GitHub x-access-token Classic PAT, fine-grained PAT, App installation token
GitLab oauth2 OAuth tokens, personal and project access tokens
Bitbucket Cloud x-token-auth Repository and workspace access tokens
Bitbucket Cloud (API token) your Bitbucket username, or x-bitbucket-api-token-auth Put it in the Username field — the default x-token-auth is for repository/workspace access tokens only
Gitea / Forgejo / Codeberg / other token, or whatever you type in Username

Never paste credentials into the URL itself (https://user:token@host/…). They are not used to authenticate — a private repository with only a URL token fails to clone, as it would with no token at all. Since 0.3.1 the user:token@ part is also removed from the URL before it is stored, and from the source view and exports; migration 0020 cleans up rows stored earlier. Enter the token in the token field; that is the only place it is read from. A token that was ever pasted into a URL may still sit in a shell history or a log, so revoke it and issue a new one.

Read access to the repository’s contents is enough — the narrowest credential each provider offers:

Provider Narrowest token Username
GitHub A fine-grained PAT limited to the repository with Contents: read, or a GitHub App installation token leave empty (x-access-token)
GitLab A project deploy token with read_repository the token’s generated username, e.g. gitlab+deploy-token-42 — the default oauth2 is refused for a deploy token
Bitbucket Cloud A repository access token with Repositories: read leave empty (x-token-auth)
Gitea / Forgejo An access token with read scope on the repository your username, or leave empty (token)

A GitHub wiki

A GitHub wiki is a git repository of its own, so it can be a source. Its clone URL is the repository’s with .wiki added:

https://github.com/YOUR-USERNAME/YOUR-REPOSITORY.wiki.git

Put that in Repository URL. GitHub creates the wiki repository when the first page is made, so the source has nothing to read before then. Set Branch to the wiki repository’s default branch — check it there rather than assume it. A private repository’s wiki is visible only to people with access to the repository, so the same Access token applies; a public one needs none.

Reaching a repository only available over SSH

Git is read over HTTP(S) only. SSH remotes (ssh://…, git@host:path) are not accepted and no SSH key can be stored. For a repository reachable only over SSH, mirror it yourself: clone or mirror it on the host, inside ALLOWED_DOC_ROOTS, keep it current on your own schedule (a cron job running git pull, or your CI), and add that directory as a Local Directory Source instead. It is then a folder like any other — no push webhook, no branch or subdirectory setting, no Test connection — and exactly as fresh as your schedule keeps it.

How syncing works

At the start of every index run:

  1. If there is no checkout yet, clone it: shallow (depth=1), single branch, no tags.
  2. Otherwise fetch the branch tip. If it moved, point the local branch at it and check out.
  3. If anything goes wrong, fall back to a fresh clone.
  4. Verify the subdirectory still exists on that branch.

The commit currently checked out is shown on the source row (main @ a1b2c3d). Checkouts live under DATA_DIR, so they survive restarts and are deleted with the source.

Automatic re-indexing on push

Every git source gets its own webhook URL and secret, shown while editing the source:

POST http://<your-host>/api/webhooks/git/<source-id>

Add it as a push webhook in the repository settings with that secret, and every push to the branch queues a re-index. Full instructions and provider-by-provider screenshots of the fields: Push Webhooks.

Common problems

Symptom Cause and fix
The operation was canceled. on Test, or on the source’s row after a sync, when a token is set, or HTTP Error: 401 … when none is The git host answered 401: the token is missing, rejected, expired, or lacking read access. Leave Username empty for a Bitbucket repository or workspace access token (the default x-token-auth is for those only); a Bitbucket Cloud API token needs your Bitbucket username or x-bitbucket-api-token-auth, and a GitLab deploy token its generated gitlab+deploy-token-N username
Connected, but branch “…” was not found on the remote (on Test) The branch name is wrong, or the default branch is master rather than main
Subdirectory “…” does not exist in the repository The path is relative to the repository root and is checked against the branch you configured
Documents disappeared after a failed sync They do not — a source that cannot be read keeps its documents. Fix the cause and press Sync
git remote response is larger than GIT_MAX_PACK_BYTES … The repository’s pack is over the response limit. Raise GIT_MAX_PACK_BYTES together with the container’s memory limit — see How large a response may be
SSH URL rejected Only HTTP(S) is supported. Use the HTTPS URL with a token

Notes and limits

  • Only one branch per source. To index two branches, add two sources with different names.
  • Very large repositories clone more slowly than the native git binary would; depth=1, a single branch and a subdirectory keep it comfortable for documentation.
  • Submodules are not fetched.
  • The repository is never written to; Contextator only ever fetches.

Arrow keys to move, Enter to open.