A project’s documentation rarely lives in one place. A source is one of those places, and a project can have as many as it needs — they are merged into a single searchable endpoint.
| Type | What it is | Kept in sync by | Page |
|---|---|---|---|
| Local directory | A folder mounted on the server, scanned in place. Nothing is copied | Reading it at index time | Local Directory Source |
| Git repository | A shallow checkout of one branch, optionally just one subdirectory | git fetch at the start of every run, or a push webhook |
Git Repository Source |
| Upload | Files, folders and archives (.zip, .tar, .tar.gz/.tgz, .rar) unpacked on the server |
Nothing to sync — you upload again when it changes | Upload Source |
| Notion | Pages shared with an internal integration, rendered to Markdown | The Notion API, re-rendering only pages that changed | Notion |
| Confluence | Every space a Confluence Cloud or Data Center account can read, or the ones you name, rendered to Markdown | The Confluence REST API, re-rendering only pages whose version changed; Data Center can also take a webhook | Confluence |
| Documentation site | A public site’s own sitemap, llms.txt or a crawl, converted to Markdown |
Re-fetching the whole site; a sitemap.xml with <lastmod> dates can be checked for changes in two requests first |
Documentation Site Source |
| S3 bucket | The objects under one prefix of an S3-compatible bucket (AWS S3, MinIO, Ceph, Cloudflare R2), keyed by the prefix-relative key | Listing the prefix, downloading only objects whose ETag or size changed | S3 buckets |
| Google Drive | The files under folders shared with a Google service account; Docs become Markdown, Sheets .xlsx, Slides .pptx |
One changes.list request for the check; a run fetches only files whose Google version or checksum changed |
Google Drive |
| GitHub issues | The issues of one repository, and optionally its Discussions, one document each with its comments | One conditional request for the check; a run reads only what changed since the last one | GitHub issues |
An Obsidian vault is a folder of Markdown, so it arrives as a local directory or as an upload — with one setting flipped. See Obsidian Vaults.
The mount name
Every source has a name, and that name is the prefix of every path it contributes:
source "handbook" contains install.md → indexed as handbook/install.md
source "api-repo" contains install.md → indexed as api-repo/install.md
This is why two sources can hold the same file name without colliding, and why every search result tells you which source it came from. It is also why the name cannot be changed after creation — every indexed path and every path an agent has seen contains it. Everything else about a source can be edited.
Names follow the same rule as project names: 1–63 characters, lowercase letters, digits, - and _, starting
with a letter or digit.
Adding one
Add source on a project opens a dialog with a tab per kind. Common fields:
| Field | Notes |
|---|---|
| Name | The mount prefix. Immutable |
| Label | Free text shown in the list. Optional, changeable |
| Content type | Plain Markdown / text, Obsidian vault, Notion export or OpenAPI / Swagger — the last reads .yaml, .yml and .json as specifications and turns one into a document per operation |
| File types | Which extensions to index: .md and .mdx by default, optionally .txt, .html/.htm, .csv, .docx and .pdf. A source created through the Admin API starts from its own type’s default instead: .md, .mdx, .txt for upload, .md for Notion and Confluence, .html, .md, .txt for a documentation site |
| Language | Optional. Which language the keyword half of search reads this source in; set, PostgreSQL stems its words. Unset by default — no stemming, which keeps identifiers such as HLY-4015 intact. Changing it re-indexes this source |
| Version | Optional free text (up to 64 characters), stamped on every document the source indexes so search_docs can ask for one release. Several sources may share a version. Changing it re-indexes this source |
| Sync every | A new source starts on the instance’s SYNC_DEFAULT_INTERVAL_MINUTES (an hour unless the operator changed it; 0 makes new sources start on Never — only when asked). The choices are Never, 5, 15, 30 minutes, hour, 6, 12 hours, day or week: the server checks whether the source changed and indexes only if it did |
| Index now | Queue a run as soon as the source is saved |
Git, Notion, Confluence, documentation-site, S3, Google Drive and GitHub sources can be checked without indexing anything: once a source is saved, press Test on its row (or Test connection in its Edit dialog). The dialog for a new source has no such button. The test reads the source as it is saved, not the unsaved form: save an edit before testing it.
Moving a source’s address needs its credential again. A Confluence source’s Site URL, a git source’s URL, an S3
source’s Endpoint and a GitHub source’s API address are where the stored credential is sent. An editor can change them
but cannot read the credential, so since 0.6.0 an edit that moves the host (scheme, name or port) while a credential is
stored is refused with 400 invalid_request, naming the setting, unless the same request also supplies the credential
again or removes it (secret: null over the API; type the token again in the dashboard). Changing only the path on the same
host, editing anything else, and editing a source that stores no credential (a public repository) are not affected.
How sources are synced
Every source is synced at the start of every index run, one after another, and then scanned:
- git sources fetch the branch tip; Notion sources pull changed pages; Confluence sources pull pages whose version changed; documentation-site sources re-fetch the site; S3, Google Drive and GitHub sources download only what changed; local and upload sources have nothing to fetch.
- Each source’s directory is walked for the file types it selected.
- Paths are prefixed with the source name and handed to the indexer.
Pressing Sync on a single source row queues the same run as Re-index in the header — there is one queue per project, not per source.
A scheduled source is not always synced when it comes due: the server first checks whether it changed and skips the sync if not. That check cannot see everything — a Notion page unshared from the integration moves nothing it looks at — so a source whose last successful sync is older than SYNC_MAX_SKIP_HOURS (24 hours by default; 0 turns it off) is synced anyway. See Configuration and Notion.
S3 buckets
An S3 bucket source (new in 0.6.0) mirrors the objects under one prefix whose extension the source reads (.md,
.mdx and .txt by default) and indexes them like any other directory. The prefix is removed from the names: with prefix
guides, the object guides/nested/a.md in a source named handbook is handbook/nested/a.md. Requests are signed with
AWS Signature V4 and sent without an SDK.
- Endpoint. Leave it empty for AWS S3 (
httpsonly, region from Region). For MinIO, Ceph, R2 or another S3-compatible server give its address. A private endpoint is refused unless its host name is listed inSOURCE_ALLOWED_HOSTS; loopback and link-local stay refused whatever is listed. - The key pair is one credential, an access key ID and a secret access key, stored encrypted with
SECRET_KEYand never returned or shown again. Temporary credentials (a session token) are not accepted, and neither are instance-profile or IRSA credentials. Grant the keys3:ListBucketon the bucket ands3:GetObjecton the prefix, and nothing else. - What a check costs. A scheduled check lists the prefix and downloads nothing; the source runs only when the listing changed. A run fetches only objects whose ETag or size changed and removes the documents of objects that are gone, but only after every download has succeeded. Give the source the narrowest prefix that holds its documents.
- Bounds. One source indexes at most
S3_MAX_OBJECTSobjects (5 000 by default), the first ones in key order, and the run’s note says in capitals that the rest are not indexed. A failed run moves nothing, so the next check runs it again. - Export and import. A project export carries the source’s settings and not its key pair; an imported S3 source arrives unscheduled and needs its credential entered again.
Google Drive
A Google Drive source (new in 0.6.0) indexes the trees under the folders you name that are shared with a Google
service account. The account is the only identity the source has: no user signs in, and the only scope requested is
drive.readonly.
- In a Google Cloud project, enable the Google Drive API.
- Create a service account and add a key of type JSON; Google lets you download it once.
- Share each folder with the account’s e-mail address (the
client_emailof the key file) as Viewer. - In the dashboard choose Add source → Google Drive, paste the folder links or ids, and load or paste the key file.
- The key is stored encrypted with
SECRET_KEYand never returned or shown again. Use one service account per source: the list of folders bounds what is indexed, not what the account can see. A Google Cloud organisation created on or after 3 May 2024 blocks key creation by default until an administrator relaxes the policy; workload identity federation and user OAuth are not supported. - Export formats. A Google Doc is exported as Markdown, a Sheet as
.xlsxand a Slides deck as.pptx; the last two are converted like any upload. A Google file over Google’s 10 MB export limit is refused by name and the run goes on. Other Google files (Forms, Drawings, Sites) and shortcuts are skipped and counted in the run’s note. - What a check costs. One
changes.listrequest; nothing is listed or downloaded. A run walks the folders and fetches only files whose Google version or checksum changed. - Bounds. One source indexes at most
GDRIVE_MAX_FILESfiles (5 000 by default), and the note says in capitals that the rest are not indexed. - Network. There is no Google client library. Requests go only to
oauth2.googleapis.comandwww.googleapis.com, overhttps, through the egress guard, and no allowlist setting applies. - Shared drives are not verified: treat them as unsupported.
- Export and import. A project export carries the folders and settings, not the key; an imported Drive source needs its key entered again.
GitHub issues and Discussions
A GitHub issues source (new in 0.6.0) indexes the issues of one repository, and optionally its Discussions, as one
document each with the comments in the same document: <name>/issues/<number>.md and <name>/discussions/<number>.md. It is
a different source from the Git repository one: the repository’s files and its wiki come through git, and this source
never reads either. Add both when you want both.
- In scope. Closed issues are left out unless you tick Closed issues too; pull requests are never indexed. Every Discussion is a document, open or closed. Issue text routinely carries names and pasted credentials, and it is indexed as written.
- The token is optional. Without one the source reads a public repository’s issues under GitHub’s anonymous rate
limit. With a token it also reads private repositories and may turn on Discussions, which GitHub answers only for a
signed-in caller: ticking Discussions with no token is refused with
400 invalid_request. A fine-grained personal access token with read access to Issues (and Discussions) works; GitHub Apps are not supported. The token is stored encrypted. - GitHub Enterprise Server. Set the API address to the server’s REST base (
https://ghe.example.com/api/v3), overhttpsonly. A server on a private address must be named inSOURCE_ALLOWED_HOSTS. - What a check costs. One conditional request (
ETag); GitHub answers304when nothing changed. A run reads only what changed since the last one. - Deletions are found on full runs. A deleted or transferred issue stays searchable until the next full run: the first
run, a run after the settings changed, or one whose last full run is older than
SYNC_MAX_SKIP_HOURS(24 hours when that is off). - Bounds and limits. One source indexes at most
GITHUB_MAX_ISSUESissues andGITHUB_MAX_DISCUSSIONSDiscussions (5 000 each by default). A rate limit ends the run aserrorwith the time it resets, and moves no cursor. - Export and import. A project export carries the repository and settings, no token; an imported source arrives unscheduled, and a tokenless one runs as a public source.
When a source fails
A source that cannot sync reports on its own row and the others still index. The project’s status
becomes error with a summary:
2/3 sources synced; notion: API token is invalid
Crucially, a source whose content could not be read at all keeps the documents it had already contributed. A revoked Notion share or an unreachable git host makes a source stale, never empty. Fix the cause and press Sync on that row.
Editing and removing
Changing a setting that alters what the source yields — path, branch, subdirectory, file types, content type — queues an index run automatically, so a source is never silently stale.
Deleting a source removes its documents, its chunks and the files it materialised. It is refused while the project is indexing.
What gets indexed
Only the file types the source selected: .md and .mdx by default, and optionally .txt,
.html/.htm, .csv, .docx and .pdf. Everything becomes Markdown on the way in — the
conversion happens once, at the edge, so the chunker, the embedder and read_document see one format.
.yaml, .yml and .json are not on that list: they are readable only by the OpenAPI / Swagger
content type, and a specification it reads becomes one document per operation rather than one document.
Always skipped:
- dotfiles and dot-directories (
.git/,.obsidian/, …) node_modules,dist,build,vendor,__pycache__- symbolic links, whatever they point at
- anything matching
IGNORE_GLOBS
| Type | What it becomes | Kept | Lost |
|---|---|---|---|
.md, .mdx, .txt |
itself, unchanged | everything | nothing |
.html, .htm |
Markdown via turndown + GFM | headings, lists, tables, code, links, <title> |
scripts, stylesheets, svg, embedded image data (the alt text stays) |
.docx |
Markdown via mammoth, then the same converter | Word’s own heading styles, numbered and bulleted lists, tables, links | images, footnotes, comments, tracked changes |
.csv |
one GFM table, ## Rows n–m sections every 200 rows |
the header above every section, quoted commas and newlines, ;/tab/pipe delimiters |
nothing of the data; cell newlines become <br> |
.pdf |
Markdown reconstructed from glyph positions | headings by font size, paragraphs rejoined across line ends and de-hyphenated, bullet and numbered lists, column-aligned tables, two-column reading order, running heads and feet dropped | footnotes, figures, and any table whose columns are not aligned |
Images and other binaries are not indexed. There is no OCR either, so a file that cannot be converted — a scanned PDF, an encrypted one, a Word file that is all images, a damaged file of any of these types, a page or spreadsheet that converts to no text at all — is refused, not indexed: the reason naming the file is shown on its source row, and the rest of the source indexes normally. A refusal never fails the sync or the project: on an incremental run the file keeps whatever document it already had, and a rebuild simply leaves it out of the new generation, the same rule applied to a source that cannot be read at all.
Conversion runs on its own worker thread, not the thread serving the dashboard and the MCP endpoint, so
one slow file never blocks a search and a parser that exhausts its heap fails only that file. Size caps
bound what a single file may cost while it converts — MAX_CONVERTED_FILE_BYTES for a document,
MAX_PDF_PAGES for a PDF that claims an implausible page count, MAX_DOCX_UNPACKED_BYTES against a zip
bomb inside a .docx — and CONVERSION_TIMEOUT_MS / CONVERSION_IDLE_MS bound how long a file, and an
idle worker, are allowed to run. Full defaults and what each one stops: Configuration.
Where a long PDF lands against MAX_STORED_DOCUMENT_BYTES (1 MB of UTF-8, past which the stored
prefix is what read_document serves and content_truncated is set — the document stays fully
searchable regardless, because the cut is on the stored text and not on the chunks). Measured on
generated manuals of 80, 200, 600 and 1600 dense pages (42 lines of about 95 characters, a running head
and foot, a chapter heading every tenth page):
| Pages | Markdown | Per page | Extraction |
|---|---|---|---|
| 80 | 316 KiB | 4.0 KiB | 0.18 s |
| 200 | 795 KiB | 4.0 KiB | 0.21 s |
| 600 | 2393 KiB | 4.0 KiB | 0.61 s |
| 1600 | 6414 KiB | 4.0 KiB | 1.73 s |
The cap bites at roughly 250 dense pages — looser real-world manuals stretch that to 300–400 pages.
Extraction costs about a millisecond a page and happens once, after the content hash says the file
changed. What is capped on the way in, before any of this, is the file itself: UPLOAD_MAX_FILE_BYTES (50 MB by default) on an upload, and
MAX_CONVERTED_FILE_BYTES (32 MiB by default) for any source when a PDF or DOCX is converted.