# Document Sources

> The nine kinds of source a project can hold, the mount name every path is prefixed with, and how syncing and failures work across all of them.

- Updated: 2026-10-30
- Source: https://contextator.com/en/docs/document-sources/
- Language: en-US
- Author: Muhammet Şafak

---
A project's documentation rarely lives in one place. A **source** is one of those places, and a project
can have as many as it needs — they are merged into a single searchable endpoint.

| Type | What it is | Kept in sync by | Page |
|------|-----------|-----------------|------|
| **Local directory** | A folder mounted on the server, scanned in place. Nothing is copied | Reading it at index time | [Local Directory Source](/en/docs/local-directory-source/) |
| **Git repository** | A shallow checkout of one branch, optionally just one subdirectory | `git fetch` at the start of every run, or a push webhook | [Git Repository Source](/en/docs/git-repository-source/) |
| **Upload** | Files, folders and archives (`.zip`, `.tar`, `.tar.gz`/`.tgz`, `.rar`) unpacked on the server | Nothing to sync — you upload again when it changes | [Upload Source](/en/docs/upload-source/) |
| **Notion** | Pages shared with an internal integration, rendered to Markdown | The Notion API, re-rendering only pages that changed | [Notion](/en/docs/notion/) |
| **Confluence** | Every space a Confluence Cloud or Data Center account can read, or the ones you name, rendered to Markdown | The Confluence REST API, re-rendering only pages whose version changed; Data Center can also take a webhook | [Confluence](/en/docs/confluence/) |
| **Documentation site** | A public site's own sitemap, `llms.txt` or a crawl, converted to Markdown | Re-fetching the whole site; a `sitemap.xml` with `<lastmod>` dates can be checked for changes in two requests first | [Documentation Site Source](/en/docs/documentation-site-source/) |
| **S3 bucket** | The objects under one prefix of an S3-compatible bucket (AWS S3, MinIO, Ceph, Cloudflare R2), keyed by the prefix-relative key | Listing the prefix, downloading only objects whose ETag or size changed | [S3 buckets](#s3-buckets) |
| **Google Drive** | The files under folders shared with a Google service account; Docs become Markdown, Sheets `.xlsx`, Slides `.pptx` | One `changes.list` request for the check; a run fetches only files whose Google version or checksum changed | [Google Drive](#google-drive) |
| **GitHub issues** | The issues of one repository, and optionally its Discussions, one document each with its comments | One conditional request for the check; a run reads only what changed since the last one | [GitHub issues](#github-issues-and-discussions) |

An **Obsidian vault** is a folder of Markdown, so it arrives as a local directory or as an upload —
with one setting flipped. See [Obsidian Vaults](/en/docs/obsidian-vaults/).

## The mount name

Every source has a **name**, and that name is the prefix of every path it contributes:

```
source "handbook"  contains  install.md   →  indexed as  handbook/install.md
source "api-repo"  contains  install.md   →  indexed as  api-repo/install.md
```

This is why two sources can hold the same file name without colliding, and why every search result tells
you which source it came from. It is also why **the name cannot be changed after creation** — every
indexed path and every path an agent has seen contains it. Everything else about a source can be edited.

Names follow the same rule as project names: 1–63 characters, lowercase letters, digits, `-` and `_`, starting
with a letter or digit.

## Adding one

**Add source** on a project opens a dialog with a tab per kind. Common fields:

| Field | Notes |
|-------|-------|
| **Name** | The mount prefix. Immutable |
| **Label** | Free text shown in the list. Optional, changeable |
| **Content type** | `Plain Markdown / text`, `Obsidian vault`, `Notion export` or `OpenAPI / Swagger` — the last reads `.yaml`, `.yml` and `.json` as specifications and turns one into a document per operation |
| **File types** | Which extensions to index: `.md` and `.mdx` by default, optionally `.txt`, `.html`/`.htm`, `.csv`, `.docx` and `.pdf`. A source created through the Admin API starts from its own type's default instead: `.md`, `.mdx`, `.txt` for upload, `.md` for Notion and Confluence, `.html`, `.md`, `.txt` for a documentation site |
| **Language** | Optional. Which language the keyword half of search reads this source in; set, PostgreSQL stems its words. Unset by default — no stemming, which keeps identifiers such as `HLY-4015` intact. Changing it re-indexes this source |
| **Version** | Optional free text (up to 64 characters), stamped on every document the source indexes so `search_docs` can ask for one release. Several sources may share a version. Changing it re-indexes this source |
| **Sync every** | A new source starts on the instance's `SYNC_DEFAULT_INTERVAL_MINUTES` (an hour unless the operator changed it; `0` makes new sources start on *Never — only when asked*). The choices are *Never*, 5, 15, 30 minutes, hour, 6, 12 hours, day or week: the server checks whether the source changed and indexes only if it did |
| **Index now** | Queue a run as soon as the source is saved |

Git, Notion, Confluence, documentation-site, S3, Google Drive and GitHub sources can be checked without indexing anything: once a source is
saved, press **Test** on its row (or **Test connection** in its **Edit** dialog). The dialog for a new source has
no such button. The test reads the source as it is saved, not the unsaved form: save an edit before testing it.

**Moving a source's address needs its credential again.** A Confluence source's *Site URL*, a git source's *URL*, an S3
source's *Endpoint* and a GitHub source's *API address* are where the stored credential is sent. An editor can change them
but cannot read the credential, so since 0.6.0 an edit that moves the **host** (scheme, name or port) while a credential is
stored is refused with `400 invalid_request`, naming the setting, unless the same request also supplies the credential
again or removes it (`secret: null` over the API; type the token again in the dashboard). Changing only the path on the same
host, editing anything else, and editing a source that stores no credential (a public repository) are not affected.

## How sources are synced

Every source is synced **at the start of every index run**, one after another, and then scanned:

1. git sources fetch the branch tip; Notion sources pull changed pages; Confluence sources pull pages whose
   version changed; documentation-site sources re-fetch the site; S3, Google Drive and GitHub sources download only what changed; local and upload sources have
   nothing to fetch.
2. Each source's directory is walked for the file types it selected.
3. Paths are prefixed with the source name and handed to the indexer.

Pressing **Sync** on a single source row queues the same run as **Re-index** in the header — there is
one queue per project, not per source.

A scheduled source is not always synced when it comes due: the server first checks whether it changed and skips the sync if not. That check cannot see everything — a Notion page unshared from the integration moves nothing it looks at — so a source whose last successful sync is older than `SYNC_MAX_SKIP_HOURS` (24 hours by default; `0` turns it off) is synced anyway. See [Configuration](/en/docs/configuration/) and [Notion](/en/docs/notion/).

## S3 buckets

An **S3 bucket** source (new in 0.6.0) mirrors the objects under one **prefix** whose extension the source reads (`.md`,
`.mdx` and `.txt` by default) and indexes them like any other directory. The prefix is removed from the names: with prefix
`guides`, the object `guides/nested/a.md` in a source named `handbook` is `handbook/nested/a.md`. Requests are signed with
AWS Signature V4 and sent without an SDK.

- **Endpoint.** Leave it empty for AWS S3 (`https` only, region from **Region**). For MinIO, Ceph, R2 or another
  S3-compatible server give its address. A private endpoint is refused unless its host name is listed in
  `SOURCE_ALLOWED_HOSTS`; loopback and link-local stay refused whatever is listed.
- **The key pair** is one credential, an access key ID and a secret access key, stored encrypted with `SECRET_KEY` and
  never returned or shown again. Temporary credentials (a session token) are not accepted, and neither are instance-profile
  or IRSA credentials. Grant the key `s3:ListBucket` on the bucket and `s3:GetObject` on the prefix, and nothing else.
- **What a check costs.** A scheduled check lists the prefix and downloads nothing; the source runs only when the listing
  changed. A run fetches only objects whose ETag or size changed and removes the documents of objects that are gone, but only
  after every download has succeeded. Give the source the narrowest prefix that holds its documents.
- **Bounds.** One source indexes at most `S3_MAX_OBJECTS` objects (5 000 by default), the first ones in key order, and the
  run's note says in capitals that the rest are not indexed. A failed run moves nothing, so the next check runs it again.
- **Export and import.** A project export carries the source's settings and not its key pair; an imported S3 source arrives
  unscheduled and needs its credential entered again.

## Google Drive

A **Google Drive** source (new in 0.6.0) indexes the trees under the folders you name that are **shared with a Google
service account**. The account is the only identity the source has: no user signs in, and the only scope requested is
`drive.readonly`.

1. In a Google Cloud project, enable the **Google Drive API**.
2. Create a **service account** and add a key of type **JSON**; Google lets you download it once.
3. Share each folder with the account's e-mail address (the `client_email` of the key file) as *Viewer*.
4. In the dashboard choose **Add source → Google Drive**, paste the folder links or ids, and load or paste the key file.

- **The key** is stored encrypted with `SECRET_KEY` and never returned or shown again. Use one service account per source:
  the list of folders bounds what is indexed, not what the account can see. A Google Cloud organisation created on or after
  3 May 2024 blocks key creation by default until an administrator relaxes the policy; workload identity federation and user
  OAuth are not supported.
- **Export formats.** A Google Doc is exported as Markdown, a Sheet as `.xlsx` and a Slides deck as `.pptx`; the last two are
  converted like any upload. A Google file over Google's 10 MB export limit is refused by name and the run goes on. Other
  Google files (Forms, Drawings, Sites) and shortcuts are skipped and counted in the run's note.
- **What a check costs.** One `changes.list` request; nothing is listed or downloaded. A run walks the folders and fetches
  only files whose Google version or checksum changed.
- **Bounds.** One source indexes at most `GDRIVE_MAX_FILES` files (5 000 by default), and the note says in capitals that the
  rest are not indexed.
- **Network.** There is no Google client library. Requests go only to `oauth2.googleapis.com` and `www.googleapis.com`,
  over `https`, through the egress guard, and no allowlist setting applies.
- **Shared drives are not verified:** treat them as unsupported.
- **Export and import.** A project export carries the folders and settings, not the key; an imported Drive source needs its
  key entered again.

## GitHub issues and Discussions

A **GitHub issues** source (new in 0.6.0) indexes the issues of one repository, and optionally its Discussions, as one
document each with the comments in the same document: `<name>/issues/<number>.md` and `<name>/discussions/<number>.md`. It is
a different source from the **Git repository** one: the repository's files and its wiki come through `git`, and this source
never reads either. Add both when you want both.

- **In scope.** Closed issues are left out unless you tick **Closed issues too**; pull requests are never indexed. Every
  Discussion is a document, open or closed. Issue text routinely carries names and pasted credentials, and it is indexed as
  written.
- **The token is optional.** Without one the source reads a **public** repository's issues under GitHub's anonymous rate
  limit. With a token it also reads private repositories and may turn on **Discussions**, which GitHub answers only for a
  signed-in caller: ticking Discussions with no token is refused with `400 invalid_request`. A fine-grained personal access
  token with read access to *Issues* (and *Discussions*) works; GitHub Apps are not supported. The token is stored encrypted.
- **GitHub Enterprise Server.** Set the **API address** to the server's REST base (`https://ghe.example.com/api/v3`), over
  `https` only. A server on a private address must be named in `SOURCE_ALLOWED_HOSTS`.
- **What a check costs.** One conditional request (`ETag`); GitHub answers `304` when nothing changed. A run reads only what
  changed since the last one.
- **Deletions are found on full runs.** A deleted or transferred issue stays searchable until the next full run: the first
  run, a run after the settings changed, or one whose last full run is older than `SYNC_MAX_SKIP_HOURS` (24 hours when that is off).
- **Bounds and limits.** One source indexes at most `GITHUB_MAX_ISSUES` issues and `GITHUB_MAX_DISCUSSIONS` Discussions
  (5 000 each by default). A rate limit ends the run as `error` with the time it resets, and moves no cursor.
- **Export and import.** A project export carries the repository and settings, no token; an imported source arrives
  unscheduled, and a tokenless one runs as a public source.

## When a source fails

A source that cannot sync **reports on its own row** and the others still index. The project's status
becomes `error` with a summary:

```
2/3 sources synced; notion: API token is invalid
```

Crucially, a source whose content could not be read **at all** keeps the documents it had already
contributed. A revoked Notion share or an unreachable git host makes a source stale, never empty. Fix
the cause and press **Sync** on that row.

## Editing and removing

Changing a setting that alters what the source yields — path, branch, subdirectory, file types, content
type — queues an index run automatically, so a source is never silently stale.

Deleting a source removes its documents, its chunks and the files it materialised. It is refused while
the project is indexing.

## What gets indexed

Only the file types the source selected: `.md` and `.mdx` by default, and optionally `.txt`,
`.html`/`.htm`, `.csv`, `.docx` and `.pdf`. **Everything becomes Markdown on the way in** — the
conversion happens once, at the edge, so the chunker, the embedder and `read_document` see one format.
`.yaml`, `.yml` and `.json` are not on that list: they are readable only by the OpenAPI / Swagger
content type, and a specification it reads becomes one document per operation rather than one document.

Always skipped:

- dotfiles and dot-directories (`.git/`, `.obsidian/`, …)
- `node_modules`, `dist`, `build`, `vendor`, `__pycache__`
- symbolic links, whatever they point at
- anything matching `IGNORE_GLOBS`

| Type | What it becomes | Kept | Lost |
|------|-----------------|------|------|
| `.md`, `.mdx`, `.txt` | itself, unchanged | everything | nothing |
| `.html`, `.htm` | Markdown via turndown + GFM | headings, lists, tables, code, links, `<title>` | scripts, stylesheets, `svg`, embedded image data (the `alt` text stays) |
| `.docx` | Markdown via mammoth, then the same converter | Word's own heading styles, numbered and bulleted lists, tables, links | images, footnotes, comments, tracked changes |
| `.csv` | one GFM table, `## Rows n–m` sections every 200 rows | the header above every section, quoted commas and newlines, `;`/tab/pipe delimiters | nothing of the data; cell newlines become `<br>` |
| `.pdf` | Markdown reconstructed from glyph positions | headings by font size, paragraphs rejoined across line ends and de-hyphenated, bullet and numbered lists, column-aligned tables, two-column reading order, running heads and feet dropped | footnotes, figures, and any table whose columns are not aligned |

Images and other binaries are not indexed. There is no OCR either, so a file that cannot be converted —
a scanned PDF, an encrypted one, a Word file that is all images, a damaged file of any of these types, a
page or spreadsheet that converts to no text at all — is **refused, not indexed**: the reason naming the
file is shown on its source row, and the rest of the source indexes normally. A refusal never fails the
sync or the project: on an incremental run the file keeps whatever document it already had, and a
rebuild simply leaves it out of the new generation, the same rule applied to a source that cannot be
read at all.

Conversion runs on its own worker thread, not the thread serving the dashboard and the MCP endpoint, so
one slow file never blocks a search and a parser that exhausts its heap fails only that file. Size caps
bound what a single file may cost while it converts — `MAX_CONVERTED_FILE_BYTES` for a document,
`MAX_PDF_PAGES` for a PDF that claims an implausible page count, `MAX_DOCX_UNPACKED_BYTES` against a zip
bomb inside a `.docx` — and `CONVERSION_TIMEOUT_MS` / `CONVERSION_IDLE_MS` bound how long a file, and an
idle worker, are allowed to run. Full defaults and what each one stops: [Configuration](/en/docs/configuration/#documents).

**Where a long PDF lands against `MAX_STORED_DOCUMENT_BYTES`** (1 MB of UTF-8, past which the stored
prefix is what `read_document` serves and `content_truncated` is set — the document stays fully
searchable regardless, because the cut is on the stored text and not on the chunks). Measured on
generated manuals of 80, 200, 600 and 1600 dense pages (42 lines of about 95 characters, a running head
and foot, a chapter heading every tenth page):

| Pages | Markdown | Per page | Extraction |
|-------|----------|----------|------------|
| 80 | 316 KiB | 4.0 KiB | 0.18 s |
| 200 | 795 KiB | 4.0 KiB | 0.21 s |
| 600 | 2393 KiB | 4.0 KiB | 0.61 s |
| 1600 | 6414 KiB | 4.0 KiB | 1.73 s |

The cap bites at roughly 250 dense pages — looser real-world manuals stretch that to 300–400 pages.
Extraction costs about a millisecond a page and happens once, after the content hash says the file
changed. What is capped on the way in, before any of this, is the file itself: `UPLOAD_MAX_FILE_BYTES` (50 MB by default) on an upload, and
`MAX_CONVERTED_FILE_BYTES` (32 MiB by default) for any source when a PDF or DOCX is converted.
