# Embedding Models

> The default local model, switching to OpenAI, changing models safely, how search actually scores results, and running fully air-gapped.

- Updated: 2026-10-10
- Source: https://contextator.com/en/docs/embedding-models/
- Language: en-US
- Author: Muhammet Şafak

---
An embedding model turns text into a vector so that similar meanings land close together. Contextator
runs one **locally on your CPU** by default — no API key, no account, nothing leaving the machine — and
can use OpenAI instead if you prefer.

## The default

| | |
|---|---|
| Model | `Xenova/multilingual-e5-small` |
| Dimensions | 384 |
| Languages | 100, including Turkish. Ask in the language the answer is written in: matching across languages is a measured limit of the model, not a switch |
| Download | ~470 MB once (`fp32`), cached in a Docker volume |
| Runs on | CPU. No GPU is used or needed |

### Smaller and faster options

```bash
EMBEDDING_DTYPE=q8                       # same model, ~120 MB, quantized
```

```bash
EMBEDDING_MODEL=Xenova/all-MiniLM-L6-v2  # English-only, ~90 MB, also 384 dimensions
EMBEDDING_DIMENSIONS=384
```

Any transformers.js feature-extraction model works. `EMBEDDING_DIMENSIONS` **must** match what the model
produces — if it does not, the server tells you the correct number.

## Using OpenAI instead

```bash
EMBEDDING_PROVIDER=openai
OPENAI_API_KEY=sk-…
OPENAI_EMBEDDING_MODEL=text-embedding-3-small
EMBEDDING_DIMENSIONS=1536
RESET_VECTORS=1        # only for the first start after the change — see below
```

Better retrieval on long chunks and a much larger input window, at the cost of an API call per chunk
indexed and per query asked. Your documentation text is sent to OpenAI; if that is not acceptable, stay
local, or use a server of your own, below.

## Using a server of your own

`EMBEDDING_PROVIDER=openai` speaks to any server that offers an OpenAI-compatible `/embeddings`
endpoint. Set `EMBEDDING_BASE_URL` to it, with the path, and name the model that server serves in
`OPENAI_EMBEDDING_MODEL`. `OPENAI_API_KEY` is optional with a base URL; when set, it is sent as a bearer
token. `EMBEDDING_DIMENSIONS` must match the model's vector size.

```bash
# Ollama
EMBEDDING_PROVIDER=openai
EMBEDDING_BASE_URL=http://localhost:11434/v1
OPENAI_EMBEDDING_MODEL=nomic-embed-text
EMBEDDING_DIMENSIONS=768

# LM Studio: its local server
EMBEDDING_BASE_URL=http://localhost:1234/v1

# Text Embeddings Inference, started with -p 8080:80
EMBEDDING_BASE_URL=http://localhost:8080/v1

# vLLM: the host and port you started `vllm serve` on
EMBEDDING_BASE_URL=http://gpu-box:<port>/v1
EMBEDDING_REQUEST_DIMENSIONS=always   # only for a model that shortens its vectors on request
```

Set the model name and `EMBEDDING_DIMENSIONS` to the model you loaded in every case; the lines above show
only what differs. `RESET_VECTORS=1` is needed only when the database already records a vector dimension and the new one differs from it, as moving a 384-dimension install to a 768-dimension model does; a first install needs none. Set it for one start only — see
[Changing the model](#changing-the-model). `EMBEDDING_REQUEST_DIMENSIONS` is `auto` by default and sends `dimensions` only for
`text-embedding-3-*`; use `always` for a model that accepts it and `never` for a server that rejects it.

**Counting tokens with the model's own tokenizer.** Chunks are sized by an estimate (characters ÷ 4) unless
`EMBEDDING_TOKENIZER` names a Hugging Face model whose `tokenizer.json` matches the served model. Only the tokenizer is
downloaded, into `MODEL_CACHE_DIR`, and startup stops if it cannot be loaded. With it, an endpoint cuts the same chunks
the local provider cuts from the same model. It is part of the model id, so setting or changing it re-indexes every
project once.

```bash
# multilingual-e5-small behind TEI or vLLM
EMBEDDING_PROVIDER=openai
EMBEDDING_BASE_URL=http://localhost:8080/v1        # TEI; vLLM: http://gpu-box:<port>/v1
OPENAI_EMBEDDING_MODEL=intfloat/multilingual-e5-small   # the real name, so the e5 prefixes apply
EMBEDDING_DIMENSIONS=384
EMBEDDING_TOKENIZER=Xenova/multilingual-e5-small
```

**The model's name decides the prefixes.** The `query: ` and `passage: ` prefixes of the `multilingual-e5-*` family
are chosen from the name in `OPENAI_EMBEDDING_MODEL`, so the model's real name has to be there. If your server serves the model under another name (a local alias, a path), the name matches no family: the model runs without prefixes and recall quietly drops. The server says so once at startup, at `info` level, unless you have set either prefix variable. Either put the model's real name in `OPENAI_EMBEDDING_MODEL`, or keep your own name and set `EMBEDDING_QUERY_PREFIX="query: "` and `EMBEDDING_PASSAGE_PREFIX="passage: "` yourself; either change re-indexes every project once.

Inside the Contextator container, `localhost` is the container, not your machine: use the address the
container can reach the server on. The endpoint's host and port become part of the model id every project
is stamped with, so pointing at another server re-indexes every project — see
[Changing the model](#changing-the-model). Text sent to such a server stays wherever that server runs.

---

## Changing the model

### Same dimensions (e.g. between the 384-dimension local models)

1. Change `EMBEDDING_MODEL`.
2. Restart.
3. Re-index each project.

Contextator records which model indexed each project. When the server's model differs, the next run is
automatically a **full** re-index, and `search_docs` refuses to search that project until then — with a
message saying exactly that, rather than returning confidently wrong results.

### Different dimensions (e.g. moving to OpenAI's 1536)

The vector column's width is fixed when the database is created, so this is a one-time destructive step:

1. Set the provider, key and `EMBEDDING_DIMENSIONS`.
2. Start **once** with `RESET_VECTORS=1`. Every chunk and document is dropped and the column is re-typed.
3. Remove `RESET_VECTORS` (or set it back to `0`).
4. Re-index every project.

Without the flag, the server refuses to start and prints this instruction:

```
The database was created with EMBEDDING_DIMENSIONS=384 but the current config says 1536.
Either set EMBEDDING_DIMENSIONS=384, or start once with RESET_VECTORS=1
(drops every indexed chunk; all projects must be re-indexed), or wipe the pgdata volume.
```

---

## How search actually works

Search is **hybrid**: meaning and exact wording are searched at once and fused by rank.

1. Your question is embedded with the same model that indexed the documents.
2. PostgreSQL runs two rankings scoped to that project — nearest chunks by cosine similarity over an
   HNSW index, and a `tsvector` keyword match over a GIN index — and combines them with reciprocal rank
   fusion, so an environment variable finds its page as readily as a question does.
3. The top matches are returned with their file path, heading breadcrumb and score.

Scores are cosine similarities: higher is better, and with the default model they sit high and close
together. On the shipped configuration the mean score is `0.881`, and a **wrong** hit sits only about `0.02`
below a correct one. A score is therefore not a truth signal, and nothing in the product
reads one to decide ranking or correctness; only the off-domain floor below looks at it.

The relevance floor, `SEARCH_SCORE_FLOOR=0.82` by default, was measured against something else
entirely: questions with **nothing to do with your documentation**. On the committed set of twelve
such questions it refuses ten, and it refuses none of the sixty-four golden questions. That is the
whole of what the floor claims: it catches most off-domain questions and costs an answerable one
nothing. It does **not** separate the two classes — the highest off-domain top hit (`0.839`) sits above
the lowest top hit of a real question (`0.833`), so no threshold does. Nor does it catch a question
shaped like your product whose answer is simply not written down — those score exactly where real
questions score. Below the floor
`search_docs` answers *no good match* rather than handing over a hit an agent would cite.

Scores are comparable within one project and one model, and meaningless across different models — change
`EMBEDDING_MODEL` and the floor has to be re-measured with `npm run eval`.

`SEARCH_SCORE_FLOOR` (default `0.82`) is the server's floor. A project can set its own from the query
panel, or turn the floor off for itself, when its corpus scores differently — prose-heavy corpora usually
want a lower one. The panel shows what a floor would have done to the searches already logged before it
is applied, and the server's `SEARCH_SCORE_FLOOR=0` still turns every project's floor off. See
[Configuration](/en/docs/configuration/).

**Ranks are fused, not scores.** Cosine similarity and PostgreSQL's `ts_rank` are not comparable
quantities, so the fusion combines the **rank** each chunk held on each side — `Σ 1/(60 + rank)` over
whichever lists it appeared on (reciprocal rank fusion) — rather than a weighted sum that would have to
be re-learned every time the embedding model changed. The consequence: **a result's score no longer
explains its position** — one scoring `0.86` can sit above one scoring `0.88`. `GET /api/projects/:id/search`
and the dashboard show the rank each result held on each side as `D3 L1`; a result with an `L` and no `D`
is an identifier the vector search alone could not see. On the golden set, rank fusion took the thirty
identifier-shaped questions from `recall@5` 76.7 % to 83.3 % and did not move the thirty-four
natural-language questions at all; `recall@1` fell from 78.1 % to 75.0 %, the trade fusion makes — a
chunk found by one half alone cannot outrank one found respectably by both.

**The default model is asymmetric**: it was trained with `query: ` in front of a search query and
`passage: ` in front of an indexed passage, so the server adds them automatically — nothing you index or
search ever mentions them, and a model without prefixes encodes a query and a passage identically at no
cost. On the retrieval corpus the prefixes were worth one question at rank 1 and nothing at `recall@5`;
`EMBEDDING_QUERY_PREFIX=none` with `EMBEDDING_PASSAGE_PREFIX=none` turns them off and restores the
previous model id exactly.

**The keyword half indexes each source in PostgreSQL's `simple` configuration by default** — words as
written, no stemming — which is the right choice for reference material: `HALYARD_DISPATCH_TIMEOUT` and
`HLY-4015` survive into the index as the strings they are. A source can instead **name its language** (the
*Language* field on the source form), which switches it to a stemmed configuration — a Turkish question
asking about `anahtarı` then finds a page that says `anahtarın`, two strings `simple` treats as unrelated.
The two live in one project without interfering: each source is searched in its own configuration and a
single query reaches all of them, each contributing its own ranked list to the fusion. Changing a
source's language re-indexes it.

## Cross-lingual search is a known limit, and it is not being fixed

**Write the question in the same language as the documentation that should answer it.** Turkish pages
surface for phrasing shaped like Turkish. English pages surface for phrasing shaped like English.
Crossing that boundary mostly returns nothing, and this is a property of the embedding model rather than
a setting to turn on. Every MCP client is a language model, and `list_topics` names a source's language
while `instructions` directs the calling agent to write its `search_docs` query to match — a direction to
the caller, not a change to search itself.

The measurement is in [`eval/BASELINE.md`](https://github.com/Contextator/Contextator/blob/main/eval/BASELINE.md)
and it is not close: thirty questions were written from the corpus, each about a page in the language the
question was not phrased in — fifteen per direction. Four are answered in the top five. Twenty-seven of
the thirty instead return a page in the phrasing's own language at rank 1: the model is not failing to
understand what was asked, it is ranking the language of the phrasing above the answer to it. Both
directions fail equally.

**Identifiers are the exception.** `HALYARD_DISPATCH_TIMEOUT`, `HLY-4015`, `X-Halyard-Signature` are the
same string in both languages, so the keyword half of search finds them regardless. Of the nine
cross-lingual questions in the set that name an identifier, four are answered; of the twenty-one phrased
as ordinary questions, none is.

**What to do instead of waiting for a fix.** Keep each language in its own source and let an agent scope
its search with `source`, `path_prefix` or `version` — two sources that each answer well beat one
collection that answers either language badly. Both known remedies have already been measured: hybrid
search (above) recovered cross-lingual *identifier* retrieval and no natural-language question; a
multilingual cross-encoder rerank behind `SEARCH_RERANK` took cross-lingual `recall@5` from 13.3 % to
33.3 % — still below the 42.9 % the embedding model before this one managed — while costing thirteen
ordinary questions their rank-1 answer and a search's latency going from 12 ms to 1.2 s. It is off by
default and should stay off. A second, translation-trained encoder as an additional index would actually
fix this — roughly four times the download, a second vector column, a full re-index on every
installation — and that price is not worth paying for a documentation server. If cross-language search
is essential to your corpus, that is a reason to choose a different tool rather than a reason to wait for
this one.

## Air-gapped installations

The only outbound request a default installation makes is the one-time model download. To remove even
that:

1. Pre-populate the models volume (`/app/.cache/models`) with the model files — for example by running
   once on a connected host and copying the volume. Do it with the same `EMBEDDING_MODEL` and
   `EMBEDDING_DTYPE` the air-gapped host will use.
2. Set `EMBEDDING_OFFLINE=1`.

The server will then never attempt a download, and will fail loudly if the model is missing.

### The reranker

The optional reranker (`SEARCH_RERANK=on`, off by default) is a second model, `SEARCH_RERANK_MODEL`
(`Xenova/bge-reranker-base` by default), loaded into the same cache directory and governed by the same
`EMBEDDING_OFFLINE`. With the reranker on and the flag set, its files must already be in the volume too:
warm the cache on the connected host with `SEARCH_RERANK=on` and the same `SEARCH_RERANK_MODEL` and
`SEARCH_RERANK_DTYPE` before you copy it.

### Moving the image and the models

On a connected host, pull the image and write it to a file, carry the file across, and load it:

```bash
docker pull contextator/contextator:<tag>
docker save --output contextator.tar contextator/contextator:<tag>
# on the air-gapped host
docker load --input contextator.tar
```

Use the tag you run, not `latest`, so the two hosts agree on the version. The models volume is not part of
the image: copy it separately (the default volume is `contextator-models`), or mount a directory with
`CONTEXTATOR_MODELS_PATH`.

### What still leaves the host

| Source type or feature | Outbound request |
|------------------------|------------------|
| Embedding, default `local` model | None once the cache is filled; otherwise the one-time model download from the Hugging Face Hub |
| Tokenizer named by `EMBEDDING_TOKENIZER` (with `EMBEDDING_PROVIDER=openai`) | Downloaded from the Hugging Face Hub unless the cache already holds it; subject to `EMBEDDING_OFFLINE`, and startup stops if it cannot be found |
| Reranker | None once the cache is filled; otherwise its one-time model download from the Hugging Face Hub |
| Embedding with `EMBEDDING_PROVIDER=openai` | Document text and queries, to `EMBEDDING_BASE_URL` (OpenAI unless you changed it) |
| Local directory, upload | None |
| Git repository | Your git host, over HTTP(S), on each sync and on a webhook-triggered one |
| Notion | Notion's API, on each sync |
| Confluence | Your Confluence instance, on each sync |
| Documentation site | The site you pointed it at, plus its `robots.txt` |

Remote sources are fetched from this server, not from the browser that set them up, so an air-gapped
host that needs one of them needs a route to that host and no other.

## Performance notes

- Embedding is CPU-bound and is the slow part of indexing; that is why one project is indexed at a time.
- `EMBEDDING_BATCH_SIZE` (default 16) controls how many chunks are embedded per call.
- A query embeds exactly one short text, so search latency is dominated by the database, not the model.
- The model is loaded in the background at startup — the dashboard works while it loads, but indexing
  and search wait for it.
