Multilingual embeddings need tests in both language directions

A Romanian question can be perfectly understandable to a model and still retrieve the wrong English document. A bilingual knowledge tool therefore needs evaluation by query language and document language separately. Broad multilingual coverage is a reason to test a model, not evidence that a particular cross-language search works.

Jina AI's 9 October 2024 article on language gaps in embeddings reports experiments on semantic alignment and contrastive learning with parallel multilingual data. It describes how equivalent meanings can remain separated by language in vector representations. Those are the vendor's experiments. This November 2026 retrospective does not turn them into measured Romanian retrieval performance or an AL+ deployment result.

Define the evidence the user needs

An embedding represents text numerically so a search system can compare a question with candidate passages. The useful business result is finding evidence that supports the user's task. Similarity between vectors is an intermediate signal, and a fluent final answer cannot repair an authoritative document that was never retrieved.

Consider a proposed internal knowledge tool with English supplier manuals and Romanian operating notes. This is an illustrative design, not a customer project. A Romanian-speaking employee might need an English manual for a device fault, while an English-speaking colleague might need a Romanian instruction describing the local procedure. Neither question is simply “does the model support Romanian?”

Before choosing a model, define which documents count as relevant for each question. A general product overview may be related to a fault without containing the required repair instruction. An old procedure may be semantically close but operationally wrong. The evaluation should distinguish topical similarity from usable evidence.

A knowledgeable reader should also mark questions with no valid answer in the permitted corpus. Otherwise a system that always returns something can appear successful by finding a related passage for every query. Abstaining or reporting insufficient evidence may be the correct outcome.

Keep four language slices visible

Evaluate English questions against English documents and Romanian questions against Romanian documents as same-language baselines. Then evaluate Romanian questions against English documents and English questions against Romanian documents separately. Report the number of questions in each group alongside the result. A combined score can hide a weak direction behind a much larger easy group.

Use questions people would naturally ask in each language. A set made entirely by translating English test questions can miss Romanian abbreviations, local procedural names and different ways of describing the same task. Translated pairs can help diagnose alignment, but they should not become the whole test of user usefulness.

For every question, record the acceptable source passages and whether the first few returned results include them. Choose how many results matter from the proposed interface: a reviewer scanning three passages has a different task from a system processing twenty. State the cutoff rather than presenting a retrieval percentage without a denominator or selection rule.

Add near misses deliberately. Similar device names, superseded procedures and documents containing the same words for a different purpose make useful evaluation cases. Keep those cases tied to the intended corpus. A synthetic trap disconnected from the business can distort model selection just as easily as an overly easy test.

Separate language failure from access failure

A document can be relevant and unavailable to the requester. Relevance does not authorize disclosure. Apply the user's access boundary before material reaches an answer generator, and ensure any search stage outside that boundary cannot expose restricted text through snippets, logs or cached responses.

For the illustrative tool, an English technical manual might be widely available while a Romanian note contains customer-specific details. Translating the note or embedding it does not remove its restrictions. Evaluate authorized retrieval with identities that have different access, including cases where the correct response is that no permitted evidence is available.

Keep this result separate from language quality. A Romanian query returning no document because access was correctly denied is not a semantic failure. Returning a restricted document accurately is an access failure. Combining both into one “correct answer” score would conceal the distinction the business needs to assess.

The same separation helps diagnosis elsewhere. A passage lost during indexing is different from a passage ranked poorly across languages. Check that the expected document is present and its revision is right before attributing a miss to the multilingual model. Improving alignment cannot restore a file that was never searchable.

Compare simpler alternatives fairly

The strongest practical counterargument is that a small bilingual corpus may not need a sophisticated shared embedding system. Translating the query into the document language, keeping separate indexes or using ordinary keyword search may serve the work with less operating complexity. A multilingual model is one candidate architecture.

Compare alternatives using the same relevance judgments and permission rules. Include translation mistakes, extra delay and the maintenance of language-specific indexes in the assessment. A translated query can lose a technical identifier or alter a local term. A shared index can simplify operations while still returning poor cross-language matches. Neither option wins from its name.

Jina also reports that controlled experiments did not establish an unambiguous effect from adding explicit cross-language pairs. That qualification matters when interpreting its broader alignment claims. Jina's discussion provides a plausible mechanism for improving alignment, not a guarantee of usefulness for every language pair. Broad benchmark results and visualization of clusters should inform the shortlist. The acceptance decision belongs to the questions, documents and constraints of the intended tool. No experiment, benchmark or search-quality measurement was conducted for this article.

For an AI agent knowledge workflow, ask for results in each required language direction, with missed evidence and access failures described separately. Decide what level of failure the actual task can tolerate before evaluating candidates. That gives a bilingual rollout a concrete acceptance condition beyond the word “multilingual.”