NLLB-200 and frontier models

NLLB-200 vs LLM translation — train a checkpoint, or prompt with examples.

NLLB-200 is Meta’s open-weight model for translation among 200 languages, and Scripture Forge fine-tunes it on a project’s completed Scripture. Few-shot LLM translation puts approved examples in the prompt and does not train a new model. Published test-set scores often still favor a specialized or fine-tuned NLLB system for low-resource languages. Aquilla uses the prompting path and does not claim a higher score.

The direct answer

How does NLLB-200 compare with an LLM for translation?

NLLB-200 is a translation model. Fine-tuning changes its weights on parallel text from one project, which is how Scripture Forge drafts a book. An LLM comparison here means in-context learning: the model sees examples and continues, and the weights are not updated. The fair reading of the public benchmarks is that fine-tuned NLLB is often stronger on a frozen low-resource test set. The fair reading of the work is that those scores do not time the loop where a person corrects and the next draft uses the correction. Aquilla is built for that loop. It is not a higher chrF number.

Same draft, different machinery

Fine-tune NLLB, or prompt an LLM.

Both can propose a first draft of a low-resource language. They differ in training, hardware, and when a completed translation starts to help.

StepFine-tuned NLLBFew-shot LLM
What is adaptedThe model’s weights, on that project’s parallel text.The prompt. The model weights stay as released.
Data the Bible path asks for firstA completed translation to learn from. For a New Testament, at least one Synoptic Gospel.Examples and approvals as you go. No finished testament is required.
Who runs the trainingSIL servers, with Scripture Forge onboarding and the Engineer-in-the-Loop team.Nobody trains a language checkpoint. The model provider runs the prompt.
Compute the translator provisionsNone for the Scripture Forge path. SIL’s service uses GPUs.None. The browser does not host the model.
When the next book sees the last bookAfter that book is in the training set and a new draft is generated.Approved decisions stay in Living Memory for a later draft.
Where the project text is processedOn SIL servers for Scripture Forge’s NLLB fine-tune and inference.The prompt and project context go to the selected model provider.
What a FLORES or chrF paper measuresA frozen checkpoint on a fixed set. Often a strong result.A frozen prompt on a fixed set. Not the live approval loop.

How Scripture Forge fine-tunes NLLB for Bible translation

Serval, SIL’s translation API, says neural translation into a new minority language is done by fine-tuning NLLB, with Scripture Forge and Lynx as reference interfaces. Back translation into more than 200 languages uses NLLB-200 as released. In October 2026 the Scripture Forge team, writing as SIL AI Engineering, said NLLB is the translation-specific model under Scripture Forge, that it is open-weight, and that fine-tuning and inference run locally on SIL servers so that data stays there and cost stays down. The same note says newer language models had begun to rival and surpass NLLB for some pairs, and that frontier models add a per-token cost and a different data-security question. They were not dismissing that work. They were not shipping it yet. The post is on the SIL community forum.

The drafting setup is per project. Sign up from the main Paratext project, not from a project that only stores generated drafts, so the model learns from completed translation. The onboarding team reviews the request and enables drafting. The reference project is the model text used for training. A second reference is allowed; a March 2025 Scripture Forge note says a team can train on a reference text and a back translation together, and both must be in the same language as the draft source. Extra training pairs can be a spreadsheet, source sentence in the first column and translation in the second. The source to translate can be a Paratext project or a Digital Bible Library resource.

For a New Testament, the drafting guide asks for at least one completed Synoptic Gospel before the first generated draft. Matthew or Luke is preferred. Mark is acceptable. John or Acts as the first book is not recommended. Drafting is book by book: finish a book with the team, then generate the next, so the finished book is in the training set. The guide says drafts of Acts are usually weaker than the Synoptics until more of those books are done, and that a new name such as Pentecost or Cappadocia can be poorly rendered when it was not in the earlier books. For the Gospels and Acts, incremental generation is an option: finish about 8 to 10 chapters, then request a new draft of the remaining chapters. Matthew, Luke, and Acts can do that twice. Mark and John, once. It is not a self-service button. The Engineer-in-the-Loop team runs it and can share a green, yellow, or red quality estimate. Those estimates are planning aids. The guide says they are not a quality check.

The translator does not stand up the GPU. SIL’s 2025 account of its production translation service describes an on-premises GPU server, cloud GPUs when the queue is deep, and, at the start of 2025, more than 300 language groups on custom fine-tuned models. That article does not name a parameter count. Meta’s own training of the 54-billion-parameter NLLB-200 used its Research SuperCluster, which is the original model, not a project fine-tune. A community reply on the same SIL thread describes SILConverters running a downloaded NLLB, including a fine-tuned one, on a local computer, and recommends a GPU. That is a different tool from the Scripture Forge server path.

Meta’s distilled 600M card still matters as a description of the base model: CC-BY-NC, research rather than production, single sentences, inputs not expected past 512 tokens, general-domain text rather than a specialized domain, and no use as a certified translation. Scripture Forge’s fine-tune is how a Scripture project specializes that general model. The drafts, Scripture Forge says, always need review by experienced translators.

Few-shot prompting, which is what Aquilla does

Moslem, Haque, Kelleher, and Way put similar approved translation pairs, and sometimes terminology, in the prompt. The model adapted the new sentence without further fine-tuning. They report that this can pass a strong encoder-decoder system especially for high-resource languages, and that mixing encoder-decoder output with those examples can help less-supported languages, including English-to-Kinyarwanda in their set. Later work on example selection finds that similar examples help low-resource directions in particular. Lu and colleagues find the same pattern as the number of examples grows, and a further gain from prompting even after fine-tuning.

Aquilla’s draft is that kind of step, inside a workspace rather than a notebook. A frontier model is prompted with project context. Examples and translations the team has approved are part of that context. Living Memory keeps an authorized decision and can apply it to a later draft. There is no per-language training job, no GPU for the team to rent, and no wait for an onboarding engineer to enable a checkpoint. A newer frontier model can be used without fitting a new model for each language. In Aquilla’s experience, this is the best way we have found to leverage AI for low-resource language translation. We state that as experience. We do not state it as a score.

The data tradeoff is the one SIL named. A built-in Aquilla draft sends the prompt and project context to the model provider selected for that feature, on API terms intended to keep customer content out of general model training. Scripture Forge’s reason for staying on local NLLB is data security and cost. Read the privacy page if that boundary decides the tool. Aquilla also does not connect the draft to Paratext Send/Receive. Teams that still check in Paratext can export USFM.

Why the papers do not crown a product

Static BLEU and chrF miss the feedback loop.

Robinson and colleagues, on FLORES-200, found NLLB ahead of ChatGPT by 11.9 chrF on average, and ahead on 169 of 201 languages. Five-shot prompting beat NLLB on 6 percent of the low-resource languages in that study, against 47 percent of the high-resource ones. Scalvini, Debess, and Simonsen found fine-tuned NLLB 1.3B second only to Claude 3.5 on their Faroese automatic scores, and first among open models, with the same ordering in a small human review. Those are real results. They score a model that does not change when a translator fixes the previous sentence.

Kumar, Jyothi, and Bhattacharyya show BLEU and chrF++ pulling apart on extremely low-resource languages: BLEU punishes morphology and word order, and chrF++ can reward surface overlap, including copies of the source. Chen and colleagues show that a large share of the variation in extremely low-resource scores lines up with train-test overlap and pretraining exposure. Castilho and Knowles describe sentence BLEU as insensitive to a contextual fix that changes only a few words. A run of approvals is that kind of change, repeated. No public BLEU table, including the ones cited here, measures it. Aquilla will not fill the gap with an unsourced number.

Common questions

NLLB-200 vs LLM translation FAQ

What is NLLB-200?

NLLB-200 is Meta’s No Language Left Behind model, a single multilingual translation model for 200 languages. Meta reports a 44 percent average BLEU improvement on FLORES-101 and describes training a 54-billion-parameter model on its Research SuperCluster. Smaller distilled checkpoints are public. The 600-million-parameter card is licensed CC-BY-NC and describes a research model for single general-domain sentences, not certified translation and not domain-specific text.

How do you fine-tune NLLB for Bible translation?

Scripture Forge fine-tunes the NLLB model that underlies its drafts. You sign up from the main Paratext project so the model learns from completed translation. A reference project, usually the model text the team consults, is the training side. The drafting source is what gets translated, often a Digital Bible Library text or a Paratext project. Optional extra pairs can be uploaded as a spreadsheet. For a New Testament, Scripture Forge’s guide asks for at least one completed Synoptic Gospel before draft generation, prefers Matthew or Luke first, and then generates the next book after the team finishes the current one. Aquilla does not run this fine-tune.

How much data and compute does an NLLB fine-tune need?

Scripture Forge trains on the project’s completed translation, plus the reference text, plus any paired sentences you upload. A second reference, such as a back translation, has to be in the same language as the draft source. Fine-tuning and inference for Scripture Forge run on SIL servers, not on the translator’s machine. SIL’s AI team describes a production service on an on-premises GPU server, with cloud GPUs when the queue is deep, supporting more than 300 language groups with custom fine-tuned models as of the start of 2025. That write-up is about SIL’s service, not a VRAM figure for one NLLB size. A separate SILConverters tool can run NLLB locally, and the person describing it recommends a GPU. Meta’s original 54-billion-parameter training run is a different job from a project fine-tune.

How often is the NLLB model retrained?

Scripture Forge’s New Testament guide is book by book. After the team uses a draft to finish a book, the next book’s draft is generated from a training set that includes the finished book. For the Gospels and Acts, incremental generation is optional: finish about 8 to 10 chapters, then ask for a new draft of the rest. Matthew, Luke, and Acts can do that twice; Mark and John once. Incremental generation is not self-service. It is requested through the Engineer-in-the-Loop team. That is a training cycle, not an update on the next sentence.

What is few-shot LLM translation?

Few-shot, or in-context, translation puts example sentence pairs in the prompt so the model continues the pattern, without a gradient update. Moslem, Haque, Kelleher, and Way found that similar approved pairs and terminology can steer a new sentence at translation time, with stronger published gains on high-resource pairs than on less-supported ones. Aquilla drafts by prompting a frontier model with project context, including translations the team has approved. In Aquilla’s experience this is the best way we have found to use AI for low-resource language translation. That is experience, not a claim that few-shot beats NLLB on chrF.

Does NLLB-200 beat LLMs on benchmarks?

On many published test sets, yes, especially for low-resource languages and for a fine-tuned NLLB. Robinson and colleagues found NLLB ahead of ChatGPT in chrF on 84.1 percent of 201 FLORES-200 languages in 2023. Scalvini, Debess, and Simonsen found a fine-tuned NLLB 1.3B model ahead of most open models, and ahead of GPT-4o in their Faroese automatic scores, while Claude 3.5 ranked first overall. SIL said in October 2026 that newer models had begun to rival and surpass NLLB for some pairs. Aquilla does not add its own score to that table.

Why is NLLB versus an LLM hard to benchmark?

The published numbers score a frozen model on a frozen set. They do not replay a translator approving a verse and the next draft using that approval before anyone retrains. BLEU and chrF++ also disagree on extremely low-resource output, and extremely low-resource scores can move with train-test overlap and with how much of the language was in pretraining. Sentence BLEU is a weak instrument for a change that affects only a few words of context. Use the papers to understand a checkpoint. Do not use them as a ranking of a live review loop.

Can Aquilla fine-tune NLLB for my language?

No. Aquilla does not train NLLB or any other model for a customer. It does not install a GPU workflow, and it does not take a finished New Testament as a training set. If that checkpoint is the requirement, Scripture Forge is the tool that documents it. Aquilla is the prompting workspace.

How much does this cost in Aquilla?

Anyone can start on the Free plan with a weekly allowance of AI assistance. Bible-translation teams and ETEN affiliates get team-plan access at no cost — contact us to activate your organization’s workspace.

Sources

Related topics

Use the examples you already trust.

Open a Free plan and review what the agent proposes. Bible-translation teams and ETEN affiliates can contact us for team-plan access at no cost. Aquilla will not fine-tune NLLB for you.