Muse Glimmer, the 30-billion-parameter open-weight model released under a genuine Apache 2.0 licence in August 2026, ships with a 128,000-token context window. That number lands on every comparison table and almost nobody translates it into something a business can act on. Before deciding a context window solves your document problem, it is worth knowing what 128,000 tokens actually holds, and where the model starts quietly losing track.
How much text does a 128K context window actually hold?
A 128,000-token context window holds roughly 90,000 to 100,000 words of English prose, depending on formatting, table density and how cleanly the text was extracted. In practical terms that is one substantial commercial contract with its schedules attached, a full quarter of board papers, or several hundred pages of a technical manual, read in a single pass with no chunking required.
Token counts are not word counts, and the difference bites hardest on exactly the documents firms most want to automate. Dense legal text with defined terms, clause numbering and cross-references burns more tokens per page than plain narrative prose. A PDF pulled through poor layout handling can add twenty per cent overhead in stray line breaks and repeated page headers before the model has read a single meaningful sentence. Budget against that overhead rather than the theoretical ceiling.
What fits, and what quietly does not
The useful question is not whether your documents are large. It is whether the specific job in front of you fits in one pass, or whether it needs a retrieval step sitting in front of the model.
A single commercial lease or services agreement, annexures and schedules included, usually fits comfortably
A quarterly board pack with appendices sits close to the edge, and extraction overhead can push it past
Twelve months of email correspondence for one client relationship rarely fits without summarising first
A document set spread across dozens of separate files almost never fits, and is better served by retrieval
The table below gives rough token loads for document types Australian firms process week to week. Treat the ranges as a starting estimate to test against your own extraction pipeline, not as a guarantee.
| Document type | Approx. tokens | Fits in 128K? |
|---|---|---|
| Commercial lease with annexures | 40,000 to 70,000 | Yes |
| Quarterly board pack with appendices | 90,000 to 130,000 | Borderline |
| Technical manual, around 300 pages | 100,000 to 120,000 | Usually |
| Twelve months of client email | 200,000 and up | No |
| Tender response library | 500,000 and up | No, use retrieval |
The accuracy problem the spec sheet does not mention
A model can accept 128,000 tokens of input and still answer worse on a fact buried in the middle of that input than on the same fact near the start or the end. Researchers named this the lost in the middle effect, and it was documented well before context windows reached this size. A larger window moves the ceiling. It does not flatten the curve underneath it.
We tested this directly for a Perth legal client reviewing long lease documents. Accuracy on a fact placed in the middle third of a 100,000-token document dropped noticeably against the same fact sitting in the opening pages. The model never refused and never flagged uncertainty. It simply answered, and it was wrong often enough to matter on a compliance-sensitive review where a missed clause carries real cost.
Test retrieval accuracy at several positions in your own documents before trusting a large window for regulated work
Put the clauses that matter most near the start or the end of the input wherever you control the ordering
Prefer chunking with retrieval over raw context stuffing anywhere a mid-document error is expensive
Write down which positions you tested, so the same evaluation can be repeated when you change models
What this costs in Australian dollars
For one client with a genuine long-document workflow, a self-hosted Muse Glimmer deployment on Australian infrastructure ran at roughly A$900 a month in GPU cost at moderate volume. The same document volume through the Claude API came to around A$2,100 a month, with better mid-document accuracy in our testing. The gap is about $14,400 a year, which sounds decisive right up until you price a single missed clause in a lease review against it. Our costing of large context windows works through the same arithmetic at the million-token end.
What not to conclude from this
None of the above says the open-weight option loses. Plenty of work inside a 128K window is not compliance-sensitive at all: drafting internal summaries, triaging inbound documents, producing a first-pass extraction that a person then checks. For that work the self-hosted path is a straightforward saving, and keeping the data on Australian infrastructure is a real benefit for organisations working under the Privacy Act.
What the numbers do say is that the context window is the wrong specification to decide on. The licence terms, the deployment path and the measured accuracy on your own documents are what settle it. Muse Glimmer is a reasonable candidate on the first two counts, and our note on why its licence line matters more than its model card covers that ground. On the third count, nobody can answer for you from a benchmark. If you want to improve results inside whatever window you land on, the prompt patterns we use for long documents are a better lever than another 100,000 tokens of capacity.
If you have a long-document workflow and want it measured before committing to either path, we run that evaluation as a fixed piece of work, testing positional accuracy on your documents rather than on a public benchmark. Start with a short conversation about what you are actually trying to read.
