A window into how a language model ‘understands’ context
Type a sentence with [MASK] where a word should go — “The capital of France is [MASK].” — and BERT returns its top predictions with probabilities (Paris at 98%, and a few alternatives far behind). It feels like magic, but it’s the exact task BERT was trained on: hide a word, predict it from context. Watching the predictions is one of the most intuitive ways to see what a language model has learned about how words relate, which makes this as much an educational tool as a practical one.
Why bidirectional matters
The reason BERT nails the blank is that it reads context from both sides of the mask at once. In “I [MASK] my keys at home,” the words before and after the gap together point at “left.” A left-to-right model only sees “I ___” and has far less to go on. This bidirectionality is BERT’s defining trait and the reason masked-language models excel at understanding tasks — classification, entity recognition, question answering — even though they don’t generate flowing text the way GPT does.
Practical and educational uses
- Beating writer’s block — mask the word you can’t think of and let context suggest it.
- Understanding NLP — a hands-on demo of how contextual prediction works.
- Exploring associations — see which words the model considers likely in a given frame.
Fill-mask as a bias probe
One genuinely important use is surfacing model bias. Because the predictions reflect patterns in the training data, masking a word in a sentence about professions, gender, or nationality reveals what the model has absorbed — “The [MASK] is a nurse” vs “The [MASK] is an engineer” can return tellingly different distributions. This is a standard technique researchers use to audit language models, and you can try it yourself here. It’s a reminder that these models mirror their training data, biases included.
Privacy
The BERT model runs entirely in your browser via Transformers.js after a one-time download, so the sentences you test stay on your device. Keep to one [MASK] per sentence for the sharpest predictions, and move it around to explore how context shifts the model’s confidence.