Recognising an entity is labeling a sequence
Named Entity Recognition means spotting, inside a text, the fragments that designate something specific - a person, an organisation, a place, a date - and assigning each a type. Formally it is not sentence classification: it is sequence labeling, one decision per token, with a consistency constraint between neighbouring tokens.
Hence the BIO annotation scheme, which separates the beginning of an entity from its continuation and from the rest of the text. The distinction looks trivial; it is what lets the model recognise two adjacent entities as two entities rather than one.
The ground: French travel queries
The project I pushed this work through turns a free-form French sentence - "je voudrais aller d'Épinal à Paris" - into a rail itinerary on the SNCF network. So two entities have to be extracted, the departure station and the arrival one, before any route can be searched.
Put that way, a regular expression sounds like enough. It does not survive three sentences. Word order moves around ("de X à Y", "à Y depuis X"); elision glues the preposition to the name ("d'Épinal"); some towns carry people's names (Albert, Florence) or are built from common words (Port-Boulet); and users write without accents, with typos and in whatever casing. Every rule added to catch one case breaks another.
This is exactly the situation where a language model beats a rule set: it does not need the stations enumerated for it, it learns the shape of a travel sentence.
What attention changes
Before Transformers, sequence labeling relied on recurrent architectures, which read text left to right and pass a state from one word to the next. Distant context dilutes as it travels along that chain.
The attention mechanism replaces that cascade with a direct relation: each token computes a weight against every other token in the sequence, and builds its representation as a weighted sum of whatever it judges relevant. A pronoun can reach its antecedent fifteen words back in a single step, and an ambiguous word can be disambiguated by the context surrounding it on both sides.
For NER this is decisive: the same token can be a person's name, a brand or a place depending on what surrounds it, and that is exactly what attention is good at exploiting.
From pre-trained model to domain model
The base model is CamemBERT, 110 million parameters, pre-trained on French: it arrives with a solid grasp of the language and no knowledge whatsoever of the target domain. Fine-tuning means continuing training on a specialised corpus, with a per-token classification head placed on top of the model.
I generated the corpus: 546 sentence templates crossed with 8,973 French towns, giving 105,474 sentences, from which I drew 12,000 intent examples and 6,000 entity examples annotated in BIO. Seven augmentation strategies - lowercasing, accent stripping, typos, punctuation, random casing and combinations - reproduce how people actually type a query.
The decisions that matter next are few but structural: the learning rate - 5×10⁻⁵ here; too high and it erases what pre-training acquired; the number of epochs, which a domain corpus saturates quickly, three were enough, with validation F1 going from 94.75% to 96.20% and then 96.81%; and how to handle subword segmentation, since the tokenizer splits station names into pieces and you have to decide how to realign labels onto the original words.
Evaluating seriously
NER is evaluated with precision, recall and F1 - but at the entity level, not the token level. The nuance is not cosmetic: a model that finds three of an entity's four tokens has not found the entity, it has produced a wrong answer. Counting per token inflates scores artificially.
Recall on rare classes deserves separate tracking, for the same reason as on imbalanced images: a global average hides exactly the cases the system was built for.
The numbers, against the rule-based system
I had first built an honest baseline: TF-IDF for intent, regular expressions for entities. That is what makes the comparison readable - the gap measures what the language model genuinely adds, not what it adds over nothing at all.
Exact extraction of both stations goes from 33.3% to 80.7%. Similarity on the arrival station, the rule-based system's weakest point, nearly doubles: 43.6% to 92.1%. The gap is wider there than on departure, for a simple reason: the departure station usually follows a regular preposition, the arrival one does not.
The price is a per-sentence latency going from 0.045 ms to 18.3 ms, a factor of 400. On an interactive query whose full pipeline fits in 100 milliseconds, pathfinding included, that is a price you pay without thinking - but it had to be measured rather than assumed.
Figures measured on the project's test set and published in the repository. Latency, 0.045 ms against 18.3 ms per sentence, is not plotted here: it shares neither the unit nor the scale.
What I take away
Fine-tuning a pre-trained model is quick to set up and deceptively easy to get wrong. Most of the work is not about the model but about what surrounds it: the quality and consistency of the annotations, label alignment after tokenisation, and an evaluation protocol that does not congratulate itself.
The other lesson is that a seriously built baseline beats a good score on its own. Without the rule-based system's 33.3%, the model's 80.7% means nothing.