Natural Language Processing (NLP): How AI Understands Human Language

Here is the sentence that broke thirty years of research.

"The trophy would not fit in the suitcase because it was too big."

What does "it" refer to? Every English speaker answers instantly: the trophy. Now change one word — because it was too small — and "it" becomes the suitcase. The grammar is identical. The pronoun flips because you know something about how objects and containers work, and no amount of parsing the sentence gets you there.

Problems like that consumed decades. The lesson eventually learned was uncomfortable: you cannot get to meaning by writing rules about language, because meaning does not live in the language. It lives in everything the speaker assumed you already knew.

What The Field Is Actually Trying To Do

NLP is the work of getting computers to handle human language rather than structured data. A database has columns and types and unambiguous values. A sentence has none of that, and it is how people actually communicate.

The applications are so embedded that most go unnoticed. Spam filtering. Translation. Search that understands what you meant. Autocomplete. Voice assistants. Sentiment analysis on reviews. Every chatbot. The reason NLP feels like a recent phenomenon is that it only became good enough to notice recently — the field itself dates to the 1950s.

Why Language Resists Rules

Three problems, each of which defeated the rule-based era on its own.

Ambiguity. "Bank" is a financial institution or the edge of a river, and no dictionary lookup resolves which — you need the surrounding sentence. Humans do this without noticing. Systems that could not do it produced confident nonsense, and the sentences where they failed were rarely the ones anyone tested.

Meaning that is not in the words. "It is cold in here" is grammatically a statement about temperature and functionally a request to close a window. "Can you pass the salt?" is formally a question about capability and nobody has ever answered it with yes. Understanding requires knowing what people do with sentences, not just what sentences contain.

Figurative language. "Break a leg" is encouragement. "Oh great, another meeting" means the opposite of what it says. Sarcasm inverts meaning entirely, and the inversion is signalled by context that may be several sentences away or not present at all.

The unifying problem: language assumes an enormous amount of shared knowledge and leaves it unstated. Rules operate on what is written down.

The Three Eras, Briefly

Rules. Linguists hand-wrote grammar and vocabulary. This worked in narrow domains and collapsed everywhere else, because exceptions in language outnumber regularities and every fix created new failures. Machine translation of this era became a byword for output that was grammatical and meaningless.

Statistics. Instead of encoding what language should do, measure what it does across large amounts of text. Which words follow which, how often, in what contexts. This was a genuine improvement and it is why translation stopped being a joke around the 1990s. It was still shallow — the systems knew that words co-occurred without any representation of why.

Learned representations. Neural approaches, and then transformers, which learn from context rather than from position. This is the shift that produced everything people now recognise as AI, and the pronoun problem above is one of the things it quietly solved — not by encoding knowledge about trophies and suitcases, but by having read enough text that the pattern is statistically obvious.

The Jobs Underneath

Most NLP systems are assembling a few standard operations.

Classification sorts text into categories. Spam or not spam, positive or negative, which department this ticket belongs to. The most commercially deployed thing in the field by a wide margin, and the least discussed.

Entity recognition pulls out the people, organisations, places, dates, and amounts. It is what lets a system read an invoice, or notice that "Tokyo" in a sentence is a location rather than a person's name.

Sentiment analysis judges emotional tone. Genuinely useful at scale and genuinely unreliable at the edges — sarcasm and politeness both break it, which is why a classifier can report that a queue full of complaints was overwhelmingly positive. The practical consequences of that are covered in customer service automation.

Translation, which went from unusable to routine within most readers' lifetimes.

Question answering and summarisation, which is where current systems are strongest and where the failure modes are least visible.

What Changed With Large Language Models

The older approach built a separate system for each job. A classifier for spam, a different model for translation, another for entity extraction — each trained on its own labelled dataset, each requiring its own expertise.

Large language models collapsed that. One model does all of it, because predicting the next word across enough text turns out to require learning most of what those specialised systems were learning separately. Ask for a translation and you get one. Ask for a summary, a classification, a rewrite — same model, no retraining.

That is why the field feels like it changed overnight. It did not get one new capability. It got a general one that made a great deal of specialised work unnecessary. The architecture is covered in what a large language model is, and the mechanics of why it works at all in how AI actually works.

What It Still Cannot Do

Worth being precise, because fluency invites overestimation.

It does not know things. It knows how text about things tends to be arranged. The distinction is invisible when the training data was consistent and becomes obvious at the edges, where it produces something well-formed and false with exactly the same confidence.

It cannot tell you when it is wrong. There is no internal signal separating a well-supported answer from a fabricated one. This is the single most consequential limitation and it is the reason every serious deployment needs a competent human reviewer. Why AI fails covers the failure modes in detail.

It inherits whatever was in the text. Bias in the training data becomes bias in the output, and the fluency makes it more persuasive rather than less. The governance side is covered in responsible AI.

Genuine ambiguity still defeats it. The trophy sentence is solved. Sentences requiring knowledge that was never written down anywhere are not, and there is no obvious route to solving them by reading more text.

Why Any Of This Is Worth Knowing

You do not need NLP theory to use a chatbot. The reason to understand it is that it tells you where the outputs are reliable.

Anything the model can do by manipulating text in front of it — summarising, restructuring, classifying, translating — is on solid ground, because everything needed is present. Anything requiring facts it must recall, or judgement about a situation it cannot see, is where the confident errors live.

That single distinction is more practically useful than any amount of terminology, and it follows directly from what NLP actually is: pattern-work over language, extraordinarily good at it, and holding no model of the world underneath.

If you want the surrounding map, AI vs machine learning vs deep learning places NLP within the wider field, and computer vision covers the equivalent story for images — a field that hit the same wall and got past it the same way.