Computer Vision Explained: How AI Sees and Understands Images

For about fifty years, computer vision was stuck on a problem that sounds trivial: write down what makes a photograph contain a cat.

Try it. Fur? So do dogs. Four legs, pointed ears, whiskers, a certain shape? Every rule you write has counterexamples — cats from behind, cats in shadow, cats curled into an unrecognisable ball, one ear visible behind a sofa. Meanwhile a two-year-old does it instantly and cannot explain how either.

That was the wall. Early systems required a human to specify which visual properties mattered — edges, corners, textures, colour histograms — and then matched against those. It worked in controlled conditions and fell apart on real photographs, because the features that actually distinguish a cat are not ones anyone can articulate.

The breakthrough was giving up on articulating them.

Learning The Features Instead

Deep neural networks do not get told what to look for. They are shown an enormous number of labelled images and work out the useful properties themselves.

What they converge on has a striking structure. Early layers detect edges and gradients — the most primitive visual information there is. Middle layers assemble those into textures and simple shapes. Later layers combine those into recognisable parts: an eye, a wheel, a leaf. The final layers put parts together into objects.

Nobody designed that hierarchy. It emerges because it is an efficient way to compress visual information, and it is loosely reminiscent of how biological vision is organised — which is interesting, and less significant than it sounds. The network is not modelling a visual system. It found a similar solution to a similar compression problem.

The practical consequence is the important part: the system's knowledge lives in millions of numeric weights that nobody can read. It works, and you cannot ask it why.

What The Field Actually Does

"Recognising images" is one task among several, and the distinctions matter when you are specifying a system.

Classification answers what is in this image, as a single label. Simplest, most common, and the basis of most benchmarks.

Detection answers what is in it and where, drawing boxes. This is what a driver-assistance system needs, because knowing a pedestrian is present is useless without knowing where.

Segmentation classifies every individual pixel rather than drawing boxes. Necessary in medical imaging, where the precise boundary of something is the entire clinical question.

Tracking follows objects across video frames, which introduces the problem of knowing that the partially hidden thing in frame 400 is the same object as in frame 12.

Reconstruction infers three-dimensional structure from two-dimensional images. Hard, and the basis of a good deal of robotics and mapping.

Where It Genuinely Works

The successful applications share a shape: narrow task, controlled conditions, high volume, and a failure that is either cheap or caught by a human.

Manufacturing inspection is the clearest case. Same product, same lighting, same camera position, thousands of times an hour. A model trained on that specific line outperforms a fatiguing human inspector, and the conditions never change.

Medical imaging support works for the same reason — consistent capture conditions and a specialist reviewing the output. The framing there matters and is often reported badly: these systems flag candidates for a radiologist, they do not diagnose. AI in healthcare covers where that line actually sits.

Document scanning, face unlock, photo search, retail checkout. All narrow, all high volume, all in conditions resembling their training data.

Why The Failures Look So Strange

This is the part worth understanding, because it is not intuitive and it predicts real problems.

A vision model learns statistical patterns in pixels, not concepts. If almost every cow in the training data is standing on grass, "grass" becomes part of what the model has learned "cow" means. Show it a cow on a beach and confidence collapses — not because the cow changed, but because the context did.

This generalises into the central limitation: models fail when conditions differ from training in ways nobody thought to vary. Different lighting. An unusual angle. A demographic underrepresented in the training set — which is how facial recognition systems came to have documented and serious accuracy disparities across skin tones, an issue covered further in responsible AI.

There is also deliberate exploitation. Carefully constructed changes to an image, sometimes imperceptible to a person, can make a classifier confidently wrong. That is not a bug awaiting a patch — it is a consequence of how the systems represent images, and it has resisted a general fix.

The practical rule: an accuracy figure describes the test set. It says nothing about your lighting, your camera, your subjects, or the edge cases you have not encountered yet.

What This Means If You Are Buying One

Ask what the training data looked like, and specifically how it differs from your conditions. This is the question that predicts deployment failure and it is rarely the one asked.

Ask what happens when the model is uncertain. A system that outputs a confident label regardless is considerably more dangerous than one that can decline and escalate — and as with language models, reliable self-assessment remains unsolved.

Assume you will need to retrain. Conditions drift, product lines change, cameras get replaced. A vision system is an ongoing commitment rather than a purchase.

And check whether you need explanation. In regulated settings, "the model said so" is not an adequate account of a decision, and these systems cannot do better than that.

Where It Sits

Computer vision and language processing arrived at the same place by the same route, which is why they broke through at roughly the same time. Both spent decades on hand-written rules, both hit a ceiling because the relevant features could not be articulated, and both were solved by learning representations directly from large amounts of data.

The parallel is exact enough to be worth noticing: natural language processing tells the same story for text. Both are applications of deep learning, which sits inside machine learning, which sits inside AI — a nesting covered in AI vs machine learning vs deep learning.

And both share the same fundamental limitation. Extraordinary pattern recognition, no model of the world underneath. A vision system that identifies a cat has never held one, does not know it is an animal, and cannot tell you anything about it that was not in the pixels. That is enough for a great many valuable applications, and it is worth remembering when someone describes such a system as understanding what it sees.

For the underlying mechanics, how AI actually works is the place to start.