Image segmentation — the process of separating an image into meaningful regions — is one of the most fundamental tasks in computer vision and image editing. Every time you select an object in Photoshop, remove a background from a photo, or use a portrait mode camera feature, segmentation is at work.
But the technology behind segmentation has changed dramatically over the decades. What once required painstaking manual work now happens with a single click, thanks to AI models like SAM2. Let’s trace the journey from the earliest selection tools to today’s foundation models.
timeline
title Image Segmentation Through the Decades
1988 : Active Contours / Snakes : Boundary snaps to nearby edges
1990s : Lasso and Magic Wand : Manual pixel-level selection
2004 : GrabCut : Graph-cut foreground vs background
2015 : FCN and U-Net : First deep-learning semantic segmentation
2017 : Mask R-CNN : Per-instance segmentation
2023 : SAM : Foundation model, click any object
2024 : SAM2 : Faster, video-aware, ambiguity handlingThe story splits into four broad eras. Each one shrinks the gap between intent and result — and the latest era essentially closes it.
The Manual Era: Pixel by Pixel
In the earliest days of digital image editing, segmentation was entirely manual. If you wanted to isolate an object from its background, you had to define the boundary yourself, pixel by pixel.
The Lasso Tool
One of the earliest selection tools in image editors was the freehand lasso. Introduced in early versions of Adobe Photoshop in the early 1990s, the lasso let you draw a selection boundary by tracing around an object with your mouse.
The lasso was simple and intuitive — just draw around what you want. But it was also painfully imprecise. Tracing around complex shapes like hair, fur, or tree branches was nearly impossible to do cleanly. A shaky hand or a momentary lapse in concentration meant starting over. Professional retouchers would spend hours carefully tracing selections around a single subject.
The polygonal lasso improved things slightly by letting you click to place straight-line segments, building up a selection point by point. This was more precise for objects with angular edges but even worse for organic, curved shapes.
The Magic Wand
The magic wand tool represented the first attempt at semi-automated selection. Instead of tracing a boundary manually, you clicked on a pixel, and the tool selected all contiguous pixels of a similar color. Adjust the “tolerance” slider and you could select larger or smaller regions.
The magic wand was revolutionary for simple cases — selecting a solid blue sky, for instance, worked beautifully with a single click. But it fell apart with textured surfaces, gradients, or anything without uniform color. Selecting a person’s face against a similarly-toned background was an exercise in frustration, requiring dozens of clicks with carefully adjusted tolerance settings.
Refinement Tools
To compensate for the limitations of basic selection tools, editors introduced refinement features. Feathering softened selection edges to reduce harsh boundaries. Grow and Similar expanded selections to adjacent or matching pixels. Quick Mask mode let you paint selections with a brush, combining the precision of manual work with the flexibility of painting.
These tools helped, but they were fundamentally band-aids on a manual process. Segmentation remained one of the most time-consuming tasks in image editing.
The Edge Detection Era: Let the Computer Help
The next major step was asking the computer to detect object boundaries automatically, at least partially.
Classical Edge Detection
Algorithms like the Sobel filter, Canny edge detector, and Laplacian of Gaussian could identify pixels where sharp intensity changes occurred — which often corresponded to object boundaries. These mathematical techniques, developed through the 1970s and 1980s, were among the first computer vision algorithms to gain practical use.
Edge detection could find boundaries that the human eye could see, but it produced raw edges without understanding what those edges meant. A Canny edge detector applied to a photo of a cat would highlight the cat’s silhouette, but also every whisker, every fur texture, every shadow, and every pattern in the background. Converting raw edges into a meaningful object selection required significant additional processing.
Active Contours (Snakes)
In 1988, Kass, Witkin, and Terzopoulos introduced active contours, playfully called “snakes.” A snake was a curve that you placed roughly around an object, and it would automatically shrink or expand to snap to nearby edges. The algorithm balanced three forces: internal forces kept the curve smooth, image forces attracted it to edges, and external forces let the user guide it.
Snakes were a significant conceptual advance — the computer was now actively helping to find the boundary. Adobe Photoshop’s Magnetic Lasso tool, introduced in version 5.0 (1998), used similar principles. As you moved your cursor near an edge, the selection line would snap to it, dramatically speeding up manual tracing.
Graph-Based Methods
The early 2000s saw the rise of graph cut methods, most notably GrabCut (2004). The idea was to model the image as a graph where pixels were nodes and edges represented similarity between neighboring pixels. By marking some pixels as definitely foreground and others as definitely background, the algorithm could compute an optimal cut through the graph that separated the object from its surroundings.
GrabCut was remarkably effective for its time. You drew a rough rectangle around an object, and the algorithm figured out the precise boundary. It handled gradual color transitions and complex textures far better than the magic wand ever could. This technology found its way into tools like Photoshop’s Quick Selection and Select and Mask features.
The Deep Learning Revolution
Everything changed when deep learning entered the picture. Starting around 2015, neural networks began outperforming all traditional approaches to image segmentation.
Fully Convolutional Networks (FCN)
In 2015, Long, Shelhamer, and Darrell published their landmark paper on Fully Convolutional Networks for semantic segmentation. Instead of classifying entire images, FCNs could classify every pixel in an image, assigning each one to a category (person, car, sky, etc.).
This was the first time a computer could automatically segment an entire image into meaningful regions without any human guidance. The results weren’t perfect — boundaries were often rough and small objects were frequently missed — but the paradigm shift was profound. Segmentation had moved from a human-guided process to a largely automated one.
U-Net: Precision for Medical Imaging
Published in 2015, U-Net introduced a symmetric encoder-decoder architecture with skip connections that preserved fine spatial details through the network. Originally designed for medical image segmentation (where precise boundaries are critical for diagnosing tumors or measuring organs), U-Net achieved remarkable accuracy even with limited training data.
U-Net’s architecture became one of the most influential designs in deep learning. Its core idea — encoding an image down to a compact representation and then decoding it back to full resolution with help from skip connections — appears in countless segmentation models to this day.
Mask R-CNN: Instance-Level Understanding
While semantic segmentation assigns a class to every pixel, it doesn’t distinguish between individual objects of the same class. In a photo of three people, semantic segmentation labels all person pixels the same way. Mask R-CNN (2017) by He et al. solved this with instance segmentation — identifying and separately masking each individual object.
Mask R-CNN combined object detection (finding bounding boxes around objects) with pixel-level segmentation within each box. For the first time, a computer could not only say “these pixels are a person” but “these pixels are Person #1, those pixels are Person #2, and those over there are Person #3.”
This capability unlocked applications from autonomous driving (tracking individual cars and pedestrians) to photo editing (selecting a specific person in a group photo). But Mask R-CNN and similar models had a significant limitation: they could only segment object categories they were trained on. A model trained to segment people and cars couldn’t segment an unusual object like a rubber duck or a specific piece of furniture without retraining.
The Foundation Model Era: Segment Anything
The most recent and arguably most transformative development came with foundation models — large AI models trained on massive datasets that can generalize to virtually any task.
SAM: Segment Anything Model (2023)
Meta’s Segment Anything Model (SAM), released in 2023, represented a fundamental shift in segmentation technology. Trained on over 1 billion masks across 11 million images, SAM could segment any object in any image — not just the categories it was explicitly trained on.
SAM’s key innovation was its promptable interface. Instead of requiring class labels or extensive training data, SAM accepts simple spatial prompts — a click, a box, or a rough mask — and generates a high-quality segmentation from them. (The paper also explored prompting by text description, but that remained a research demonstration rather than part of the released model.)
It also answered a question every interactive tool eventually hits: a single click is ambiguous. Click on someone’s shirt and you might mean the shirt, the torso, or the whole person. SAM’s answer was to return several candidate masks at different scopes along with confidence scores, leaving the choice to the caller rather than guessing.
This made professional-quality segmentation accessible to everyone. No training data needed. No category limitations. No technical expertise required. Just click on what you want, and SAM figures out the rest.
SAM2: Faster, Better, and Video-Ready (2024)
SAM2 improved on the original with better accuracy, faster inference, and the ability to segment objects in video (tracking them across frames). The architecture was refined to be more efficient, making it practical to run on consumer hardware rather than just research servers.
The multi-mask answer to ambiguity carried over from SAM, and SAM2 improved the quality of what comes back — challenging boundaries like hair, transparent objects, and objects with holes are handled more accurately than by its predecessor. The headline addition is memory across frames: SAM2 can carry a selection through a video, which is what turns segmentation from a per-image operation into a tracking one.
The progression across eras can be seen by what the user has to do to get a mask:
flowchart LR
M["Manual era<br/>Trace every pixel<br/>(minutes to hours)"]
E["Edge-based era<br/>Click + algorithm<br/>(seconds to minutes)"]
D["Deep learning era<br/>Pre-trained categories<br/>(seconds, fixed classes)"]
F["Foundation model era<br/>Click anything<br/>(milliseconds)"]
M --> E --> D --> FSAM2 in Practice: WobblePic
WobblePic puts SAM2 to work on an unusual application: physics-based image interaction. When you click on an object, SAM2 generates a segmentation mask defining its boundary, and that mask becomes a per-vertex weight on the deformation mesh — the object stretches and bounces under your cursor while everything outside the mask stays anchored where it started.
This is the whole arc compressed into a single click. What once took a retoucher hours of lasso work now happens in about fifty milliseconds, on a model running locally — through ONNX Runtime with DirectML on Windows, Apple’s CoreML build on Apple Silicon, and ONNX on the CPU for Intel Macs. Nothing is uploaded anywhere.
AI Segmentation with SAM2 covers how those three paths differ and why they had to; the tutorial page walks through the segmentation controls.
What’s Next?
Current research pushes on real-time video segmentation at higher resolutions, 3D segmentation from single images, selection driven by natural language, and tighter coupling with generative models so that a mask becomes an edit rather than just a selection.
The through-line across all four eras is the same: each one moved the work of deciding where the boundary goes from the person to the machine. You drew it yourself. Then you seeded an algorithm and it drew the rest. Then a model drew every boundary in the image, but only for the categories it had been taught. Now you indicate one object and the model decides the rest, whatever that object happens to be.
What has not changed is that a mask is still an opinion about where an object ends — modern models are simply right often enough that we stopped noticing the question was ever being asked.