Animated WebP, GIF, and APNG playback feels trivial from the outside — most browsers do it for free, after all. But once you try to build a player yourself, the seemingly simple problem fragments into a half-dozen subtle ones: how do you keep frame timing accurate, what state should the player be in when nothing is playing, what happens when the decoder is slower than the framerate, and how do you stay responsive to user input without dropping frames? This post walks through how those pieces fit together, using a Pillow-based implementation as the running example.

(See Recording Real-Time Graphics for the producer side.)

The Surface vs. the Iceberg

From the outside, the player is just an image and a controls bar. Inside, it’s five interacting components:

flowchart LR
    Decoder["Decoder<br/>(Pillow)"] --> Cache["Frame<br/>Cache"]
    Clock["Clock<br/>(monotonic)"] --> SM["State<br/>Machine"]
    Cache --> SM
    Input["Input<br/>Events"] --> SM
    SM --> GL["GL Renderer"]

Five components, each with a job. The next sections walk through them.

The Timing Model

Animated images store a per-frame duration, not a fixed framerate. A typical animated WebP loop looks like this — note that frame 3 displays for only half as long as the others (an intentional fast cut):

gantt
    title Animated WebP frame timing (one loop ≈ 350 ms)
    dateFormat x
    axisFormat %L ms
    section Frames
    Frame 1 (100 ms) :0, 100
    Frame 2 (100 ms) :100, 200
    Frame 3 (50 ms — fast cut) :200, 250
    Frame 4 (100 ms) :250, 350

Both Pillow and the underlying file formats expose this as info["duration"] (in milliseconds) on each frame. A naive player ignores this and just calls time.sleep(1/30) between frames — which produces visible drift on any image where the durations vary, and outright wrong playback on simple cases like animated emoji that often use 50ms or 80ms frames.

The right pattern uses a monotonic clock and an absolute deadline per frame:

import time

start = time.monotonic()
deadline = start
for frame in frames:
    deadline += frame.duration_ms / 1000.0
    render(frame)
    sleep_until(deadline)

time.monotonic() is critical: it never goes backward (unlike wall-clock time, which can if the user adjusts their system clock or if NTP corrects a drift). Drift correction by carrying the deadline forward — instead of resetting it each frame — is what makes a multi-minute animation stay in sync.

The durations are not there when you need them

The loop above assumes you know every frame’s duration up front. For animated WebP in Pillow, you do not. Seeking to a frame is not enough to populate info["duration"] — the value only appears once that frame is actually decoded. Reading all durations at open time therefore means decoding the entire file at open time, which is precisely the stall you were trying to avoid.

The way out is to start with an estimate and correct it in flight:

  1. At load, take the first frame’s real duration and fill the rest with a placeholder — 40 ms, about 25 fps, a plausible value for the overwhelming majority of animations.
  2. As the background decoder works through the file, it overwrites durations[i] with each frame’s true value.
  3. Playback takes a snapshot of the duration table when it starts, and re-snapshots at every loop boundary.

The first pass through the animation runs on estimates, and every pass after it runs on measured values. Since playback begins paused on frame 1 and the decoder is usually finished before anyone presses play, most users never see the estimated pass at all.

One detail in step 1 matters more than it looks: filling the placeholder with the first frame’s duration instead of a fixed 40 ms seems more principled, and it is worse. Screen recordings routinely open with one long frame while the recorder settles, and propagating that value makes the whole animation crawl. An arbitrary-but-typical constant is the safer wrong answer.

Re-snapshotting only at loop boundaries — rather than continuously — is what keeps this from being visible. Frame timings never change midway through a cycle, so nothing stretches or jumps while you are watching it; the correction lands at the seam.

The State Machine

The player has three primary states:

stateDiagram-v2
    [*] --> INACTIVE
    INACTIVE --> PAUSED: attach(animated)
    PAUSED --> PLAYING: play()
    PLAYING --> PAUSED: pause()
    PLAYING --> PAUSED: rewind_and_pause()
    PAUSED --> INACTIVE: detach()
    PLAYING --> INACTIVE: detach()
    INACTIVE: INACTIVE<br/>(no animation attached)
    PAUSED: PAUSED<br/>(holding a frame)
    PLAYING: PLAYING<br/>(clock advancing)

Three states is fewer than most people first sketch, and the discipline is worth keeping. Every extra state is a new set of transitions to get wrong, and “the user is interacting with the current frame” is almost never a state of the player — it is a state of the application that happens to have paused the player.

PAUSED is the default for any newly opened animated image. This sounds counterintuitive — why not autoplay? — but autoplay is hostile to two things: accessibility, and workflows where the user wants to inspect one frame. Browsers learned this the hard way; most now respect prefers-reduced-motion.

The interesting transition is rewind_and_pause(): clicking the image while it plays does not pause where it is, it jumps back to frame 1 and pauses there. That looks heavy-handed until you know why. Our player sits inside an image viewer that can deform whatever it is showing, and the segmentation embedding needed for that is precomputed for the first frame only. Pausing on frame 40 and deforming it would mean the mask belongs to a frame the user is not looking at. Rewinding is the honest option: it guarantees that what you interact with is what was prepared.

The general lesson is the one worth carrying: input cancels playback, and when your interactive features have preconditions, the cancel path should land somewhere those preconditions hold.

Decoder Pacing: Wait or Skip?

This is where naive implementations fall apart. The decoder isn’t free — Pillow has to seek, decode, and convert each frame. On a slow machine or a large APNG, decoding can occasionally take longer than the frame’s duration:

Frame budget: 100 ms each.
                                              decode took 130 ms
                                                   v
Frames:   --[1]--[2]--[3]--[4]--[5]--[6]X--[7]--
            v    v    v    v    v
                                         30 ms behind wall-clock

There are two possible responses:

Skip strategy: drop the late frame and jump to whichever frame the wall clock now demands.

                          wall-clock now
                              v
Frames:   --[1]--[2]--[3]--[4]--[6]--[7]--
                                  ^
                              frame 5 skipped

Wait strategy: render frame N when it’s ready, even if late. The clock catches up on the next frame budget.

Frames:   --[1]--[2]--[3]--[4]--[5]----[6]--[7]--
                                  ^
                              130 ms (late, but rendered)

The skip strategy keeps wall-clock sync but causes visible jumps. The wait strategy preserves visual continuity but lets jitter accumulate. For most viewers, wait is the right default, because:

  1. Animated images are usually short (under 10 seconds). Drift never gets large enough to matter.
  2. A skipped frame is more visually disturbing than a slightly delayed one — humans notice missing frames more than slow ones.
  3. If the user doesn’t notice a 30ms hiccup, you’ve gained nothing by dropping a frame.

Waiting properly means stopping the clock

There is a second, sharper version of this problem that only appears once decoding is asynchronous. If a background thread is filling the frame buffer while playback runs, the player will eventually ask for a frame that does not exist yet — not because decoding was slow this tick, but because the decoder has not reached that index at all.

Falling back to “hold the current frame and let time keep running” is the intuitive fix and the wrong one: when the frame finally lands, the clock has moved on, and the player skips ahead to wherever wall-clock time now points — throwing away exactly the frames it was waiting for.

So the player freezes elapsed time itself while waiting. It records the elapsed value at the moment it stalls and keeps rebasing its start timestamp against that frozen figure, so real seconds pass while playback time does not:

# Entering the tick, if we stalled previously, pin elapsed time to the frozen value
if self._wait_elapsed_ms is not None:
    self._play_start_monotonic = now - self._wait_elapsed_ms / 1000.0

Every frame is then shown, in order, however far behind the decoder falls. The animation plays slower than real time in the worst case, which is a far better failure than dropped frames — and if the decoder stalls permanently, moving to another image tears the whole thing down and recovers naturally.

Memory Strategies

Three reasonable approaches:

StrategyMemoryDecode latencyBest for
Preload allHigh (full image × N frames)Zero per frameShort animations, plenty of RAM
On-demandLow (one frame)Full decode each frameLong animations, memory-constrained
Sliding windowMedium (k frames around current)Zero for in-window, full for outBalanced workloads

A useful heuristic:

total_frame_pixels = width × height × N_frames × 4 bytes
if total_frame_pixels < 200 MB:
    preload_all()
else:
    sliding_window(k=8)

200 MB is roughly the point where a typical desktop user starts noticing memory pressure from a single image, and 8 frames of slack covers most decoding hiccups without ballooning memory.

For an interactive image viewer where the user can scrub or jump around, preload-all is dramatically simpler and almost always feasible — animated images are usually small. For a video player, sliding window is mandatory.

Format Quirks

Pillow handles all three popular formats, but each has gotchas:

FormatQuirkMitigation
WebP animatedseek() alone does not populate info["duration"] — the real value appears only after the frame is decoded.Placeholder durations corrected in flight (above).
GIFFrames are in palette mode, not RGB, and delays are stored in centiseconds, so anything not a multiple of 10 ms is quantised..convert("RGBA") before rendering; expect timing to land on 10 ms boundaries.
APNGPillow has supported it since 7.1, but a file with a single frame is indistinguishable from a static PNG through the same API.Check n_frames > 1 rather than trusting the extension.

A clean abstraction normalizes these into a single Frame(image_rgba, duration_ms) representation right after decode, and the rest of the player never needs to know which format produced it.

Edge Cases Worth Handling

Two surprises that bit me in practice:

Two decoders on one file. Our viewer preloads neighbouring images so navigation is instant, and in a folder containing a single animation — or at the point where the index wraps around — the “next” image is the one already open. That spawned a second decoder thread against the same file, and the symptom was not a crash but corrupted frames in the back half of the animation: two threads seeking through one Pillow image object, quietly interleaving its internal position.

The fix is to skip preloading anything already open or cached, which is one line, but the diagnosis took considerably longer than that — because the first half always looked fine.

The wider lesson: a decoder that holds internal seek state is not a pure function of the file. Treat it as a resource with an owner, and be certain only one part of your program is that owner.

Truncated files. Some animated WebPs found in the wild have a frame count in the header that doesn’t match the actual decoded frames available. Catch EOFError from seek() and treat the last successfully decoded frame as the loop point.

Header says:    24 frames
Decoder finds:  18 frames before EOFError
                                |
                                v
                Loop back to frame 1 here

Wrapping Up

A frame-accurate animated image player is roughly 300-500 lines of Python on top of Pillow once you’ve nailed:

  • Monotonic-clock-based deadline timing
  • A small explicit state machine — three states, not five
  • A “wait, don’t skip” pacing default, where waiting means freezing playback time, not holding a frame while the clock runs on
  • Durations treated as data that arrives late: estimate, correct in the background, re-snapshot at loop boundaries
  • Resume position stored as a time offset, not a frame index, so a corrected duration table cannot move you somewhere you did not pause
  • Preload-all memory unless the image is unusually large
  • Per-format normalization at the decode boundary

If you’re considering building one, the biggest hidden cost is testing — the only way to catch timing bugs is with a stopwatch and a wall of test images that exercise every duration pattern you can think of. Open-source test corpora like the WebP gallery and Mozilla’s APNG samples are good starting points.

Once the player is working, the natural next features are frame export (let the user save individual frames as PNG), per-frame seek scrubbing, and palette-aware GIF rendering for the rare case where palette transparency matters. Each is its own rabbit hole — but with a solid timing model and state machine in place, they’re all straightforward additions rather than rewrites.