Making of: Shadow-play motion in Sing It Into Memory

Sing It Into Memory, the opening track of The Dreamers, is meant to read as a shadow play. Story figures are flat shadows on a cave wall or an amphitheater wall, firelight flickers, and the camera stays nearly still. Live-stage figures can stand in front of the wall, but the story itself should not step off it, grow a face, or morph into someone new. Quiet shots — an ember in a dark cave, an empty stage, a night beach — should stay as empty as the still.

This is what happened when an image-to-video model was asked to add that motion, including the parts that did not work, and the local compositing method now used instead.

The goal

Hold the accepted still as the first frame, then add only the motion the scene calls for: a slow push-in, a light flicker, smoke or sparks that already exist, waves that already exist. No new people, no new fires, no new props. Action has to stay on the wall.

A small test: five clips

A small paid test came first, so the shadow-play motion could be checked before a larger batch. On 9 November 2026, five clips were generated with an image-to-video model at 720p 16:9 and 24 frames per second. The mix was two cave-wall Iliad scenes and three amphitheater Odyssey scenes, including both quiet and action stills.

Scene	Role
13	Quiet Iliad wall (Troy, sentries)
37	Quiet Odyssey wall (Ithaca lamp)
60	Action Iliad wall (Achilles vs Hector)
61	Action Odyssey wall (storm and trident)
62	Action Odyssey wall (Sirens)

All five completed, with no retries.

What worked. Scene 61 was the best of the five. Poseidon stays a flat shadow on the stage wall and raises the trident; the shadow ship pitches; the waves heave; they do not step off the wall. That is the language the video wants: shadows moving on the wall while the real stage and audience stay in front.

Scene 62 was the next-best action test: Odysseus and the Siren shadows stay on the wall and do not fly off as 3D birds, though several new braziers spawn in the empty orchestra.

What failed. Scene 60 is the exact failure the test was meant to catch. The opening frame still reads as wall shadows, but within about a second Achilles’ legs overlap the foreground fire, and by mid-clip both warriors are dueling on the cave floor as 3D silhouettes.

Scene 37 holds the wall image at first, then invents two foreground onlookers at the edges of the stage, one with a readable profile.

Scene 13 mostly holds, but the two tiny sentry shadows on the rampart morph into a dark banner.

The action-peel problem and the invented-onlookers problem were already clear. Prompts for the next batch were tightened from those lessons: camera locked, shadows stay on the wall, no new people, no faces, no identity changes.

The quiet batch: 41 clips, and 22 failures

On 10 November 2026, a batch of 41 quiet clips was generated the same way — every remaining quiet scene from the song. No action scenes were included. All 41 completed on the first pass.

Quality checks found a systematic problem: on quiet or empty stills, the model often invents a gathering of people by the middle of the clip — the same failure as test Scene 37.

A three-clip probe used an even stronger hold-the-first-frame prompt: if the first frame has no people, the entire clip should have no people, and it should not invent a campfire gathering.

Scene	First-pass problem	Retry result
44	Empty stage fills with a line of figures	Failed the same way (figures, and some animal silhouettes, appear mid-clip)
51	Empty star-trail ruins fill with a ring of people around a new fire	Failed the same way
81	Empty beach fills with a line of people and a fire	Failed the same way

The tighter prompt did not fix it. There were no further retries. Twenty-two of the 41 clips failed.

The pattern is consistent. The clips that work tend to already have a clear subject in the still: the bard, a weaver, an audience, father and child, the child with the lyre. Scene 77 (a child and a faint bard shadow) is one of the better quiet clips. The ones that fail tend to be empty, nearly empty, or very minimal — an ember, an empty stage, an empty coast, a fading glow — and the model fills them with a story-circle of people, often around a new fire, sometimes with children, sometimes gone again by the last frame. Scene 18 was a different kind of miss: hands morph into wave shapes.

Stricter prompts did not change the tendency. Three retries on the emptiest stills failed the same way.

The insight

Image-to-video models of this kind treat an empty frame as a scene that still needs a story. Given a cave, a theater, or a beach and a few seconds of motion, they invent an audience. They do best when the still already contains a clear subject to hold onto, and they are unreliable when the job is to leave a frame empty.

That is a poor match for a shadow-play video whose quiet shots are supposed to stay quiet. More retries of the same model would likely have bought more invented crowds.

The pivot: local compositing

The alternative is the same kind of work that would be done in After Effects: Ken Burns push-ins and slight pans, a few depth layers with soft masks, firelight as a brightness and warmth pulse, and a few pixels of drift on smoke, sparks, or existing wave highlights. Nothing new is drawn.

A repeatable local renderer was built with ordinary compositing and video tools (OpenCV, NumPy, and ffmpeg), with no paid model calls. Three failed quiet scenes were used as samples:

Scene	Still	Local motion	Duration
11	Cave, bard from behind, a single teardrop ember	Slow push-in toward the ember, warmth pulse, existing sparks held	about 5 seconds
44	Empty dusk amphitheater, a lyre in the orchestra, stage braziers	Slow push-in toward the empty stage, firelight pulse, existing smoke columns drift	about 3 seconds
81	Night pebble beach, moonlight path, no people	Slight pan along the shore, shimmer on existing highlights, a slight breathe in the existing foam	about 3.5 seconds

The clips are 1280×720 at 24 frames per second. The model-generated versions were kept, so the two methods can be compared.

On the model side, the middle and end of all three shots fill with people (and, in a retry of Scene 44, animal silhouettes). On the local side, the still’s contents are what remain: the ember and the bard, the empty stage and the lyre, the empty beach.

As of 10 November 2026, the local method is now used for 23 scenes. Known limits, written honestly:

• The motion is a living still, not a full physical simulation. Scene 81 shimmers and breathes along the foam that is already in the frame; it does not draw new wave crests, so it does not truly wash in and out.
• Depth layers are crude soft masks. A slightly loose figure mask can carry a thin halo of background with the near layer. Smoke is a warp of existing plumes and can smear a little at the top of a column.
• Flicker and drift are gentle by design, so that nothing new appears.

Lessons learned

The test was worth running. Scene 61 showed that the model can move a shadow on a wall when the still already has that shadow as the subject, and Scene 60 showed that fight action is where it peels figures into 3D. That was useful. The quiet batch then showed a second failure mode that the tighter prompts did not fix: emptiness is treated as a vacancy to fill.

Paid image-to-video is a poor tool for “keep this empty frame empty.” It is a better tool for a shot that already has a clear subject you want to see move. When the job is a slow camera move, a fire pulse, or a little smoke on a still that must not gain new characters, a local compositing pass is cheaper, more predictable, and easier to constrain.

Retries of the same model, with a stronger version of the same instruction, are not a plan once the failure is systematic.

What we’d tell anyone attempting the same thing

Problem	Reach for	Why
A quiet or empty still that must stay empty: slow push-in, firelight flicker, smoke or highlight drift, no new figures	Local 2.5D compositing (Ken Burns moves, soft depth layers, and modulation of existing pixels)	The model will invent a crowd. A pixel-level pass cannot.
A still that already has a clear subject, especially wall-shadow action that should stay on the wall	A small paid image-to-video test first, with a hard stopping point	It can work (Scene 61). It can also peel figures off the wall (Scene 60). Watch before running a batch.
A systematic failure (empty frames fill with people; figures leave the wall)	Stop retrying and change the method	A tighter prompt did not fix Scenes 44, 51, or 81. More of the same would have bought more of the same.
A batch of many quiet shots	Sort by whether the still has a clear subject. Plan the empty ones for a local pass.	In this batch, 22 of 41 failed, and they were the empty or minimal stills.
Any generated clip	Lock the first frame to the accepted still, keep a copy of each attempt, and do not overwrite a good take	So a failed attempt can be compared, and a good take is not lost.

A few practical tips:

• Treat “no new people” as a property of the method, not only of the prompt. If the tool’s prior is to fill a scene, the prompt will not reliably override it.
• Run a small test that includes both an action wall shot and a quiet empty shot. The two failure modes are different, and one test set can show both.
• When you change method, keep the earlier clips. The model-generated files stay as a record; local versions use a different name.
• Match local clip length to the scene, even if the paid model had a longer minimum duration.
• Check the first frame against the still before judging the motion. If the opening frame has already drifted, the rest of the shot will too.
• Do not draw new shapes to fake fire, smoke, or waves. Warp or modulate what is already there, or the “nothing new” rule is broken in a different way.

Credits: Malchome.