ResourcesVideo 05

AI Video Generation in ComfyUI: The Complete Guide

The workflows, prompts and model links from the video.

18 chapters, each with its workflow, input files and prompt, ZIP, 50.5 MB

Workflows

The workflows, input images and prompts used throughout the video are all available here.

A young red-haired woman in a green hooded cloak smiling at a dark blue scaled dragon egg in her hands, in a stable.

01

Text to Video

A prompt in, a video with sound out.

Prompt

integrated_multimodal_description: [Shot 1] Live-action, cinematic, a fantasy film shot on 35mm film with fine grain and a shallow depth of field: a medium close-up, framed from the chest up, of a young woman kneeling in a stone castle stable at dawn. She is in her early twenties, with long copper hair in a loose braid over one shoulder, freckles, bright green eyes and fine natural skin texture, and she wears a dark green wool cloak over a cream linen dress. In her cupped hands, at chest height, she holds a large, smooth egg covered in small deep blue scales. Warm light from an iron lantern on the wall to her left lights the side of her face, and behind her, pale dawn light falls through a high arched window onto soft, out-of-focus straw and wooden beams, with dust drifting in the light. The egg trembles in her hands, and one fine, dark hairline crack slowly creeps across the shell near its top, while the shell stays whole. Her eyes widen a little and a small smile spreads across her face, and looking down at the egg, the young woman with a soft, breathy voice (S1) whispers quietly, with gentle excitement: <d>[English] It's finally coming out.</d> The camera pushes in with small amplitude at slow speed. [Shot 2] At 00:02.600, the camera cuts to a close-up of the same egg in her hands, filling the frame, the fine hairline crack near its top. The shell is real eggshell, thin, hard and brittle, deep blue scales on the outside and pale cream on the inside. Something pushes from inside: the shell beside the crack bulges outward, and with a small dry snap one irregular, curved piece of shell, about a third as wide as the egg, breaks loose with rough, slightly chipped edges. A baby dragon's head pushes up through the opening, lifting that piece of shell on top of its head, so the broken piece sits right beside the ragged hole it came from, the same shape and size. The piece slides off its head, tumbles down and drops into the straw. The baby dragon, with glossy deep blue scales, two small curled horns and big golden eyes, looks out of the ragged hole with the broken edge of the shell around its neck, blinks slowly up at her and lets out a tiny, high chirp. The camera holds a static shot. overall_soundscape: The shell creaks and gives one small, dry snap as the piece breaks off and drops softly into the straw. Straw rustles under her knees, and birds sing faintly outside the high window. A horse snorts softly in a nearby stall. non_diegetic_music: A soft Celtic harp melody at a slow tempo over a low sustained cello note, rising gently as the dragon appears.

Download JSON for Text to Video
A young woman in a cream lace dress by a tall window with white flowers, looking back over her shoulder.

02

Image to Video

Starts the video from your picture.

Prompt

For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced. integrated_multimodal_description: [Shot 1] Live-action period drama, cinematic, shot on 35mm with soft grain: the young woman in the cream linen dress with the lace collar from <Picture 1> stands by the tall window in the old manor drawing room, the white peonies soft in the foreground, her eyes already wet with tears. She draws a slow, shaky breath, a tear slowly rolls further down her cheek, and the young woman with a soft, trembling voice (S1) says, <d>[English] Promise me you'll come back.</d> Then she lowers her eyes. Soft window light, very shallow depth of field. The camera pushes in with small amplitude at slow speed. overall_soundscape: A hushed old room, the soft patter of rain on the window glass. non_diegetic_music: A tender, aching piano and cello theme.

Download JSON for Image to Video
A white hillside village above the sea at night under the Milky Way, the night picture this video ends on.

03

First and Last Frame

You give the first and last picture, the model makes what happens between them.

Prompt

How the reference pictures align with the target video - Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot 1) aligns with the 5.17-second mark of the target video. integrated_multimodal_description: [Shot 1] Live-action time-lapse, cinematic: the whitewashed cliffside village above the sea, as in Picture 1. The camera holds a static shot as time passes quickly: the warm golden light slides off the white walls, the sky deepens to blue and then to a clear night, stars appear one by one and the Milky Way rises over the sea, and warm lights switch on in the windows and along the stairway, all mirrored in the calm water below, pose and composition established by Picture 2 at the end of the shot. overall_soundscape: A quiet seaside evening: soft waves, a light breeze and distant church bells fading into crickets. non_diegetic_music: A slow, warm ambient piano with soft strings that swell as the stars appear.

Download JSON for First and Last Frame
A vintage red and cream convertible under an autumn tree on a leaf-covered lane beside a stone cottage.

04

Seamless Loop

The same picture as first and last frame, so the video loops.

Prompt

How the reference pictures align with the target video - Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot 1) aligns with the 5.17-second mark of the target video. integrated_multimodal_description: [Shot 1] Live-action, cinematic, warm golden autumn light: the vintage convertible parked under the great maple tree from Picture 1. The camera holds a perfectly static shot on a tripod and the light stays exactly the same the whole time. Golden, orange and red maple leaves keep drifting slowly down from the canopy and tumble gently through the warm backlight, a soft breeze stirs the branches and the leaves on the cobblestones, and fine dust glows in the sunbeams. The leaves keep falling gently and evenly from the first frame to the last; the car stays perfectly still. This is one seamless loop: the shot ends on exactly the same frame it starts with, Picture 2. overall_soundscape: N/A. non_diegetic_music: N/A.

Download JSON for Seamless Loop
A hand placing a Model Chief perfume bottle with a pearl cap on a cream tray with petals and a compact.

05

Last Frame

The video builds up to your picture.

Prompt

How the reference pictures align with the target video - Picture 1 (from Shot 1) aligns with the 5.17-second mark of the target video. integrated_multimodal_description: [Shot 1] Live-action luxury perfume commercial, intimate and calm, soft warm morning window light from the left with gently swaying dappled leaf shadows. The shot opens on the glossy cream ceramic tray on the blush-pink tabletop with the loose pink peony petals, the strand of pearls, the open gold compact and the folded linen napkin, and an empty space on the tray. A young woman's slender, graceful hand with soft natural nails and a delicate thin gold ring enters from the top right holding the rounded square pink glass perfume bottle with the large pearly white sphere cap and the cream MODEL CHIEF label from Picture 1 by its body, lowers it slowly and sets it down gently on the tray in the empty space, turns it slightly so the label faces the camera, then lets go and glides smoothly back out of the frame to the top right. The camera holds steady with a very slow push in. The last moments are calm and still, with no hand in the frame, and the shot ends exactly on Picture 1. overall_soundscape: A soft clink of glass on ceramic, the faint rustle of linen, birdsong outside the window. non_diegetic_music: An elegant, minimal piano melody with soft, warm strings.

Download JSON for Last Frame
The woman from the reference picture smiling at a letter in her hands, by a tall window in a green room.

06

Reference to Video

Puts the person from your picture in a new scene.

Prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with auburn-brown hair pinned up in a loose low bun with soft loose strands, and a cream linen dress with a lace collar. summary: [reference generation] The target video shows <Subject 1> in a softly lit 19th-century drawing room, reading a letter by a tall window and smiling. retention_analysis: <Subject 1> (appears in [Shot 1], [Shot 2]): fully_preserved - her face, hair, cream linen dress and lace collar are retained. detailed_description: The target video is a live-action 19th-century period drama shot on 35mm film with fine grain, soft natural window light and a shallow depth of field. [Shot 1] A medium close-up from the chest up: <Subject 1> stands by a tall window with white curtains in an elegant drawing room with pale green walls, soft overcast daylight on her face. She is reading a letter that she holds still in both hands just below the frame; her eyes move slowly along the lines and a slow, hopeful smile spreads across her face. The camera pushes in with small amplitude at slow speed. [Shot 2] At 00:02.600, the camera cuts to a close-up of <Subject 1> by the window, her face filling the frame, white peonies softly blurred behind her; she lifts her eyes from the letter toward the window, her eyes bright with joy and a slow smile on her lips, and <Subject 1> (S1), with a soft, warm young female voice, says, <d>[English] He's coming home.</d> overall_soundscape: A quiet room: a clock ticking and birdsong outside the window. non_diegetic_music: A gentle solo piano melody with soft strings.

Download JSON for Reference to Video
The man from the second reference picture in a dark coat and white cravat, smiling by a tall window.

07

Reference to Video: Two Characters

Two people from two pictures in one scene.

Prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with auburn-brown hair pinned up in a loose low bun with soft loose strands, and a cream linen dress with a lace collar. <Subject 2> is the man in <Picture 2>, with dark wavy hair, light stubble and brown eyes, in a dark frock coat over a cream waistcoat and a white linen cravat. summary: [reference generation] The target video shows <Subject 1> and <Subject 2> meeting again in a softly lit 19th-century drawing room. retention_analysis: <Subject 1> (appears in [Shot 1], [Shot 3]): fully_preserved - her face, hair, cream linen dress and lace collar are retained. <Subject 2> (appears in [Shot 2]): fully_preserved - his face, dark wavy hair, stubble, frock coat, waistcoat and cravat are retained. detailed_description: The target video is a live-action 19th-century period drama shot on 35mm film with fine grain, soft natural window light and a shallow depth of field. [Shot 1] A medium close-up, framed from the chest up: <Subject 1> stands by a tall window in an elegant drawing room with pale green walls, close to the camera, her face almost toward the camera, turned slightly to the right, toward a door off-screen. Soft, even daylight from the window on the left of the frame lights her whole face, which is clear and in sharp focus. Her hands are empty and out of frame. From the very first frame her eyes are fully open, clear and bright, looking toward the door, and her head stays still. At the click of the door latch off-screen, her eyes lift a little and her lips part in quiet, hopeful surprise. The camera holds a static shot. [Shot 2] At 00:02.400, the camera cuts to a medium close-up of <Subject 2> in the doorway, his face in soft window light; he looks at her, smiles gently, and <Subject 2> (S1), with a low, warm male voice, says clearly and unhurriedly, <d>[English] I told you I'd come back.</d> [Shot 3] At 00:05.200, the camera cuts to a close-up of <Subject 1>; a slow, radiant smile spreads across her face, and <Subject 1> (S2), with a soft, warm young female voice, says, <d>[English] You're late.</d> She lets out a small, happy laugh. overall_soundscape: The door latch clicks, footsteps on wooden floorboards, a clock ticking and birdsong outside the window. non_diegetic_music: A gentle solo piano melody with soft strings.

Download JSON for Reference to Video: Two Characters
The woman from the reference picture speaking and smiling in a green drawing room, holding a tied bundle of letters.

08

Custom Voice

Your character speaks in the voice from your audio file.

Prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with auburn-brown hair pinned up in a loose low bun with soft loose strands, and a cream linen dress with a lace collar. <Audio 1> is the voice-timbre reference for <Subject 1> (S1). summary: [reference generation + audio reference] The target video shows <Subject 1> in a softly lit 19th-century drawing room, holding a bundle of old letters and quietly telling someone she kept every one of them. retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - her face, hair, cream linen dress and lace collar are retained. <Audio 1>: reference - its voice timbre guides the speech of <Subject 1> without copying the original signal. detailed_description: The target video is a live-action 19th-century period drama shot on 35mm film with fine grain, soft natural window light and a shallow depth of field. [Shot 1] A close-up framed from the shoulders up: <Subject 1> stands by a tall window with white curtains in an elegant drawing room with pale green walls, white peonies softly blurred behind her. Warm, soft afternoon daylight from the window lights her whole face, which is clear and in sharp focus. Her hands rest below the frame, holding a bundle of old letters tied with a faded blue ribbon. From the first frame she looks toward someone just off-camera, her eyes clear and bright, with a calm, warm and quietly happy expression, and her head stays still. Almost immediately, <Subject 1> (S1), in the voice referenced from <Audio 1>, says in a warm, gentle and sincere tone at a natural, unhurried pace, with a soft smile in her voice and a short pause at the comma, <d>[English] I kept every letter, every single one.</d> Her lips move in clear sync with every word. As she finishes, a fond, slightly shy smile spreads across her face and she glances down at the letters for a moment, then back up. The camera pushes in with small amplitude at slow speed. overall_soundscape: A quiet room: a clock ticking softly and distant birdsong outside the window; her voice is clear and close. non_diegetic_music: A gentle solo piano melody with soft strings, kept low under her voice.

Write the audio file's exact words in the prompt.

Download JSON for Custom Voice
The man from the reference picture speaking with a wide smile in a period drawing room.

09

Custom Voices: Two Characters

Each character gets their own voice file.

Prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with auburn-brown hair pinned up in a loose low bun with soft loose strands, and a cream linen dress with a lace collar. <Subject 2> is the man in <Picture 2>, with dark wavy hair, light stubble and brown eyes, in a dark frock coat over a cream waistcoat and a white linen cravat. <Audio 1> is the voice-timbre reference for <Subject 1> (S1). <Audio 2> is the voice-timbre reference for <Subject 2> (S2). summary: [reference generation + audio reference] The target video shows <Subject 1> and <Subject 2> reunited in a softly lit 19th-century drawing room: she tells him she kept every letter he wrote, and he answers that he read every one of hers many times. retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - her face, hair, cream linen dress and lace collar are retained. <Subject 2> (appears in [Shot 2]): fully_preserved - his face, dark wavy hair, stubble, frock coat, waistcoat and cravat are retained. <Audio 1>: reference - its voice timbre guides the speech of <Subject 1> without copying the original signal. <Audio 2>: reference - its voice timbre guides the speech of <Subject 2> without copying the original signal. detailed_description: The target video is a live-action 19th-century period drama shot on 35mm film with fine grain, soft natural window light and a shallow depth of field. [Shot 1] A close-up framed from the shoulders up: <Subject 1> stands by a tall window in an elegant drawing room with pale green walls, her face almost toward the camera, turned slightly to the right toward <Subject 2> just off-screen. Warm, soft afternoon daylight from the window lights her whole face, which is clear and in sharp focus. Her hands rest below the frame, holding a bundle of old letters tied with a faded blue ribbon. Her eyes are clear and bright, her expression calm, warm and quietly happy, and her head stays still. Almost immediately, <Subject 1> (S1), in the voice referenced from <Audio 1>, says in a warm, gentle and sincere tone at a natural, unhurried pace, with a soft smile in her voice and a short pause at the comma, <d>[English] I kept every letter, every single one.</d> Her lips move in clear sync with every word. The camera holds a static shot. [Shot 2] At 00:03.667, the camera cuts to a close-up of <Subject 2>, framed from the shoulders up, facing her and turned slightly to the left, his face in the same warm window light and in sharp focus. A slow, delighted smile spreads across his face, and right away <Subject 2> (S2), in the voice referenced from <Audio 2>, answers, using the low, smooth and relaxed voice timbre referenced from <Audio 2>, in a warm and tender tone at a natural, unhurried pace, with a short pause at the comma, <d>[English] And I read every one of yours, multiple times.</d> His lips move in clear sync with every word, and he holds her gaze with a warm smile as he finishes. The camera holds a static shot. overall_soundscape: A quiet room: a clock ticking softly and birdsong outside the window; both voices are clear and close. non_diegetic_music: A gentle solo piano melody with soft strings, kept low under their voices.

Write each audio file's exact words in the prompt.

Download JSON for Custom Voices: Two Characters
A woman from behind in a black satin bomber jacket embroidered with a crane and red sun, against a terracotta wall.

10

Multi-Shot Keyframes

Four pictures, each one starts a new shot.

Prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with sleek dark brown hair in a neat low bun and small gold hoop earrings, wearing an oversized black satin souvenir bomber jacket with black-and-gold striped collar and cuffs over a plain white t-shirt, a large embroidered golden crane, red rising sun and indigo waves on its back. <Picture 1> is the first frame of [Shot 1], showing <Subject 1> from the waist up, facing the camera against a sunlit terracotta wall with soft window shadows. summary: [keyframe completion + reference generation] The target video is a short fashion campaign film for the jacket in four shots: it begins with <Picture 1>, cuts to a close-up of the embroidery, then to <Subject 1> from behind, and ends on a close-up of her face. retention_analysis: <Subject 1> (appears in [Shot 1], [Shot 3], [Shot 4]): fully_preserved - her face, hair, earrings and the jacket stay the same. <Picture 1> ([Shot 1] first frame): fully_preserved - the video starts exactly from <Picture 1>. detailed_description: The target video is a fashion campaign film in a cinematic live-action style, warm late-afternoon sunlight, rich film-like colours. [Shot 1] The shot begins from <Picture 1>. The camera pushes in very slowly. From the first frame <Subject 1> lifts her chin and looks into the lens with calm confidence, then begins to turn away, the black satin shimmering. [Shot 2] At 00:02.208, the shot cuts to an extreme close-up of the back of the jacket: the large embroidered golden crane with spread wings across the big red rising sun, shiny gold, cream and red silk threads on black satin; the camera glides slowly sideways across the stitching as the warm light slides over the threads. [Shot 3] At 00:04.417, the shot cuts to a waist-up shot of <Subject 1> from behind, the whole embroidered crane, red sun and indigo waves on her back; she turns her head to glance back over her shoulder at the camera. [Shot 4] At 00:06.625, the shot cuts to a close-up of <Subject 1>'s face from the shoulders up, facing the camera, the black-and-gold striped collar and her gold hoop earrings in the warm sunlight; the camera pushes in slowly as she lifts her chin and a slow, confident smile spreads across her face. No captions or subtitles appear. overall_soundscape: The soft rustle of satin in a quiet room. non_diegetic_music: A minimal, stylish electronic beat with a warm bassline that runs through all four shots.

Download JSON for Multi-Shot Keyframes
The same jacket shot made in 8 steps: the crane and red sun on the back of the black jacket, terracotta wall behind.

11

Speed LoRA

The same four shots in 8 steps instead of 20.

Prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with sleek dark brown hair in a neat low bun and small gold hoop earrings, wearing an oversized black satin souvenir bomber jacket with black-and-gold striped collar and cuffs over a plain white t-shirt, a large embroidered golden crane, red rising sun and indigo waves on its back. <Picture 1> is the first frame of [Shot 1], showing <Subject 1> from the waist up, facing the camera against a sunlit terracotta wall with soft window shadows. summary: [keyframe completion + reference generation] The target video is a short fashion campaign film for the jacket in four shots: it begins with <Picture 1>, cuts to a close-up of the embroidery, then to <Subject 1> from behind, and ends on a close-up of her face. retention_analysis: <Subject 1> (appears in [Shot 1], [Shot 3], [Shot 4]): fully_preserved - her face, hair, earrings and the jacket stay the same. <Picture 1> ([Shot 1] first frame): fully_preserved - the video starts exactly from <Picture 1>. detailed_description: The target video is a fashion campaign film in a cinematic live-action style, warm late-afternoon sunlight, rich film-like colours. [Shot 1] The shot begins from <Picture 1>. The camera pushes in very slowly. From the first frame <Subject 1> lifts her chin and looks into the lens with calm confidence, then begins to turn away, the black satin shimmering. [Shot 2] At 00:02.208, the shot cuts to an extreme close-up of the back of the jacket: the large embroidered golden crane with spread wings across the big red rising sun, shiny gold, cream and red silk threads on black satin; the camera glides slowly sideways across the stitching as the warm light slides over the threads. [Shot 3] At 00:04.417, the shot cuts to a waist-up shot of <Subject 1> from behind, the whole embroidered crane, red sun and indigo waves on her back; she turns her head to glance back over her shoulder at the camera. [Shot 4] At 00:06.625, the shot cuts to a close-up of <Subject 1>'s face from the shoulders up, facing the camera, the black-and-gold striped collar and her gold hoop earrings in the warm sunlight; the camera pushes in slowly as she lifts her chin and a slow, confident smile spreads across her face. No captions or subtitles appear. overall_soundscape: The soft rustle of satin in a quiet room. non_diegetic_music: A minimal, stylish electronic beat with a warm bassline that runs through all four shots.

Download JSON for Speed LoRA
The woman from the cafe picture dancing on the sunny street, hands raised, in a denim jacket and white t-shirt.

12

Motion Transfer

The person in your picture does the moves from a video.

Prompt

subject_definitions: <Subject 1> is the woman in <Picture 1>, with a short black bob, an open faded light-wash denim jacket over a plain white t-shirt and black trousers. She has a round face with soft features, dark brown eyes, faint freckles across her nose and a small closed-mouth smile. summary: [reference generation] The target video shows <Subject 1> dancing a relaxed, laid-back groove on a narrow sunny side street outside a small corner cafe, with a chalkboard menu in the cafe window, a silver bicycle with a front basket parked against the wall and power lines overhead, following the guide movement beat by beat. retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - her face, hairstyle, eye colour, skin, clothes and the place from <Picture 1> are retained exactly; only her body movement is new. detailed_description: The target video is real live-action footage filmed on a locked-off camera, with bright low morning sun from the front-left with crisp shadows on the road, the same light as in <Picture 1>, natural skin texture with visible pores, individual hair strands and real fabric folds; it looks like a real dance video, not CGI. [Shot 1] A static medium shot frames <Subject 1> from the waist up in the centre of the frame, facing the camera, on a narrow sunny side street outside a small corner cafe, with a chalkboard menu in the cafe window, a silver bicycle with a front basket parked against the wall and power lines overhead. She sways gently from side to side with a small bounce in her knees, her shoulders rising and falling softly with the beat, rolls her hands loosely in front of her waist, softly snaps her fingers at chest height with a small, easy shoulder shimmy, sways her hips, nods to the beat with a calm smile and rests both hands back on her hips; every move is small, smooth and relaxed. Her body follows the guide movement exactly, with the same timing and speed. Her eyes stay clearly visible, her hands are bare, and her face stays clear, sharp, natural and the same in every frame. She does not speak or sing. The camera does not move and the background stays still. No captions or subtitles appear. overall_soundscape: Quiet street sounds and a bicycle bell far away. non_diegetic_music: A smooth, mid-tempo instrumental R&B groove with a soft kick, light finger snaps and a warm bass line, no vocals.

Download JSON for Motion Transfer
The same dance on the sunny street after the upscale pass, at 1920 x 1088.

13

Upscale

Motion Transfer plus a second pass, up to 1920 x 1088.

Prompt

subject_definitions: <Subject 1> is the woman in <Picture 1>, with a short black bob, an open faded light-wash denim jacket over a plain white t-shirt and black trousers. She has a round face with soft features, dark brown eyes, faint freckles across her nose and a small closed-mouth smile. summary: [reference generation] The target video shows <Subject 1> dancing a relaxed, laid-back groove on a narrow sunny side street outside a small corner cafe, with a chalkboard menu in the cafe window, a silver bicycle with a front basket parked against the wall and power lines overhead, following the guide movement beat by beat. retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - her face, hairstyle, eye colour, skin, clothes and the place from <Picture 1> are retained exactly; only her body movement is new. detailed_description: The target video is real live-action footage filmed on a locked-off camera, with bright low morning sun from the front-left with crisp shadows on the road, the same light as in <Picture 1>, natural skin texture with visible pores, individual hair strands and real fabric folds; it looks like a real dance video, not CGI. [Shot 1] A static medium shot frames <Subject 1> from the waist up in the centre of the frame, facing the camera, on a narrow sunny side street outside a small corner cafe, with a chalkboard menu in the cafe window, a silver bicycle with a front basket parked against the wall and power lines overhead. She sways gently from side to side with a small bounce in her knees, her shoulders rising and falling softly with the beat, rolls her hands loosely in front of her waist, softly snaps her fingers at chest height with a small, easy shoulder shimmy, sways her hips, nods to the beat with a calm smile and rests both hands back on her hips; every move is small, smooth and relaxed. Her body follows the guide movement exactly, with the same timing and speed. Her eyes stay clearly visible, her hands are bare, and her face stays clear, sharp, natural and the same in every frame. She does not speak or sing. The camera does not move and the background stays still. No captions or subtitles appear. overall_soundscape: Quiet street sounds and a bicycle bell far away. non_diegetic_music: A smooth, mid-tempo instrumental R&B groove with a soft kick, light finger snaps and a warm bass line, no vocals.

Download JSON for Upscale
The dancer from the source video with her white tank top changed to deep cobalt blue, same terracotta backdrop.

14

Edit Videos

Changes one thing in a video from a written instruction.

Prompt

subject_definitions: <Subject 1> is the young woman dancing in <Video 1>, with dark hair in a high ponytail, a white ribbed tank top and black high-waisted trousers. <Subject 2> is the photo-studio environment in <Video 1>: a plain, seamless terracotta-orange paper backdrop lit by soft warm studio light with a gentle glow in the centre, and black light stands at the right edge of the frame. summary: [video editing] The target video is an edited version of <Video 1>. The colour of <Subject 1>'s tank top changes from white to a deep cobalt blue in every frame; her dance, her face, her trousers, <Subject 2>, the light and the camera stay exactly the same. retention_analysis: <Subject 1> (appears in [Shot 1]): partially_preserved - her face, hair, body, trousers and every dance move are retained; the tank top is now deep cobalt blue with the same fine ribs and fit. <Subject 2> (appears in [Shot 1]): fully_preserved - the terracotta-orange backdrop, the warm light and the light stands are retained. <Video 1> (framing and timing): fully_preserved - the locked-off framing and the timing are retained. detailed_description: The target video is real live-action dance footage filmed on a locked-off camera with soft warm studio light. [Shot 1] <Subject 1> dances in front of <Subject 2> exactly as in <Video 1>, in a fitted, finely ribbed deep cobalt-blue cotton tank top and black high-waisted trousers. The whole tank top is the same rich cobalt blue from the straps to the hem in every frame, with real fabric folds that stretch with her body; no white is left on it. The camera does not move. No captions or subtitles appear. overall_soundscape: The soft shuffle of her feet on the studio floor. non_diegetic_music: A smooth, mid-tempo instrumental R&B groove with a soft kick, light finger snaps and a warm bass line, no vocals.

Download JSON for Edit Videos
The vertical convertible clip widened to 16:9: the whole black car on a gravel drive, oak trees and a manor house behind.

15

Outpaint

Turns a vertical video into a wide one.

Prompt

subject_definitions: <Subject 1> is the woman in <Picture 1>, in a cream silk headscarf, black cat-eye sunglasses, a black wool coat and cream leather gloves. <Subject 2> is the long glossy black vintage convertible with tan leather seats in <Picture 1>. <Subject 3> is the English country estate in autumn in <Picture 1>: a gravel drive, tall old oak trees and falling golden leaves. summary: [reference generation] A wide 16:9 shot of <Subject 1> sitting behind the wheel of <Subject 2>, parked on the gravel drive of <Subject 3>, with the whole car and the estate in view. retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - her headscarf, sunglasses, coat, gloves and her movement are retained. <Subject 2> (appears in [Shot 1]): fully_preserved - the glossy black body, the chrome trim and the tan leather seats are retained, with the rest of the car now in view. <Subject 3> (appears in [Shot 1]): fully_preserved - the gravel drive, the oak trees and the falling leaves are retained and continue to both sides. detailed_description: The target video is a live-action 1960s fashion film, shot on 35mm with soft grain. [Shot 1] A wide 16:9 frame: <Subject 1> sits behind the wheel of <Subject 2>, parked on the gravel drive of <Subject 3>. The whole car is in view: on the left its long black bonnet, chrome grille and round chrome headlamp; on the right the chrome-trimmed door, the rear wing and the sloping boot with a small chrome tail-lamp. Behind the car the gravel drive leads between tall old oak trees to a pale stone Georgian manor house; on the left a wide lawn covered in orange leaves under the oaks, on the right a clipped hedge and a stone urn at the edge of the drive. Nobody else is in the scene. She slowly turns her head toward the camera and, with one gloved hand, lowers her sunglasses a little to look over them, a faint confident smile. Golden autumn leaves drift down across the whole frame and a gust lifts the loose end of her silk scarf. The camera pushes in with small amplitude at slow speed. overall_soundscape: Dry autumn leaves skittering over gravel, a soft breeze and the low idle of a vintage car engine. non_diegetic_music: A slow, moody retro soul groove with brushed drums and a muted trumpet.

Download JSON for Outpaint
The dancer with only her top changed to the blue and yellow floral top, same terracotta backdrop.

16

Inpaint

Changes only what SAM 3.1 finds in the video.

Prompt

subject_definitions: <Subject 1> is the floral top in <Picture 1>: a fitted sleeveless top in white linen with large painted cobalt-blue and marigold-yellow flowers and green leaves, a scoop neckline and wide straps. summary: [reference generation] A woman with dark hair in a high ponytail and black trousers dances a relaxed groove in a warm studio, wearing <Subject 1>. retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - the white linen, the large cobalt-blue and marigold-yellow flowers, the green leaves, the scoop neckline and the wide straps are retained. detailed_description: The target video is realistic live-action dance footage in warm studio light. [Shot 1] A medium shot of a woman with dark hair in a high ponytail dancing a gentle, relaxed groove in front of a terracotta-orange studio backdrop. She wears <Subject 1>, tucked into her black trousers. The same large cobalt-blue and marigold-yellow flowers stay printed in the same places on the white linen in every frame, and the fabric folds naturally as her torso moves. Her face, hair, bare skin and black trousers stay natural. The camera holds a static shot. overall_soundscape: Quiet studio room tone. non_diegetic_music: A smooth instrumental dance groove.

Download JSON for Inpaint
The woman from the reference picture smiling by a window with white peonies, made with the smaller files in 8 steps.

17

Faster on Low VRAM

Smaller model files, the speed LoRA and faster attention.

Prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with auburn-brown hair pinned up in a loose low bun with soft loose strands, and a cream linen dress with a lace collar. summary: [reference generation] The target video shows <Subject 1> in a softly lit 19th-century drawing room, reading a letter by a tall window and smiling. retention_analysis: <Subject 1> (appears in [Shot 1], [Shot 2]): fully_preserved - her face, hair, cream linen dress and lace collar are retained. detailed_description: The target video is a live-action 19th-century period drama shot on 35mm film with fine grain, soft natural window light and a shallow depth of field. [Shot 1] A medium close-up from the chest up: <Subject 1> stands by a tall window with white curtains in an elegant drawing room with pale green walls, soft overcast daylight on her face. She is reading a letter that she holds still in both hands just below the frame; her eyes move slowly along the lines and a slow, hopeful smile spreads across her face. The camera pushes in with small amplitude at slow speed. [Shot 2] At 00:02.600, the camera cuts to a close-up of <Subject 1> by the window, her face filling the frame, white peonies softly blurred behind her; she lifts her eyes from the letter toward the window, her eyes bright with joy and a slow smile on her lips, and <Subject 1> (S1), with a soft, warm young female voice, says, <d>[English] He's coming home.</d> overall_soundscape: A quiet room: a clock ticking and birdsong outside the window. non_diegetic_music: A gentle solo piano melody with soft strings.

If comfy kitchen attention isn't in the list, choose pytorch attention.

Download JSON for Faster on Low VRAM
The same shot of the woman by the white peonies, made with the GGUF model.

18

Low VRAM: GGUF

The same workflow with a GGUF model.

Prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with auburn-brown hair pinned up in a loose low bun with soft loose strands, and a cream linen dress with a lace collar. summary: [reference generation] The target video shows <Subject 1> in a softly lit 19th-century drawing room, reading a letter by a tall window and smiling. retention_analysis: <Subject 1> (appears in [Shot 1], [Shot 2]): fully_preserved - her face, hair, cream linen dress and lace collar are retained. detailed_description: The target video is a live-action 19th-century period drama shot on 35mm film with fine grain, soft natural window light and a shallow depth of field. [Shot 1] A medium close-up from the chest up: <Subject 1> stands by a tall window with white curtains in an elegant drawing room with pale green walls, soft overcast daylight on her face. She is reading a letter that she holds still in both hands just below the frame; her eyes move slowly along the lines and a slow, hopeful smile spreads across her face. The camera pushes in with small amplitude at slow speed. [Shot 2] At 00:02.600, the camera cuts to a close-up of <Subject 1> by the window, her face filling the frame, white peonies softly blurred behind her; she lifts her eyes from the letter toward the window, her eyes bright with joy and a slow smile on her lips, and <Subject 1> (S1), with a soft, warm young female voice, says, <d>[English] He's coming home.</d> overall_soundscape: A quiet room: a clock ticking and birdsong outside the window. non_diegetic_music: A gentle solo piano melody with soft strings.

If comfy kitchen attention isn't in the list, choose pytorch attention.

Download JSON for Low VRAM: GGUF

Models

Update ComfyUI first, as these use its newest nodes. Then put each file in its folder and press R.

Workflow 13 needs the latent upscaler node: in ComfyUI, click Extensions, search MiniMax H3 latent upscaler, install and restart. Workflow 18 needs ComfyUI GGUF (molbal's fork), installed the same way.

MiniMax H3 prompt guide

On MiniMax's website, a second model turns your idea into a long prompt before H3 sees it. In ComfyUI you write that prompt yourself. Every workflow above already has one that works, so the easiest start is to change it.

Pick a layout

Find your workflow. Its layout is next to it. Copy the labels exactly, colon included.

Layout

integrated_multimodal_description: [Shot 1] Live-action, cinematic, a close-up of ... overall_soundscape: ... non_diegetic_music: ...

Layout

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with ... summary: [reference generation] The target video shows <Subject 1> ... retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - her face, hair and dress are retained. detailed_description: The target video is a live-action ... [Shot 1] A close-up of <Subject 1> ... overall_soundscape: ... non_diegetic_music: ...

Three labels

Workflows 01 to 05. Start with Text to Video: the others build on it.

01

Text to Video

A prompt in, a video with sound out.

See workflow
Important rules
  • Anything you leave out, H3 makes up, so describe everything you want to see and hear.
  • Open each shot with the kind of video and how close the camera is. Faces blur when they're small in the frame, so go closer for a sharp face.
  • The story has to fill the length: 5 seconds fits one or two shots. When our 5 second prompt ran at 15 seconds, the video held still, then slowed down.
  • The length is set in the workflow. Writing “10 seconds” in the prompt doesn't change it.
  • We tried 32 camera moves. Pans, tilts and orbits came out as a push-in, so use First and Last Frame for those.
  • Always fill in the sound. Left empty, H3 makes up its own, sometimes voices.
Tags to use
  • integrated_multimodal_description:
  • overall_soundscape:
  • non_diegetic_music:
  • [Shot 1]
  • (S1)
  • <d>[English] ...</d>
See full example
Workflow 01 prompt

integrated_multimodal_description: [Shot 1] Live-action, cinematic, a fantasy film shot on 35mm film with fine grain and a shallow depth of field: a medium close-up, framed from the chest up, of a young woman kneeling in a stone castle stable at dawn. She is in her early twenties, with long copper hair in a loose braid over one shoulder, freckles, bright green eyes and fine natural skin texture, and she wears a dark green wool cloak over a cream linen dress. In her cupped hands, at chest height, she holds a large, smooth egg covered in small deep blue scales. Warm light from an iron lantern on the wall to her left lights the side of her face, and behind her, pale dawn light falls through a high arched window onto soft, out-of-focus straw and wooden beams, with dust drifting in the light. The egg trembles in her hands, and one fine, dark hairline crack slowly creeps across the shell near its top, while the shell stays whole. Her eyes widen a little and a small smile spreads across her face, and looking down at the egg, the young woman with a soft, breathy voice (S1) whispers quietly, with gentle excitement: <d>[English] It's finally coming out.</d> The camera pushes in with small amplitude at slow speed. [Shot 2] At 00:02.600, the camera cuts to a close-up of the same egg in her hands, filling the frame, the fine hairline crack near its top. The shell is real eggshell, thin, hard and brittle, deep blue scales on the outside and pale cream on the inside. Something pushes from inside: the shell beside the crack bulges outward, and with a small dry snap one irregular, curved piece of shell, about a third as wide as the egg, breaks loose with rough, slightly chipped edges. A baby dragon's head pushes up through the opening, lifting that piece of shell on top of its head, so the broken piece sits right beside the ragged hole it came from, the same shape and size. The piece slides off its head, tumbles down and drops into the straw. The baby dragon, with glossy deep blue scales, two small curled horns and big golden eyes, looks out of the ragged hole with the broken edge of the shell around its neck, blinks slowly up at her and lets out a tiny, high chirp. The camera holds a static shot. overall_soundscape: The shell creaks and gives one small, dry snap as the piece breaks off and drops softly into the straw. Straw rustles under her knees, and birds sing faintly outside the high window. A horse snorts softly in a nearby stall. non_diegetic_music: A soft Celtic harp melody at a slow tempo over a low sustained cello note, rising gently as the dragon appears.

  1. Everything we see, plus speech and the sounds of the action, in the order they happen.
  2. Each shot opens with the kind of video.
  3. Then how close the camera is. Faces blur when they're small in the frame, so go closer for a sharp face.
  4. Be specific. This works, “beautiful lighting” doesn't.
  5. The speaker: a number and what the voice sounds like, outside the tags.
  6. Inside the tags, only the language and the exact words, ending with . ? or !
  7. The camera: the move, how much and how fast.
  8. A new shot starts with its time.
  9. The background sound. Always fill it in: left empty, H3 makes up its own sound, sometimes voices.
  10. The soundtrack. Name the instruments and the speed, or write N/A for none.
02

Image to Video

Starts the video from your picture.

See workflow
Important rules
  • Put the first line on top, then leave one empty line before the labels.
  • Start with what's in the picture, then describe what happens next.
Tags to use
  • integrated_multimodal_description:
  • overall_soundscape:
  • non_diegetic_music:
  • <Picture 1>
  • [Shot 1]
  • (S1)
  • <d>[English] ...</d>
First line, copy it on top

For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced.

See full example
Workflow 02 prompt

For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced. integrated_multimodal_description: [Shot 1] Live-action period drama, cinematic, shot on 35mm with soft grain: the young woman in the cream linen dress with the lace collar from <Picture 1> stands by the tall window in the old manor drawing room, the white peonies soft in the foreground, her eyes already wet with tears. She draws a slow, shaky breath, a tear slowly rolls further down her cheek, and the young woman with a soft, trembling voice (S1) says, <d>[English] Promise me you'll come back.</d> Then she lowers her eyes. Soft window light, very shallow depth of field. The camera pushes in with small amplitude at slow speed. overall_soundscape: A hushed old room, the soft patter of rain on the window glass. non_diegetic_music: A tender, aching piano and cello theme.

  1. The first line. Copy it as it is.
  2. Point at what's in the picture, and keep using the same words for her.
  3. Then the action.
03

First and Last Frame

You give the first and last picture, the model makes what happens between them.

See workflow
Important rules
  • Change the second time to your video's length in seconds: frames ÷ 24, so 124 frames is 5.17.
  • Keep it to one shot and describe the change between the two pictures, not the pictures themselves.
  • If something should move, like leaves or water, say how, or it stays frozen.
  • For a real pan, tilt or zoom out, use two crops of one bigger picture. That worked 8 times out of 8.
Tags to use
  • integrated_multimodal_description:
  • overall_soundscape:
  • non_diegetic_music:
  • Picture 1
  • Picture 2
  • [Shot 1]
First line, copy it on top

How the reference pictures align with the target video - Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot 1) aligns with the 5.17-second mark of the target video.

See full example
Workflow 03 prompt

How the reference pictures align with the target video - Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot 1) aligns with the 5.17-second mark of the target video. integrated_multimodal_description: [Shot 1] Live-action time-lapse, cinematic: the whitewashed cliffside village above the sea, as in Picture 1. The camera holds a static shot as time passes quickly: the warm golden light slides off the white walls, the sky deepens to blue and then to a clear night, stars appear one by one and the Milky Way rises over the sea, and warm lights switch on in the windows and along the stairway, all mirrored in the calm water below, pose and composition established by Picture 2 at the end of the shot. overall_soundscape: A quiet seaside evening: soft waves, a light breeze and distant church bells fading into crickets. non_diegetic_music: A slow, warm ambient piano with soft strings that swell as the stars appear.

  1. The end time.
  2. The change between the two pictures, in the order it happens.
  3. Where it ends up.
04

Seamless Loop

The same picture as first and last frame, so the video loops.

See workflow
Important rules
  • Use the same picture at both ends, and set the second time to your video's length.
  • Keep the camera and the light still, or the ends won't match.
  • Say what keeps moving, and that it moves evenly from start to end.
Tags to use
  • integrated_multimodal_description:
  • overall_soundscape:
  • non_diegetic_music:
  • Picture 1
  • Picture 2
  • [Shot 1]
First line, copy it on top

How the reference pictures align with the target video - Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot 1) aligns with the 5.17-second mark of the target video.

See full example
Workflow 04 prompt

How the reference pictures align with the target video - Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot 1) aligns with the 5.17-second mark of the target video. integrated_multimodal_description: [Shot 1] Live-action, cinematic, warm golden autumn light: the vintage convertible parked under the great maple tree from Picture 1. The camera holds a perfectly static shot on a tripod and the light stays exactly the same the whole time. Golden, orange and red maple leaves keep drifting slowly down from the canopy and tumble gently through the warm backlight, a soft breeze stirs the branches and the leaves on the cobblestones, and fine dust glows in the sunbeams. The leaves keep falling gently and evenly from the first frame to the last; the car stays perfectly still. This is one seamless loop: the shot ends on exactly the same frame it starts with, Picture 2. overall_soundscape: N/A. non_diegetic_music: N/A.

  1. Camera and light stay still.
  2. What keeps moving.
  3. Say that it loops.
05

Last Frame

The video builds up to your picture.

See workflow
Important rules
  • Set the time to your video's length, so the picture lands on the last frame.
  • Describe a believable lead-up, then end on the picture.
Tags to use
  • integrated_multimodal_description:
  • overall_soundscape:
  • non_diegetic_music:
  • Picture 1
  • [Shot 1]
First line, copy it on top

How the reference pictures align with the target video - Picture 1 (from Shot 1) aligns with the 5.17-second mark of the target video.

See full example
Workflow 05 prompt

How the reference pictures align with the target video - Picture 1 (from Shot 1) aligns with the 5.17-second mark of the target video. integrated_multimodal_description: [Shot 1] Live-action luxury perfume commercial, intimate and calm, soft warm morning window light from the left with gently swaying dappled leaf shadows. The shot opens on the glossy cream ceramic tray on the blush-pink tabletop with the loose pink peony petals, the strand of pearls, the open gold compact and the folded linen napkin, and an empty space on the tray. A young woman's slender, graceful hand with soft natural nails and a delicate thin gold ring enters from the top right holding the rounded square pink glass perfume bottle with the large pearly white sphere cap and the cream MODEL CHIEF label from Picture 1 by its body, lowers it slowly and sets it down gently on the tray in the empty space, turns it slightly so the label faces the camera, then lets go and glides smoothly back out of the frame to the top right. The camera holds steady with a very slow push in. The last moments are calm and still, with no hand in the frame, and the shot ends exactly on Picture 1. overall_soundscape: A soft clink of glass on ceramic, the faint rustle of linen, birdsong outside the window. non_diegetic_music: An elegant, minimal piano melody with soft, warm strings.

  1. Start from how things look before your picture.
  2. End on the picture.

Six labels

Workflows 06 to 18. Start with Reference to Video: the others build on it.

06

Reference to Video

Puts the person from your picture in a new scene.

See workflow
Important rules
  • Give each person a label and a description, then use the label every time they appear.
  • In retention_analysis, say what to keep from each reference and in which shots.
  • The scene, the camera, speech and sound work the same as in Text to Video.
Tags to use
  • [reference generation]at the start of the summary
  • <Subject 1>
  • <Picture 1>
  • fully_preserved
  • [Shot 1]
  • (S1)
  • <d>[English] ...</d>
See full example
Workflow 06 prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with auburn-brown hair pinned up in a loose low bun with soft loose strands, and a cream linen dress with a lace collar. summary: [reference generation] The target video shows <Subject 1> in a softly lit 19th-century drawing room, reading a letter by a tall window and smiling. retention_analysis: <Subject 1> (appears in [Shot 1], [Shot 2]): fully_preserved - her face, hair, cream linen dress and lace collar are retained. detailed_description: The target video is a live-action 19th-century period drama shot on 35mm film with fine grain, soft natural window light and a shallow depth of field. [Shot 1] A medium close-up from the chest up: <Subject 1> stands by a tall window with white curtains in an elegant drawing room with pale green walls, soft overcast daylight on her face. She is reading a letter that she holds still in both hands just below the frame; her eyes move slowly along the lines and a slow, hopeful smile spreads across her face. The camera pushes in with small amplitude at slow speed. [Shot 2] At 00:02.600, the camera cuts to a close-up of <Subject 1> by the window, her face filling the frame, white peonies softly blurred behind her; she lifts her eyes from the letter toward the window, her eyes bright with joy and a slow smile on her lips, and <Subject 1> (S1), with a soft, warm young female voice, says, <d>[English] He's coming home.</d> overall_soundscape: A quiet room: a clock ticking and birdsong outside the window. non_diegetic_music: A gentle solo piano melody with soft strings.

  1. Who is who: a label for each person or thing, then what they look like.
  2. The tag that names the job. It goes at the start of the summary.
  3. What to keep from each reference, and in which shots.
  4. The style, in one sentence before [Shot 1].
  5. Someone who speaks keeps the label and also gets a speaker number.

What the words mean

LabelWhat it means
<Subject 1>A person, thing or place to reuse from your files.
<Picture 1>A picture. Pictures are numbered in input order: ref_image_0 is Picture 1, ref_image_1 is Picture 2.
<Video 1>A video you give it, like the one to edit.
<Audio 1>An audio file, like a voice.
In retention_analysisMeans
fully_preservedKept as it is.
partially_preservedKept, but with a change, like a new colour.
referenceFor audio: use the voice, not the recording.
07

Reference to Video: Two Characters

Two people from two pictures in one scene.

See workflow
Important rules
  • One label per person, in input order: Picture 1 is Subject 1, Picture 2 is Subject 2.
  • Each speaker gets their own number and keeps it in every shot.
Tags to use
  • [reference generation]at the start of the summary
  • <Subject 1>
  • <Subject 2>
  • <Picture 1>
  • <Picture 2>
  • fully_preserved
  • (S1)
  • (S2)
See full example
Workflow 07 prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with auburn-brown hair pinned up in a loose low bun with soft loose strands, and a cream linen dress with a lace collar. <Subject 2> is the man in <Picture 2>, with dark wavy hair, light stubble and brown eyes, in a dark frock coat over a cream waistcoat and a white linen cravat. summary: [reference generation] The target video shows <Subject 1> and <Subject 2> meeting again in a softly lit 19th-century drawing room. retention_analysis: <Subject 1> (appears in [Shot 1], [Shot 3]): fully_preserved - her face, hair, cream linen dress and lace collar are retained. <Subject 2> (appears in [Shot 2]): fully_preserved - his face, dark wavy hair, stubble, frock coat, waistcoat and cravat are retained. detailed_description: The target video is a live-action 19th-century period drama shot on 35mm film with fine grain, soft natural window light and a shallow depth of field. [Shot 1] A medium close-up, framed from the chest up: <Subject 1> stands by a tall window in an elegant drawing room with pale green walls, close to the camera, her face almost toward the camera, turned slightly to the right, toward a door off-screen. Soft, even daylight from the window on the left of the frame lights her whole face, which is clear and in sharp focus. Her hands are empty and out of frame. From the very first frame her eyes are fully open, clear and bright, looking toward the door, and her head stays still. At the click of the door latch off-screen, her eyes lift a little and her lips part in quiet, hopeful surprise. The camera holds a static shot. [Shot 2] At 00:02.400, the camera cuts to a medium close-up of <Subject 2> in the doorway, his face in soft window light; he looks at her, smiles gently, and <Subject 2> (S1), with a low, warm male voice, says clearly and unhurriedly, <d>[English] I told you I'd come back.</d> [Shot 3] At 00:05.200, the camera cuts to a close-up of <Subject 1>; a slow, radiant smile spreads across her face, and <Subject 1> (S2), with a soft, warm young female voice, says, <d>[English] You're late.</d> She lets out a small, happy laugh. overall_soundscape: The door latch clicks, footsteps on wooden floorboards, a clock ticking and birdsong outside the window. non_diegetic_music: A gentle solo piano melody with soft strings.

  1. The second person gets the next label.
  2. Each person has their own line here.
  3. This one speaks first.
  4. This one speaks second.
08

Custom Voice

Your character speaks in the voice from your audio file.

See workflow
Important rules
  • Type the audio file's exact words inside the tags. With new words, the voice only sounded similar in our tests.
  • Set the length just long enough for the line.
Tags to use
  • [reference generation + audio reference]at the start of the summary
  • <Subject 1>
  • <Picture 1>
  • <Audio 1>
  • fully_preserved
  • reference
  • (S1)
See full example
Workflow 08 prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with auburn-brown hair pinned up in a loose low bun with soft loose strands, and a cream linen dress with a lace collar. <Audio 1> is the voice-timbre reference for <Subject 1> (S1). summary: [reference generation + audio reference] The target video shows <Subject 1> in a softly lit 19th-century drawing room, holding a bundle of old letters and quietly telling someone she kept every one of them. retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - her face, hair, cream linen dress and lace collar are retained. <Audio 1>: reference - its voice timbre guides the speech of <Subject 1> without copying the original signal. detailed_description: The target video is a live-action 19th-century period drama shot on 35mm film with fine grain, soft natural window light and a shallow depth of field. [Shot 1] A close-up framed from the shoulders up: <Subject 1> stands by a tall window with white curtains in an elegant drawing room with pale green walls, white peonies softly blurred behind her. Warm, soft afternoon daylight from the window lights her whole face, which is clear and in sharp focus. Her hands rest below the frame, holding a bundle of old letters tied with a faded blue ribbon. From the first frame she looks toward someone just off-camera, her eyes clear and bright, with a calm, warm and quietly happy expression, and her head stays still. Almost immediately, <Subject 1> (S1), in the voice referenced from <Audio 1>, says in a warm, gentle and sincere tone at a natural, unhurried pace, with a soft smile in her voice and a short pause at the comma, <d>[English] I kept every letter, every single one.</d> Her lips move in clear sync with every word. As she finishes, a fond, slightly shy smile spreads across her face and she glances down at the letters for a moment, then back up. The camera pushes in with small amplitude at slow speed. overall_soundscape: A quiet room: a clock ticking softly and distant birdsong outside the window; her voice is clear and close. non_diegetic_music: A gentle solo piano melody with soft strings, kept low under her voice.

  1. Say whose voice the file is.
  2. The tag for voices.
  3. Use the voice, not the recording.
  4. Point at the file when they speak.
  5. The file's exact words.
09

Custom Voices: Two Characters

Each character gets their own voice file.

See workflow
Important rules
  • One file per character: Picture 1, Subject 1, Audio 1 and S1 are the same person, and Picture 2, Subject 2, Audio 2 and S2 the next.
  • Type each file's exact words inside the tags.
Tags to use
  • [reference generation + audio reference]at the start of the summary
  • <Subject 1>
  • <Subject 2>
  • <Picture 1>
  • <Picture 2>
  • <Audio 1>
  • <Audio 2>
  • fully_preserved
  • reference
  • (S1)
  • (S2)
See full example
Workflow 09 prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with auburn-brown hair pinned up in a loose low bun with soft loose strands, and a cream linen dress with a lace collar. <Subject 2> is the man in <Picture 2>, with dark wavy hair, light stubble and brown eyes, in a dark frock coat over a cream waistcoat and a white linen cravat. <Audio 1> is the voice-timbre reference for <Subject 1> (S1). <Audio 2> is the voice-timbre reference for <Subject 2> (S2). summary: [reference generation + audio reference] The target video shows <Subject 1> and <Subject 2> reunited in a softly lit 19th-century drawing room: she tells him she kept every letter he wrote, and he answers that he read every one of hers many times. retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - her face, hair, cream linen dress and lace collar are retained. <Subject 2> (appears in [Shot 2]): fully_preserved - his face, dark wavy hair, stubble, frock coat, waistcoat and cravat are retained. <Audio 1>: reference - its voice timbre guides the speech of <Subject 1> without copying the original signal. <Audio 2>: reference - its voice timbre guides the speech of <Subject 2> without copying the original signal. detailed_description: The target video is a live-action 19th-century period drama shot on 35mm film with fine grain, soft natural window light and a shallow depth of field. [Shot 1] A close-up framed from the shoulders up: <Subject 1> stands by a tall window in an elegant drawing room with pale green walls, her face almost toward the camera, turned slightly to the right toward <Subject 2> just off-screen. Warm, soft afternoon daylight from the window lights her whole face, which is clear and in sharp focus. Her hands rest below the frame, holding a bundle of old letters tied with a faded blue ribbon. Her eyes are clear and bright, her expression calm, warm and quietly happy, and her head stays still. Almost immediately, <Subject 1> (S1), in the voice referenced from <Audio 1>, says in a warm, gentle and sincere tone at a natural, unhurried pace, with a soft smile in her voice and a short pause at the comma, <d>[English] I kept every letter, every single one.</d> Her lips move in clear sync with every word. The camera holds a static shot. [Shot 2] At 00:03.667, the camera cuts to a close-up of <Subject 2>, framed from the shoulders up, facing her and turned slightly to the left, his face in the same warm window light and in sharp focus. A slow, delighted smile spreads across his face, and right away <Subject 2> (S2), in the voice referenced from <Audio 2>, answers, using the low, smooth and relaxed voice timbre referenced from <Audio 2>, in a warm and tender tone at a natural, unhurried pace, with a short pause at the comma, <d>[English] And I read every one of yours, multiple times.</d> His lips move in clear sync with every word, and he holds her gaze with a warm smile as he finishes. The camera holds a static shot. overall_soundscape: A quiet room: a clock ticking softly and birdsong outside the window; both voices are clear and close. non_diegetic_music: A gentle solo piano melody with soft strings, kept low under their voices.

  1. Each voice has its own line.
  2. Point at the right file when each one speaks.
10

Multi-Shot Keyframes

Four pictures, each one starts a new shot.

See workflow
Important rules
  • Each shot starts where its picture sits: frame_idx ÷ 24, so frame 53 is 00:02.208.
  • Say which picture starts the video, and describe every shot.
Tags to use
  • [keyframe completion + reference generation]at the start of the summary
  • <Subject 1>
  • <Picture 1>
  • fully_preserved
  • [Shot 1]
See full example
Workflow 10 prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with sleek dark brown hair in a neat low bun and small gold hoop earrings, wearing an oversized black satin souvenir bomber jacket with black-and-gold striped collar and cuffs over a plain white t-shirt, a large embroidered golden crane, red rising sun and indigo waves on its back. <Picture 1> is the first frame of [Shot 1], showing <Subject 1> from the waist up, facing the camera against a sunlit terracotta wall with soft window shadows. summary: [keyframe completion + reference generation] The target video is a short fashion campaign film for the jacket in four shots: it begins with <Picture 1>, cuts to a close-up of the embroidery, then to <Subject 1> from behind, and ends on a close-up of her face. retention_analysis: <Subject 1> (appears in [Shot 1], [Shot 3], [Shot 4]): fully_preserved - her face, hair, earrings and the jacket stay the same. <Picture 1> ([Shot 1] first frame): fully_preserved - the video starts exactly from <Picture 1>. detailed_description: The target video is a fashion campaign film in a cinematic live-action style, warm late-afternoon sunlight, rich film-like colours. [Shot 1] The shot begins from <Picture 1>. The camera pushes in very slowly. From the first frame <Subject 1> lifts her chin and looks into the lens with calm confidence, then begins to turn away, the black satin shimmering. [Shot 2] At 00:02.208, the shot cuts to an extreme close-up of the back of the jacket: the large embroidered golden crane with spread wings across the big red rising sun, shiny gold, cream and red silk threads on black satin; the camera glides slowly sideways across the stitching as the warm light slides over the threads. [Shot 3] At 00:04.417, the shot cuts to a waist-up shot of <Subject 1> from behind, the whole embroidered crane, red sun and indigo waves on her back; she turns her head to glance back over her shoulder at the camera. [Shot 4] At 00:06.625, the shot cuts to a close-up of <Subject 1>'s face from the shoulders up, facing the camera, the black-and-gold striped collar and her gold hoop earrings in the warm sunlight; the camera pushes in slowly as she lifts her chin and a slow, confident smile spreads across her face. No captions or subtitles appear. overall_soundscape: The soft rustle of satin in a quiet room. non_diegetic_music: A minimal, stylish electronic beat with a warm bassline that runs through all four shots.

  1. Say what the picture is for.
  2. Start exactly from it.
  3. Then describe the motion from there.
  4. Each new shot starts at its frame's time.
11

Speed LoRA

The same four shots in 8 steps instead of 20.

See workflow
Important rules
  • The prompt is the same as Multi-Shot Keyframes. Only the speed LoRA and the 8 steps change, and those are already set in the workflow.
Tags to use
  • [keyframe completion + reference generation]at the start of the summary
  • <Subject 1>
  • <Picture 1>
  • fully_preserved
  • [Shot 1]
See full example
Workflow 11 prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with sleek dark brown hair in a neat low bun and small gold hoop earrings, wearing an oversized black satin souvenir bomber jacket with black-and-gold striped collar and cuffs over a plain white t-shirt, a large embroidered golden crane, red rising sun and indigo waves on its back. <Picture 1> is the first frame of [Shot 1], showing <Subject 1> from the waist up, facing the camera against a sunlit terracotta wall with soft window shadows. summary: [keyframe completion + reference generation] The target video is a short fashion campaign film for the jacket in four shots: it begins with <Picture 1>, cuts to a close-up of the embroidery, then to <Subject 1> from behind, and ends on a close-up of her face. retention_analysis: <Subject 1> (appears in [Shot 1], [Shot 3], [Shot 4]): fully_preserved - her face, hair, earrings and the jacket stay the same. <Picture 1> ([Shot 1] first frame): fully_preserved - the video starts exactly from <Picture 1>. detailed_description: The target video is a fashion campaign film in a cinematic live-action style, warm late-afternoon sunlight, rich film-like colours. [Shot 1] The shot begins from <Picture 1>. The camera pushes in very slowly. From the first frame <Subject 1> lifts her chin and looks into the lens with calm confidence, then begins to turn away, the black satin shimmering. [Shot 2] At 00:02.208, the shot cuts to an extreme close-up of the back of the jacket: the large embroidered golden crane with spread wings across the big red rising sun, shiny gold, cream and red silk threads on black satin; the camera glides slowly sideways across the stitching as the warm light slides over the threads. [Shot 3] At 00:04.417, the shot cuts to a waist-up shot of <Subject 1> from behind, the whole embroidered crane, red sun and indigo waves on her back; she turns her head to glance back over her shoulder at the camera. [Shot 4] At 00:06.625, the shot cuts to a close-up of <Subject 1>'s face from the shoulders up, facing the camera, the black-and-gold striped collar and her gold hoop earrings in the warm sunlight; the camera pushes in slowly as she lifts her chin and a slow, confident smile spreads across her face. No captions or subtitles appear. overall_soundscape: The soft rustle of satin in a quiet room. non_diegetic_music: A minimal, stylish electronic beat with a warm bassline that runs through all four shots.

  1. Say what the picture is for.
  2. Start exactly from it.
  3. Then describe the motion from there.
  4. Each new shot starts at its frame's time.
12

Motion Transfer

The person in your picture does the moves from a video.

See workflow
Important rules
  • Describe the moves from the dance video in order, so the prompt and the pose video agree.
  • Say the camera and background stay still, and that the person doesn't speak or sing.
Tags to use
  • [reference generation]at the start of the summary
  • <Subject 1>
  • <Picture 1>
  • fully_preserved
  • [Shot 1]
See full example
Workflow 12 prompt

subject_definitions: <Subject 1> is the woman in <Picture 1>, with a short black bob, an open faded light-wash denim jacket over a plain white t-shirt and black trousers. She has a round face with soft features, dark brown eyes, faint freckles across her nose and a small closed-mouth smile. summary: [reference generation] The target video shows <Subject 1> dancing a relaxed, laid-back groove on a narrow sunny side street outside a small corner cafe, with a chalkboard menu in the cafe window, a silver bicycle with a front basket parked against the wall and power lines overhead, following the guide movement beat by beat. retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - her face, hairstyle, eye colour, skin, clothes and the place from <Picture 1> are retained exactly; only her body movement is new. detailed_description: The target video is real live-action footage filmed on a locked-off camera, with bright low morning sun from the front-left with crisp shadows on the road, the same light as in <Picture 1>, natural skin texture with visible pores, individual hair strands and real fabric folds; it looks like a real dance video, not CGI. [Shot 1] A static medium shot frames <Subject 1> from the waist up in the centre of the frame, facing the camera, on a narrow sunny side street outside a small corner cafe, with a chalkboard menu in the cafe window, a silver bicycle with a front basket parked against the wall and power lines overhead. She sways gently from side to side with a small bounce in her knees, her shoulders rising and falling softly with the beat, rolls her hands loosely in front of her waist, softly snaps her fingers at chest height with a small, easy shoulder shimmy, sways her hips, nods to the beat with a calm smile and rests both hands back on her hips; every move is small, smooth and relaxed. Her body follows the guide movement exactly, with the same timing and speed. Her eyes stay clearly visible, her hands are bare, and her face stays clear, sharp, natural and the same in every frame. She does not speak or sing. The camera does not move and the background stays still. No captions or subtitles appear. overall_soundscape: Quiet street sounds and a bicycle bell far away. non_diegetic_music: A smooth, mid-tempo instrumental R&B groove with a soft kick, light finger snaps and a warm bass line, no vocals.

  1. Keep everything from the picture except the movement.
  2. The moves, in order.
  3. Tie it to the dance video.
  4. So no words get made up.
  5. Only she moves.
13

Upscale

Motion Transfer plus a second pass, up to 1920 x 1088.

See workflow
Important rules
  • The prompt is the same as Motion Transfer. The second pass doesn't change how you write it.
Tags to use
  • [reference generation]at the start of the summary
  • <Subject 1>
  • <Picture 1>
  • fully_preserved
  • [Shot 1]
See full example
Workflow 13 prompt

subject_definitions: <Subject 1> is the woman in <Picture 1>, with a short black bob, an open faded light-wash denim jacket over a plain white t-shirt and black trousers. She has a round face with soft features, dark brown eyes, faint freckles across her nose and a small closed-mouth smile. summary: [reference generation] The target video shows <Subject 1> dancing a relaxed, laid-back groove on a narrow sunny side street outside a small corner cafe, with a chalkboard menu in the cafe window, a silver bicycle with a front basket parked against the wall and power lines overhead, following the guide movement beat by beat. retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - her face, hairstyle, eye colour, skin, clothes and the place from <Picture 1> are retained exactly; only her body movement is new. detailed_description: The target video is real live-action footage filmed on a locked-off camera, with bright low morning sun from the front-left with crisp shadows on the road, the same light as in <Picture 1>, natural skin texture with visible pores, individual hair strands and real fabric folds; it looks like a real dance video, not CGI. [Shot 1] A static medium shot frames <Subject 1> from the waist up in the centre of the frame, facing the camera, on a narrow sunny side street outside a small corner cafe, with a chalkboard menu in the cafe window, a silver bicycle with a front basket parked against the wall and power lines overhead. She sways gently from side to side with a small bounce in her knees, her shoulders rising and falling softly with the beat, rolls her hands loosely in front of her waist, softly snaps her fingers at chest height with a small, easy shoulder shimmy, sways her hips, nods to the beat with a calm smile and rests both hands back on her hips; every move is small, smooth and relaxed. Her body follows the guide movement exactly, with the same timing and speed. Her eyes stay clearly visible, her hands are bare, and her face stays clear, sharp, natural and the same in every frame. She does not speak or sing. The camera does not move and the background stays still. No captions or subtitles appear. overall_soundscape: Quiet street sounds and a bicycle bell far away. non_diegetic_music: A smooth, mid-tempo instrumental R&B groove with a soft kick, light finger snaps and a warm bass line, no vocals.

  1. Keep everything from the picture except the movement.
  2. The moves, in order.
  3. Tie it to the dance video.
  4. So no words get made up.
  5. Only she moves.
14

Edit Videos

Changes one thing in a video from a written instruction.

See workflow
Important rules
  • Change one clear thing, like a colour or a piece of clothing, and say what stays exactly the same.
  • A whole new style, like claymation, came back as a copy of the original in all six of our tries.
Tags to use
  • [video editing]at the start of the summary
  • <Subject 1>
  • <Subject 2>
  • <Video 1>
  • partially_preserved
  • fully_preserved
See full example
Workflow 14 prompt

subject_definitions: <Subject 1> is the young woman dancing in <Video 1>, with dark hair in a high ponytail, a white ribbed tank top and black high-waisted trousers. <Subject 2> is the photo-studio environment in <Video 1>: a plain, seamless terracotta-orange paper backdrop lit by soft warm studio light with a gentle glow in the centre, and black light stands at the right edge of the frame. summary: [video editing] The target video is an edited version of <Video 1>. The colour of <Subject 1>'s tank top changes from white to a deep cobalt blue in every frame; her dance, her face, her trousers, <Subject 2>, the light and the camera stay exactly the same. retention_analysis: <Subject 1> (appears in [Shot 1]): partially_preserved - her face, hair, body, trousers and every dance move are retained; the tank top is now deep cobalt blue with the same fine ribs and fit. <Subject 2> (appears in [Shot 1]): fully_preserved - the terracotta-orange backdrop, the warm light and the light stands are retained. <Video 1> (framing and timing): fully_preserved - the locked-off framing and the timing are retained. detailed_description: The target video is real live-action dance footage filmed on a locked-off camera with soft warm studio light. [Shot 1] <Subject 1> dances in front of <Subject 2> exactly as in <Video 1>, in a fitted, finely ribbed deep cobalt-blue cotton tank top and black high-waisted trousers. The whole tank top is the same rich cobalt blue from the straps to the hem in every frame, with real fabric folds that stretch with her body; no white is left on it. The camera does not move. No captions or subtitles appear. overall_soundscape: The soft shuffle of her feet on the studio floor. non_diegetic_music: A smooth, mid-tempo instrumental R&B groove with a soft kick, light finger snaps and a warm bass line, no vocals.

  1. The people and things come from the video.
  2. The tag for edits.
  3. The one change.
  4. Everything that stays.
  5. Kept, but with that one change.
15

Outpaint

Turns a vertical video into a wide one.

See workflow
Important rules
  • Describe the whole wide frame, including what fills the new parts on the left and the right.
Tags to use
  • [reference generation]at the start of the summary
  • <Subject 1>
  • <Subject 2>
  • <Subject 3>
  • <Picture 1>
  • fully_preserved
See full example
Workflow 15 prompt

subject_definitions: <Subject 1> is the woman in <Picture 1>, in a cream silk headscarf, black cat-eye sunglasses, a black wool coat and cream leather gloves. <Subject 2> is the long glossy black vintage convertible with tan leather seats in <Picture 1>. <Subject 3> is the English country estate in autumn in <Picture 1>: a gravel drive, tall old oak trees and falling golden leaves. summary: [reference generation] A wide 16:9 shot of <Subject 1> sitting behind the wheel of <Subject 2>, parked on the gravel drive of <Subject 3>, with the whole car and the estate in view. retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - her headscarf, sunglasses, coat, gloves and her movement are retained. <Subject 2> (appears in [Shot 1]): fully_preserved - the glossy black body, the chrome trim and the tan leather seats are retained, with the rest of the car now in view. <Subject 3> (appears in [Shot 1]): fully_preserved - the gravel drive, the oak trees and the falling leaves are retained and continue to both sides. detailed_description: The target video is a live-action 1960s fashion film, shot on 35mm with soft grain. [Shot 1] A wide 16:9 frame: <Subject 1> sits behind the wheel of <Subject 2>, parked on the gravel drive of <Subject 3>. The whole car is in view: on the left its long black bonnet, chrome grille and round chrome headlamp; on the right the chrome-trimmed door, the rear wing and the sloping boot with a small chrome tail-lamp. Behind the car the gravel drive leads between tall old oak trees to a pale stone Georgian manor house; on the left a wide lawn covered in orange leaves under the oaks, on the right a clipped hedge and a stone urn at the edge of the drive. Nobody else is in the scene. She slowly turns her head toward the camera and, with one gloved hand, lowers her sunglasses a little to look over them, a faint confident smile. Golden autumn leaves drift down across the whole frame and a gust lifts the loose end of her silk scarf. The camera pushes in with small amplitude at slow speed. overall_soundscape: Dry autumn leaves skittering over gravel, a soft breeze and the low idle of a vintage car engine. non_diegetic_music: A slow, moody retro soul groove with brushed drums and a muted trumpet.

  1. Say it's the wide frame.
  2. What fills the new part on the left.
  3. And on the right.
  4. So nobody new appears in the new parts.
16

Inpaint

Changes only what SAM 3.1 finds in the video.

See workflow
Important rules
  • The new thing is Subject 1, from your picture. Describe the person around it too, so it fits in.
Tags to use
  • [reference generation]at the start of the summary
  • <Subject 1>
  • <Picture 1>
  • fully_preserved
See full example
Workflow 16 prompt

subject_definitions: <Subject 1> is the floral top in <Picture 1>: a fitted sleeveless top in white linen with large painted cobalt-blue and marigold-yellow flowers and green leaves, a scoop neckline and wide straps. summary: [reference generation] A woman with dark hair in a high ponytail and black trousers dances a relaxed groove in a warm studio, wearing <Subject 1>. retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - the white linen, the large cobalt-blue and marigold-yellow flowers, the green leaves, the scoop neckline and the wide straps are retained. detailed_description: The target video is realistic live-action dance footage in warm studio light. [Shot 1] A medium shot of a woman with dark hair in a high ponytail dancing a gentle, relaxed groove in front of a terracotta-orange studio backdrop. She wears <Subject 1>, tucked into her black trousers. The same large cobalt-blue and marigold-yellow flowers stay printed in the same places on the white linen in every frame, and the fabric folds naturally as her torso moves. Her face, hair, bare skin and black trousers stay natural. The camera holds a static shot. overall_soundscape: Quiet studio room tone. non_diegetic_music: A smooth instrumental dance groove.

  1. The new thing, from your picture.
  2. How it sits on the person.
  3. What stays as it is.
17

Faster on Low VRAM

Smaller model files, the speed LoRA and faster attention.

See workflow
Important rules
  • The prompt is the same as Reference to Video. Only the smaller files and the speed LoRA change, and those are already set in the workflow.
Tags to use
  • [reference generation]at the start of the summary
  • <Subject 1>
  • <Picture 1>
  • fully_preserved
  • [Shot 1]
  • (S1)
See full example
Workflow 17 prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with auburn-brown hair pinned up in a loose low bun with soft loose strands, and a cream linen dress with a lace collar. summary: [reference generation] The target video shows <Subject 1> in a softly lit 19th-century drawing room, reading a letter by a tall window and smiling. retention_analysis: <Subject 1> (appears in [Shot 1], [Shot 2]): fully_preserved - her face, hair, cream linen dress and lace collar are retained. detailed_description: The target video is a live-action 19th-century period drama shot on 35mm film with fine grain, soft natural window light and a shallow depth of field. [Shot 1] A medium close-up from the chest up: <Subject 1> stands by a tall window with white curtains in an elegant drawing room with pale green walls, soft overcast daylight on her face. She is reading a letter that she holds still in both hands just below the frame; her eyes move slowly along the lines and a slow, hopeful smile spreads across her face. The camera pushes in with small amplitude at slow speed. [Shot 2] At 00:02.600, the camera cuts to a close-up of <Subject 1> by the window, her face filling the frame, white peonies softly blurred behind her; she lifts her eyes from the letter toward the window, her eyes bright with joy and a slow smile on her lips, and <Subject 1> (S1), with a soft, warm young female voice, says, <d>[English] He's coming home.</d> overall_soundscape: A quiet room: a clock ticking and birdsong outside the window. non_diegetic_music: A gentle solo piano melody with soft strings.

  1. Who is who: a label for each person or thing, then what they look like.
  2. The tag that names the job. It goes at the start of the summary.
  3. What to keep from each reference, and in which shots.
  4. The style, in one sentence before [Shot 1].
  5. Someone who speaks keeps the label and also gets a speaker number.
18

Low VRAM: GGUF

The same workflow with a GGUF model.

See workflow
Important rules
  • The prompt is the same as Reference to Video. Only the model file changes, and it's already set in the workflow.
Tags to use
  • [reference generation]at the start of the summary
  • <Subject 1>
  • <Picture 1>
  • fully_preserved
  • [Shot 1]
  • (S1)
See full example
Workflow 18 prompt

subject_definitions: <Subject 1> is the young woman in <Picture 1>, with auburn-brown hair pinned up in a loose low bun with soft loose strands, and a cream linen dress with a lace collar. summary: [reference generation] The target video shows <Subject 1> in a softly lit 19th-century drawing room, reading a letter by a tall window and smiling. retention_analysis: <Subject 1> (appears in [Shot 1], [Shot 2]): fully_preserved - her face, hair, cream linen dress and lace collar are retained. detailed_description: The target video is a live-action 19th-century period drama shot on 35mm film with fine grain, soft natural window light and a shallow depth of field. [Shot 1] A medium close-up from the chest up: <Subject 1> stands by a tall window with white curtains in an elegant drawing room with pale green walls, soft overcast daylight on her face. She is reading a letter that she holds still in both hands just below the frame; her eyes move slowly along the lines and a slow, hopeful smile spreads across her face. The camera pushes in with small amplitude at slow speed. [Shot 2] At 00:02.600, the camera cuts to a close-up of <Subject 1> by the window, her face filling the frame, white peonies softly blurred behind her; she lifts her eyes from the letter toward the window, her eyes bright with joy and a slow smile on her lips, and <Subject 1> (S1), with a soft, warm young female voice, says, <d>[English] He's coming home.</d> overall_soundscape: A quiet room: a clock ticking and birdsong outside the window. non_diegetic_music: A gentle solo piano melody with soft strings.

  1. Who is who: a label for each person or thing, then what they look like.
  2. The tag that names the job. It goes at the start of the summary.
  3. What to keep from each reference, and in which shots.
  4. The style, in one sentence before [Shot 1].
  5. Someone who speaks keeps the label and also gets a speaker number.

What doesn't work in ComfyUI

Don'tWhyInstead
A negative prompt, or a list like “no text, no blur”H3 runs at CFG 1, so the negative is never used, and a list in the prompt adds the very things you banned.Describe what you want. One plain fact is fine: No captions or subtitles appear.

From MiniMax's prompt-writing guides for the three-label layout and the six-label layout, ComfyUI's H3 prompt notes, and our own test runs for this video.