What is ControlNet? How to control the shape of an AI image

On this page9 sections

Try getting a pose out of a prompt. "One hand raised to her hair, the other resting on her hip" gets you something close, then something else close, then a third variation, and none of them is the pose you had in mind. Words are good at what and who. They're bad at exactly where.

ControlNet fixes that. In the video I put it this way: a LoRA influences how the model generates, a ControlNet influences the structure it should follow. This guide shows what that means on real renders, which kind of control image to use for what, and how it's wired and tuned in ComfyUI.

What ControlNet is

ControlNet comes from a 2023 paper, Adding Conditional Control to Text-to-Image Diffusion Models, by Lvmin Zhang, Anyi Rao and Maneesh Agrawala at Stanford. It won the Marr Prize, the best paper award, at ICCV 2023, and it's the reason "follow this pose" became something an image model can actually do.

The idea is neat. The original model is locked, so nothing it knows can be damaged. Alongside it sits a trainable copy of the part of the model that reads an image, and that copy learns to read a control image, such as an edge map. The copy is connected to the model through layers that start at zero. On the first day of training they add nothing at all, so the model behaves exactly as it did before, and the guidance grows in from there as training goes on.

What you download is that extra network, as a file of its own. It doesn't change your model the way a LoRA does. It adds guidance while the model samples, for the image you're making right now.

One prompt, with and without ControlNet

Here's the difference on a real render. I gave Qwen-Image this prompt, which says nothing about a pose:

A watercolour illustration of a young woman with long dark hair, wearing a cream knitted jumper and wide light-blue jeans, in a bright room with white curtains.

Then I ran it again with a Canny ControlNet, fed the edges of a reference image. Same prompt, same seed, same everything else.

prompt only

control

Without it, the model chose everything the prompt left open: a closer crop, arms behind her back, no plant. With it, the figure stands exactly as in the reference, down to the cropped jumper, while the paint, the colours and the soft light all come from the prompt.

The same edges work for something the reference never was:

prompt only

control

Look at the right-hand side of the statue. The plant and the window from the reference turned up in a museum, because an edge map carries every line in the picture, not just the person. That's the most useful thing to understand about ControlNet: the control image decides what carries over, so choosing it is half the job.

The control image comes first

A ControlNet doesn't look at your photo directly. It reads one particular kind of image, and a preprocessor makes that image from your reference. Here's one reference turned into three kinds of control image:

Referencethe picture you start from

Canny edgesevery outline, background too

Depthwhat's near and what's far

Posethe body, and nothing else

Each kind keeps something different, which is how you choose between them:

ComfyUI ships with the Canny node. For the others, install the comfyui_controlnet_aux node pack through the Manager: it adds preprocessors for depth, pose, lineart, scribble, segmentation and more.

How to use ControlNet in ComfyUI

The classic setup uses two nodes: Load ControlNet Model, which loads the file from ComfyUI/models/controlnet/, and Apply ControlNet, which does the work. Unlike a LoRA, it doesn't touch the model wire. It sits on the prompt wires:

1. Get a ControlNet for your model

   Download one made for the model family you're using and put it in ComfyUI/models/controlnet/, then press R.

2. Make the control image

   Add Load Image for your reference, then the preprocessor that matches your ControlNet: Detect Edges (Canny) for a Canny model, a depth estimator for a depth model, and so on.

3. Put Apply ControlNet between your prompts and KSampler

   Wire the positive and negative prompts into it, the ControlNet into control_net, and the preprocessed image into image. Then take its two outputs to KSampler's positive and negative inputs.

4. Match the shape

   Give the empty latent the same aspect ratio as the reference. ComfyUI resizes the control image to fit the latent and crops from the centre when the shapes differ, so a portrait reference on a square canvas loses its top and bottom.

You can use more than one at once. Chain Apply ControlNet nodes: the prompts come out of the first and go into the second, and each one has its own ControlNet, control image and strength. Pose plus depth is a common pair, the body from one and the room from the other.

A second way in: model patches

Some newer ControlNets arrive as a model patch instead. The renders in this guide used one: DiffSynth's Canny ControlNet for Qwen-Image. It goes in ComfyUI/models/model_patches/, loads with Load Model Patch, and Apply Qwen Image DiffSynth ControlNet puts it on the model wire rather than the prompt wires. The idea is the same: a control image steering a model that stays unchanged. Here's the actual graph that made the bronze statue:

Strength, start and end

strength is how firmly the model is held to the control image. At 1.0 it follows closely, and lower values give it more room to make its own choices. How quickly it lets go depends on the ControlNet. With this one, 0.5 still held the pose completely, and 0.25 let go of it almost entirely:

strength 1.0

strength

The classic Apply ControlNet node adds two more settings, start_percent and end_percent, which decide which part of the sampling run the control applies to, from 0 (the first step) to 1 (the last). The early steps settle the composition and the late ones the fine detail, so ending the control early is a common trick: at 0.6 or 0.7 the layout is already fixed, and the model finishes the detail its own way, which often looks more natural than holding it to every line until the end.

Picking a ControlNet for your model

A ControlNet is trained alongside one model family, and it only works with that family. The originals were made for Stable Diffusion 1.5, and there are now separate sets for SDXL, FLUX, Qwen-Image and others. The model family is always in the file's description, and usually in its name. Loading one made for a different family either fails outright or does nothing.

You'll also see union ControlNets, which understand several kinds of control image in one file. ComfyUI's Set Union ControlNet Type node tells one which kind you're giving it. Its list covers pose, depth, soft edges and scribble, Canny and lineart, normal maps, segmentation, tile and repaint, or auto to let the model work it out.

ControlNet, LoRA or image-to-image?

All three change what comes out, and they're easy to mix up:

They combine well. A character LoRA plus a pose ControlNet puts the same person into the pose you choose, which neither can do alone.

When ControlNet doesn't behave

• It's ignored. Check that the ControlNet matches your model family, that strength isn't near zero, and that KSampler's prompts come out of Apply ControlNet rather than straight from the text encoders.
• The wrong image is going in. A Canny ControlNet expects an edge map. Feed it the reference photo itself and it can't read it.
• It copies too much. Canny carries the background along with the subject, as the bronze statue showed. Switch to pose or depth, or lower the strength.
• The picture looks stiff or traced. Lower the strength, or end the control earlier with end_percent.
• The subject is cut off. The reference and the latent have different shapes, so the control image was cropped to fit. Match the aspect ratio.

Questions people ask