Guide ComfyUI the complete
Workflows, ControlNet, IP-Adapter, AnimateDiff & Python API

What is ComfyUI?

ComfyUI is a node-based graphical interface for Stable Diffusionmodels, developed by comfyanonymous in 2023 and released as open source on GitHub. Unlike traditional interfaces like AUTOMATIC1111 that present you with tabs and ready-made parameters, ComfyUI exposes the entire image-generation pipeline as a graph of connected nodes — each node is a function, and each connection between nodes is a data flow.

This approach gives full transparency and control: you see exactly what happens at every stage of the denoising process, who gets what, and you can change any connection with a click. It also makes ComfyUI faster than its competitors — its memory (VRAM) management is much more efficient, and you can run heavy models on a GPU with less VRAM.

Beyond that, ComfyUI has become the de-facto standard in the professional community. When someone publishes a new Workflow — whether it's advanced ControlNet, IP-Adapter, or AnimateDiff — they publish it in ComfyUI's JSON format. Dragging an image created in ComfyUI onto its interface will automatically load the entire Workflow, including all parameters.

Comparison: ComfyUI vs AUTOMATIC1111 vs Forge

A roadmap to mastering ComfyUI

Below is a structured learning path — from a first installation to building a full production Pipeline with the Python API. Each stage builds on the previous, and you can stop at any point and work with what you've learned.

Installation

There are two main ways to install ComfyUI — the easy graphical way, and the manual way that gives full control.

Method 1: ComfyUI Desktop (recommended for beginners)

ComfyUI Desktop is a dedicated Windows and Mac application with one-click installation. It includes a built-in Python, CUDA drivers, and a graphical model manager. You can download it fromcomfy.org directly.

Method 2: Git Clone (recommended for developers)

git clone https://github.com/comfyanonymous/ComfyUI
cd ComfyUI
pip install -r requirements.txt
python main.py --listen

The --listen flag allows access from the local network (not just localhost). If you have an Nvidia GPU, ComfyUI will use it automatically. For Apple Silicon (M1/M2/M3) add --force-fp16.

The folder structure

After installation, you'll need to place models in the correct folders:

• models/checkpoints/ — main models (SDXL, SD1.5, Flux)
• models/loras/ — LoRA files
• models/controlnet/ — ControlNet models
• models/vae/ — external VAE models
• models/ipadapter/ — IP-Adapter models
• custom_nodes/ — community extensions

Nodes & Workflows — the basic paradigm

In ComfyUI, each node represents a single function — loading a model, encoding a prompt, creating a latent space, denoising, decoding. The lines connecting them (edges) represent data flow — the output of one node feeds into the input of the next. This lets you see exactly what happens at each stage, and change each connection separately.

The core nodes of every Workflow

• CheckpointLoaderSimple — loads an SDXL / SD1.5 / Flux model from the checkpoints folder
• CLIPTextEncode — encodes a prompt into a vector representation (positive and negative separately)
• EmptyLatentImage — creates an empty latent space at the desired size (width, height, batch_size)
• KSampler — the heart of the denoising process — runs the algorithm on the latent
• VAEDecode — decodes the latent space into a regular pixel image
• SaveImage — saves the image to disk with a filename and folder

The basic Workflow — Text to Image

The most basic Workflow connects 6 nodes in a chain: CheckpointLoaderSimple loads the model and outputs three outputs (model, clip, vae). The clip feeds into two CLIPTextEncode nodes — one for the positive prompt and one for the negative. EmptyLatentImage creates an empty latent. All the inputs come together in theKSampler, which outputs a processed latent. VAEDecode decodes it into an image, andSaveImage saves it.

KSampler in depth

The KSampler is the most important node in ComfyUI — it runs the denoising algorithm that turns noise into an image. Every parameter of it affects the final result, and it's important to understand what each one does.

All the parameters

• model — input from CheckpointLoaderSimple: the diffusion model itself
• positive — positive conditioning from CLIPTextEncode: what you want
• negative — negative conditioning: what you don't want
• latent_image — an empty latent (text2img) or from an image (img2img)
• seed — a number that defines the initial noise. The same seed = the same image
• steps — the number of denoising steps (20–30 for quality, 4–8 for LCM)
• cfg — Classifier Free Guidance: how closely to stick to the prompt
• sampler_name — the algorithm: euler, dpm++_2m, lcm and more
• scheduler — the noise schedule: karras, normal, exponential
• denoise — 1.0 for a new image, 0.5–0.8 for img2img

Samplers table

Schedulers: Usekarras in most cases — it distributes the noise in a way that produces sharper images. normal is the default, exponential suits SDXL, andsgm_uniform is dedicated to Flux.

ControlNet — full control over the structure

ControlNet is one of the most important innovations in the Stable Diffusion world. It lets you control the space, pose, edges and depth of the generated image — not just through the prompt, but through a reference image that guides the denoising process.

Types of ControlNet and when to use each

• Canny — detects sharp edges in the image. Suited to copying structure and pattern from a source to a new image with a different style
• Depth — a 3D map of the scene's depth. Preserves perspective and space even in a completely new style
• OpenPose — detects human body pose and bones. Lets you replicate a precise pose from a model to a new image
• Scribble — a rough sketch → a realistic image. Excellent for quick ideas
• SoftEdge — softer edges than Canny. Suited to subtle and artistic styles
• Tile — preserves existing details while Upscaling. The best way to increase resolution

A ControlNet Workflow

In ComfyUI, ControlNet adds two nodes to the regular Workflow: AuxPreprocessor (Canny/Depth/Pose) processes the reference image, ControlNetLoader loads the appropriate ControlNet model, andControlNetApply connects the conditioning to the KSampler. The order is: LoadImage → AuxPreprocessor → ControlNetLoader → ControlNetApply (+ CLIPTextEncode outputs) → KSampler.

ControlNet parameters

• strength — how much ControlNet controls: 0.5–1.0 for most cases. Above 1.0 loses creativity
• start_percent — at which stage ControlNet starts: 0.0 = from the beginning
• end_percent — at which stage ControlNet ends: 0.8 = ends before the finish (allows creativity in the final stages)

IP-Adapter — Style Transfer without training

IP-Adapter (Image Prompt Adapter) lets you transfer style and identity from a reference image to a new image — without training a LoRA, without waiting hours. You simply provide an image, and set how much weight to give it.

IP-Adapter versions

• IP-Adapter Standard — general style transfer: style, colors, textures
• IP-Adapter Plus — preserves more details from the source, suited to a complex reference
• IP-Adapter FaceID — preserving the identity of a specific face — the face is kept even in different scenes
• IP-Adapter Plus Face — combining the best of both: FaceID with Plus accuracy

A basic Workflow

The additional nodes that enter the Workflow: LoadImage (a reference image) → IPAdapterModelLoader (loading the IP-Adapter model) → IPAdapterApply (connects the reference to the model and the conditioning) → KSampler.

Important parameters

• weight 0.5–1.0 — for a full restyle: strong style transfer
• weight 0.3–0.6 — for a subjective style: a subtler effect, the prompt is in control
• start_at / end_at — set at which part of the sampling the IP-Adapter is active

AnimateDiff — from a set of images to coherent video

AnimateDiff adds temporal consistency — consistency between frames — to the image-generation process. Instead of producing independent frames that look different from each other, AnimateDiff "knows" the frames should be part of a coherent sequence and produces smooth video.

The core nodes of AnimateDiff

• ADE_AnimateDiffLoaderWithContext — loads AnimateDiff v3 with support for a long context window
• ADE_UseEvolvedSampling — wraps the regular KSampler with temporal attention
• VHS_VideoCombine — combines all the frames into a GIF or MP4

Main parameters

• frame_rate — frame rate: 8–16fps for smooth animation
• loop_count — 0 = an infinite loop (for GIF), 1+ for a number of loops
• context_length — how many frames are generated together: 16 = one second at 16fps

Motion LoRAs

Motion LoRAs are special LoRA files that add specific motion to the video: zoom-in, zoom-out, pan-left, pan-right, tilt-up, roll. You can combine several Motion LoRAs at different weights to create complex movements.

Python API — automation at scale

ComfyUI runs as an HTTP server on port 8188 with a WebSocket for progress updates. This lets you send Workflows programmatically, wait for completion, and download images — all from Python code.

A full Client with WebSocket

import json
import urllib.request
import urllib.parse
import uuid
import websocket

SERVER_ADDRESS = "127.0.0.1:8188"
CLIENT_ID = str(uuid.uuid4())

def queue_prompt(prompt_workflow: dict) -> dict:
    """Send a Workflow to the queue"""
    data = json.dumps({
        "prompt": prompt_workflow,
        "client_id": CLIENT_ID
    }).encode("utf-8")
    req = urllib.request.Request(
        f"http://{SERVER_ADDRESS}/prompt",
        data=data,
        headers={"Content-Type": "application/json"}
    )
    return json.loads(urllib.request.urlopen(req).read())

def get_image(filename: str, subfolder: str, folder_type: str) -> bytes:
    """Download a generated image"""
    params = urllib.parse.urlencode({
        "filename": filename,
        "subfolder": subfolder,
        "type": folder_type
    })
    return urllib.request.urlopen(
        f"http://{SERVER_ADDRESS}/view?{params}"
    ).read()

def wait_for_completion(prompt_id: str) -> list:
    """Wait for completion and fetch images via WebSocket"""
    ws = websocket.WebSocket()
    ws.connect(f"ws://{SERVER_ADDRESS}/ws?clientId={CLIENT_ID}")
    images = []
    while True:
        msg = json.loads(ws.recv())
        if msg["type"] == "executing":
            if msg["data"]["node"] is None and \
               msg["data"]["prompt_id"] == prompt_id:
                break
        elif msg["type"] == "executed":
            for img_data in msg["data"].get("output", {}).get("images", []):
                images.append(get_image(
                    img_data["filename"],
                    img_data["subfolder"],
                    img_data["type"]
                ))
    ws.close()
    return images

# a basic text-to-image Workflow
WORKFLOW = {
    "4": {"class_type": "CheckpointLoaderSimple",
          "inputs": {"ckpt_name": "v1-5-pruned-emaonly.safetensors"}},
    "5": {"class_type": "EmptyLatentImage",
          "inputs": {"width": 512, "height": 512, "batch_size": 1}},
    "6": {"class_type": "CLIPTextEncode",
          "inputs": {"clip": ["4", 1],
                     "text": "a beautiful sunset over Tel Aviv beach, photorealistic"}},
    "7": {"class_type": "CLIPTextEncode",
          "inputs": {"clip": ["4", 1],
                     "text": "ugly, blurry, watermark, low quality"}},
    "3": {"class_type": "KSampler",
          "inputs": {"model": ["4", 0], "positive": ["6", 0],
                     "negative": ["7", 0], "latent_image": ["5", 0],
                     "seed": 42, "steps": 20, "cfg": 7.0,
                     "sampler_name": "euler_ancestral",
                     "scheduler": "karras", "denoise": 1.0}},
    "8": {"class_type": "VAEDecode",
          "inputs": {"samples": ["3", 0], "vae": ["4", 2]}},
    "9": {"class_type": "SaveImage",
          "inputs": {"images": ["8", 0], "filename_prefix": "TLV"}}
}

result = queue_prompt(WORKFLOW)
images = wait_for_completion(result["prompt_id"])
from pathlib import Path
for i, img in enumerate(images):
    Path(f"output_{i}.png").write_bytes(img)
    print(f"Saved: output_{i}.png")

Batch Generation

import json
from pathlib import Path

def generate_batch(prompts: list, output_dir: str = "batch") -> None:
    """Generate a batch of images from multiple prompts"""
    Path(output_dir).mkdir(exist_ok=True)
    for idx, prompt_text in enumerate(prompts):
        wf = json.loads(json.dumps(WORKFLOW))
        wf["6"]["inputs"]["text"] = prompt_text
        wf["3"]["inputs"]["seed"] = idx
        result = queue_prompt(wf)
        imgs = wait_for_completion(result["prompt_id"])
        for img in imgs:
            out = Path(output_dir) / f"image_{idx:03d}.png"
            out.write_bytes(img)
            print(f"+ {out}")

# usage
generate_batch([
    "street art in Jerusalem old city, vibrant colors",
    "Negev desert at sunrise, dramatic lighting",
    "Tel Aviv skyline at night, long exposure",
    "Haifa bay from above, aerial photography",
])

5 practical projects

Below are 5 projects graded by difficulty — from a basic Workflow to a full production Pipeline with Python.

Cheat sheet

Keyboard shortcuts

CFG by model

Recommended Samplers

Essential Custom Nodes