Guide LoRA Training
Training models for characters and Styles

What is LoRA? Mode 2 — Beginner

LoRA, short for Low-Rank Adaptation, is a Fine-tuning technique that lets you train a huge AI model (like Stable Diffusion) on your images — without changing all of the model's weights. The result: a small 3–50MB file that you can "attach" to any base model and get images with the character, style, or object you trained.

The analogy — a book with margin notes

Imagine a huge 7GB book (the base model). Full Fine-tuning = writing a whole new book from scratch. LoRA = adding margin notes to the pages — a small file that says: "for this character, change the output like so". The original model stays intact, but now it "remembers" your character.

Why LoRA and not Full Fine-tuning?

Approach	GPU RAM	Training time	File size
Full Fine-tuning	24GB+	24+ hours	7GB
LoRA	6–8GB	1–3 hours	3–50MB
QLoRA (Quantized)	4–6GB	2–4 hours	3–20MB

What can you train with LoRA?

• Characters (Face/Character LoRA) — a specific face that will appear in any scene you want
• Styles (Style LoRA) — an artist's aesthetic, anime style, a specific photography look, vintage and more
• Objects (Object LoRA) — a product, logo, vehicle, clothing item
• Concepts (Concept LoRA) — specific poses, lighting, a special composition

How does LoRA work? Mode 4 — Deep Concept

To understand LoRA you need to understand the idea of Low-Rank matrices. When an AI model learns, it stores the "knowledge" inside huge matrices of numbers (Weights). Full Fine-tuning = changing all of them. LoRA = changing a cheap approximation of those matrices.

The math behind LoRA — A × B instead of W

Instead of changing the full matrix W (very large), LoRA adds two small matrices: A and B. The product A × B approximates the desired change while keeping a low dimension (Low-Rank). The original W stays frozen — only A and B are updated.

Original Layer W (huge — tens of millions of parameters)
         ↓
W_frozen + (A × B) = W_new
        (only a 3MB LoRA)

A numeric example:
  W  = a 1024 × 1024 matrix = ~4,000,000 parameters
  A  = a 1024 × 16 matrix   =    16,384  parameters
  B  = a 16 × 1024 matrix   =    16,384  parameters
  LoRA total = 32,768 instead of 4,000,000 (less than 1%!)

Rank (r) — how "smart" is the LoRA?

The Rank parameter determines the dimension of the A and B matrices. Common values: 4, 8, 16, 32, 64. The larger the Rank — the more the LoRA can learn, but it's also larger and may overfit on the dataset.

Rank	File size	Best for	Risk
r=4	~3MB	Light Style, Flux.1	Underfitting
r=8	~6MB	Medium Style, objects	Low
r=16	~12MB	Character / complex Style	Low–medium
r=32	~24MB	A character with many details	Overfits easily
r=64	~48MB	Special cases only	High

Alpha — the learning ratio

The Alpha parameter determines how strongly the LoRA "affects" the model during training. The internal formula: weight = alpha / rank. A common best practice: alpha = rank / 2. For example with r=16, use alpha=8 (weight=0.5).

Network Dimension vs Alpha — the practical difference

In Kohya SS, "Network Dimension" = Rank. "Network Alpha" = Alpha. Setting Alpha=Rank (e.g. r=16, alpha=16) gives weight=1.0 — stronger but less stable. An Alpha lower than the Rank (alpha=rank/2 to alpha=rank/4) is the more stable approach.

Where LoRA is applied — UNet Attention Layers

LoRA isn't applied to all the model's layers. It focuses on theAttention Layers of the UNet — the layers responsible for "understanding" the prompt and forming relationships between parts of the image. The reason it works: most of the generic visual information is already "ready" in the frozen convolution layers.

Preparing the Dataset Mode 3 — Curriculum

Dataset quality is the most important factor for a LoRA's success. Bad images = a bad LoRA, even with the best settings. The preparation stage is usually the hardest.

How many images do you need?

• Character (Face LoRA): 20–30 high-quality images
• Style (Style LoRA): 50–100 representative images
• Object: 15–25 images from different angles
• Abstract concept: 30–50 example images

Image requirements

• Resolution: at least 512×512 pixels, preferably 768×768 and up for SDXL
• Backgrounds: varied — not all with the same white background
• Angles: images from different directions — frontal, side, three-quarter
• Lighting: different lighting conditions — natural, artificial, dramatic, soft
• Expressions (for characters): smiling, serious, laughing, surprised — variety helps
• Format: JPG or PNG, no watermarks, no text overlays

Caption Writing — describing each image

Each image in the Dataset needs a matching TXT file with an accurate caption. The Caption teaches the LoRA what to associate with your trigger word and what is already known from the base model. The more accurate the caption, the more cleanly the LoRA learns.

Tools for Auto-Captioning

• WD14 Tagger — automatically adds tags to anime and illustration images. Fast and accurate for illustrations
• BLIP2 — generates natural-language descriptions for photorealistic images
• CLIP Interrogator — maps an image to a prompt-style description
• Florence-2 — a new and very accurate model (2024), recommended as the default

Trigger Word — the key to your character

The Trigger Word is a unique word in the prompt that activates the LoRA. It's important that it be rare and not a word the model already knows and associates with something else.

• For a character: ohwx man, sks woman, xyz person
• For a style: style_myname, paintby_artist artstyle
• For an object: myobj_thing, unique_productname

Dataset folder structure

dataset/
├── 20_ohwx_man/                   ← "20" = repeats, "ohwx_man" = class name
│   ├── photo001.jpg
│   ├── photo001.txt               ← "ohwx man, portrait, white background, looking at camera"
│   ├── photo002.jpg
│   ├── photo002.txt               ← "ohwx man, outdoor, smiling, casual clothing, sunny day"
│   ├── photo003.jpg
│   ├── photo003.txt               ← "ohwx man, side profile, studio lighting, dark background"
│   └── ... (20-30 pairs in total)
└── regularization/                ← optional — images of "some man" to prevent drift
    ├── reg001.jpg
    ├── reg001.txt                 ← "a man, portrait, photorealistic"
    └── ...

Training with Kohya SS Mode 4 — Technical

Kohya SS is the most common tool for training LoRA. It has a convenient GUI and support for SDXL, SD 1.5, and Flux.1. You can run it locally or on Google Colab.

Installation options

• Windows (GPU Nvidia): Download from GitHub, run setup.bat, open with gui.bat
• Mac (MPS / CPU): Run setup.sh, limited support — very slow training, Colab is better
• Google Colab (free): ready-made Notebooks exist — fast and convenient, with a time limit
• RunPod / Vast.ai: a cloud GPU — ~$0.20–0.80/hour, no time limit

Important settings in the Kohya GUI

Setting	Recommended value (character)	Explanation
Base Model	SDXL 1.0 / Flux.1-dev	The model you're training on
LoRA Rank (network_dim)	16	The size of the A,B matrices
Alpha	8	LoRA strength (rank/2)
Learning Rate (UNet)	1e-4	0.0001 — the UNet learning rate
Learning Rate (Text Encoder)	5e-5	0.00005 — more conservative
Epochs	10–15	The number of passes over the Dataset
Batch Size	1–2	images in parallel (1 for a 6GB GPU)
Noise Offset	0.1	improving contrast and color in the output
Optimizer	AdamW8bit	GPU-memory efficient
LR Scheduler	cosine_with_restarts	a smooth decay in the learning rate
Save every N epochs	5	saving checkpoints for testing

Running directly with the Kohya Script

accelerate launch train_network.py \
  --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \
  --dataset_config=dataset.toml \
  --output_dir="./lora_output" \
  --output_name="my_character" \
  --network_module=networks.lora \
  --network_dim=16 \
  --network_alpha=8 \
  --learning_rate=1e-4 \
  --unet_lr=1e-4 \
  --text_encoder_lr=5e-5 \
  --optimizer_type="AdamW8bit" \
  --lr_scheduler="cosine_with_restarts" \
  --lr_warmup_steps=100 \
  --noise_offset=0.1 \
  --max_train_epochs=15 \
  --save_every_n_epochs=5 \
  --save_model_as=safetensors \
  --mixed_precision=fp16 \
  --gradient_checkpointing \
  --xformers

The dataset.toml file

[general]
shuffle_caption = true
caption_extension = ".txt"
keep_tokens = 1          # the trigger word is always first

[[datasets]]
resolution = 1024        # for SDXL (512 for SD 1.5)
batch_size = 1

  [[datasets.subsets]]
  image_dir = "./dataset/20_ohwx_man"
  class_tokens = "ohwx man"
  num_repeats = 20

  # Regularization — optional
  [[datasets.subsets]]
  image_dir = "./dataset/regularization"
  class_tokens = "a man"
  num_repeats = 1
  is_reg = true

Using LoRA Mode 4 — Practical

After training finishes, you get a .safetensorsfile. Now you can load it in any interface and start generating images.

Loading in ComfyUI

In ComfyUI you add a Load LoRA node between the Checkpoint Loader and the KSampler:

Checkpoint Loader
      ↓ model + clip
 Load LoRA Node
  ├── lora_name: my_character.safetensors
  ├── strength_model: 0.8
  └── strength_clip:  0.8
      ↓ model + clip
CLIP Text Encode → KSampler → VAE Decode → image

Loading in Automatic1111 (A1111)

In the Prompt field, add the LoRA syntax directly:

# Basic syntax:
<lora:LoRA_filename_without_extension:strength>

# character example:
ohwx man wearing a spacesuit, professional photo, 4k <lora:my_character:0.8>

# art-style example:
a serene landscape <lora:monet_style:0.6>

# combining two LoRAs:
ohwx man in anime style, vibrant colors <lora:my_character:0.7> <lora:anime_xl:0.5>

LoRA strength — Strength Guide

LoRA type	Recommended range	Note
Character (Face)	0.7–1.0	Higher than 1.0 = distortions
Style	0.3–0.7	Low = a subtle blend with the style
Object	0.6–0.9	Depends on the object's prominence
A second LoRA (Stack)	About 0.1 lower than the first	to prevent a conflict between LoRAs

Python with diffusers

from diffusers import StableDiffusionXLPipeline
import torch

# loading the base model
pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    use_safetensors=True
).to("cuda")

# loading the LoRA
pipe.load_lora_weights(
    "./loras/my_character.safetensors",
    adapter_name="character"
)

# setting the strength
pipe.set_adapters(["character"], adapter_weights=[0.8])

# generating an image
image = pipe(
    prompt="ohwx man wearing a spacesuit, professional photo, 4k, detailed",
    negative_prompt="blurry, low quality, distorted face, extra limbs",
    num_inference_steps=30,
    guidance_scale=7.5,
    width=1024,
    height=1024,
).images[0]

image.save("output.png")
print("Saved successfully!")

Stacking LoRAs — combining several LoRAs together

# Python — loading two LoRAs together
pipe.load_lora_weights("./loras/character.safetensors", adapter_name="char")
pipe.load_lora_weights("./loras/style.safetensors",     adapter_name="style")

# setting the strength for each
pipe.set_adapters(["char", "style"], adapter_weights=[0.8, 0.5])

# a prompt with the trigger words of both:
# "ohwx man in style_myname artstyle, ..."

Common troubleshooting

Overfitting — the character breaks proportions

Symptoms: images look too "locked" onto the training data, strange recurring backgrounds, distorted faces at non-frontal angles, a lack of flexibility in the prompt.

• Reduce epochs — try epoch 8–10 instead of 15
• Add Regularization Images (200–500 images of "some man")
• Reduce the Learning Rate to5e-5
• Lower the Rank — try r=8 instead of r=16
• Grow the Dataset with more varied images

Underfitting — the character doesn't look right

Symptoms: the trigger word doesn't produce the character, the face differs from the original, the LoRA looks "weak" even at high strength.

• Increase epochs — try 15–20
• Check the captions — does the trigger word appear inall of them?
• Increase the Learning Rate to2e-4
• Add Dataset images — at least 20 images for a character
• Check thatkeep_tokens=1 is set (trigger word first)

Style Bleeding — the LoRA "sticks" to every image

Symptoms: even without the trigger word, the model produces images that look like the training. The style/character "leaks" into all the outputs.

• Use a more specific trigger word in all the captions
• Reduce the LoRA strength in use to 0.5–0.6
• Add more Regularization Images
• Check thatshuffle_caption=true is enabled
• Lower the number of Repeats (from 20 to 10)

OOM — Out of Memory

Symptoms: A CUDA out of memory error right at the start of training. The GPU can't hold everything it needs in memory.

• Reduce batch size to 1
• Enable --gradient_checkpointing (saves 30–40% VRAM)
• Add --xformers for Nvidia cards
• Work at resolution 768 instead of 1024
• Use--mixed_precision=fp16 or bf16
• Try Colab Pro with an A100 — 40GB VRAM with no limits

Projects Mode 5 — Hands-On

Four projects in increasing order of difficulty — from an easy start to a complex Workflow. Each builds on the previous.

Cheat sheet — LoRA Quick Reference Mode 7 — Reference

Recommended settings by goal

Goal	Rank	Alpha	UNet LR	TE LR	Epochs	Images
Character (SDXL)	16	8	1e-4	5e-5	15	20–30
Style (SDXL)	8	4	5e-5	2e-5	10	50–100
Character (Flux.1)	4	2	1e-4	—	10	20–25
Style (Flux.1)	4	4	5e-5	—	8	40–60
Object (SDXL)	8	8	1e-4	5e-5	12	15–25

Trigger Word Best Practices

• Use 2–3 unique letters + a common word: sks, ohwx, xyz
• Add a class token: ohwx man, xyz woman, sks dog
• For a style — add artstyle at the end: myartist artstyle
• Don't use celebrity names as a trigger word — the model confuses them
• Set keep_tokens=1 In the TOML file — the trigger word is always first in the caption
• Check on Civitai which trigger words are common for similar LoRAs

Dataset Checklist before training

• Each image at least 512×512 pixels (preferably 768+ for SDXL training)
• Each image accompanied by a .txt matching file (same name, different extension)
• The trigger word appears inall the captions
• No blurry, cropped, watermarked images, or heavy filters
• Variety in backgrounds, angles and lighting — at least 3 different kinds
• Folder name in the format: NUMBER_classname
• Regularization Images prepared (optional but recommended for characters)
• GPU Driver and CUDA updated before running

Recommended tools — Ecosystem

Tool	Use	Price
Kohya SS	LoRA training — the main tool	Free (Open Source)
ComfyUI	Running LoRA + Workflows	Free (Open Source)
Automatic1111	A simple interface for using LoRA	Free (Open Source)
Civitai	A LoRA marketplace + community sharing	Free + premium
RunPod	A cloud GPU for training	$0.20–0.80/h
Google Colab	Free training with limits	Free / Pro $9.99/mo
Florence-2	Accurate Auto-captioning	Free (HuggingFace)
WD14 Tagger	Auto-captioning for illustrations	Free (HuggingFace)