/stable-diffusion

name: stable-diffusion
description: Run Stable Diffusion locally with diffusers — text-to-image, img2img, inpainting, ControlNet, and SDXL.
version: 1.0.0
author: hermes-CCC (ported from Hermes Agent by NousResearch)
license: MIT
metadata:
  hermes:
    tags: [MLOps, Stable-Diffusion, Image-Generation, Diffusers, Text-to-Image]
    related_skills: []

Stable Diffusion

Purpose

• Use this skill to generate or edit images locally with Hugging Face `diffusers`.
• Prefer it for text-to-image, img2img, inpainting, and model composition workflows.
• This skill covers both classic Stable Diffusion models and SDXL.
• It is useful for scripted generation, reproducible experiments, and GPU-backed image pipelines.

Install

pip install diffusers transformers accelerate torch

• You typically also need a compatible CUDA-enabled PyTorch build for GPU inference.
• Confirm the install in Python before pulling large checkpoints.

Core Libraries

• `diffusers` for pipeline abstractions
• `transformers` for text encoders and related model components
• `accelerate` for efficient device loading and memory movement
• `torch` for runtime execution

Text-to-Image With SD 1.5

• Classic Stable Diffusion 1.5 uses `StableDiffusionPipeline`.

import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")

image = pipe(
    prompt="a cinematic photo of a mountain observatory at sunrise",
    negative_prompt="blurry, low quality, distorted",
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]

image.save("output.png")

SDXL

• SDXL typically uses `StableDiffusionXLPipeline`.

import torch
from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")

image = pipe(
    prompt="a highly detailed editorial photo of a futuristic library interior",
    negative_prompt="low resolution, deformed, extra limbs",
    num_inference_steps=35,
    guidance_scale=6.5,
).images[0]

image.save("sdxl-output.png")

• SDXL generally produces stronger prompt fidelity and image quality than SD 1.5, but it also requires more VRAM.

Key Parameters

• `prompt`: the main text instruction
• `negative_prompt`: what to suppress
• `num_inference_steps`: denoising step count
• `guidance_scale`: classifier-free guidance strength
• `seed`: random seed for reproducibility

Seeded Generation

• Use a seed when you need repeatable outputs:

import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
).to("cuda")

generator = torch.Generator(device="cuda").manual_seed(42)

image = pipe(
    prompt="a clean product photo of a ceramic mug on a wood table",
    negative_prompt="blurry, noisy, warped",
    num_inference_steps=28,
    guidance_scale=7.0,
    generator=generator,
).images[0]

image.save("seeded-output.png")

• The same seed and settings help with debugging prompt and LoRA changes.

Save Output

• Save the generated image with PIL:

image.save("output.png")

• Always save prompt metadata separately if you need auditability or experiment tracking.

Img2Img

• Use `StableDiffusionImg2ImgPipeline` to transform an existing image while preserving composition.

import torch
from diffusers import StableDiffusionImg2ImgPipeline
from PIL import Image

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
).to("cuda")

init_image = Image.open("input.png").convert("RGB").resize((768, 768))

image = pipe(
    prompt="turn this concept sketch into a polished sci-fi matte painting",
    negative_prompt="blurry, low contrast, artifacts",
    image=init_image,
    strength=0.65,
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]

image.save("img2img-output.png")

• Lower `strength` preserves more of the input image.
• Higher `strength` pushes the result further away from the source.

Inpainting

• Use `StableDiffusionInpaintPipeline` to replace or repair masked regions.

import torch
from diffusers import StableDiffusionInpaintPipeline
from PIL import Image

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    torch_dtype=torch.float16,
).to("cuda")

image = Image.open("scene.png").convert("RGB").resize((512, 512))
mask = Image.open("mask.png").convert("RGB").resize((512, 512))

result = pipe(
    prompt="replace the missing area with a wooden chair",
    negative_prompt="blurry, malformed, duplicate objects",
    image=image,
    mask_image=mask,
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]

result.save("inpaint-output.png")

• White mask regions are typically where edits are applied.
• Good masks matter as much as prompts for reliable inpainting.

ControlNet

• ControlNet is useful when you want stronger control over pose, depth, edges, or composition.
• Typical uses include pose-guided character generation, depth-aware edits, and line-art conditioning.
• Pair ControlNet with SD 1.5 or SDXL depending on the model combination you are using.

Memory Optimization

• Reduce memory pressure with built-in helpers:

pipe.enable_model_cpu_offload()
pipe.enable_attention_slicing()

• `pipe.enable_model_cpu_offload()` is often helpful on constrained GPUs.
• `pipe.enable_attention_slicing()` can reduce peak memory at some performance cost.
• These settings are practical for laptops and single-GPU consumer machines.

LoRA Loading

• Load LoRA adapters to specialize style, subject, or composition behavior:

pipe.load_lora_weights("./lora.safetensors")