Generating Image Scribbles with Stable Diffusion and ControlNet

Below snippet of code installs the necessary packages. It uses pip to install the following packages:

diffusers version 0.14.0: A package for stable diffusion and ControlNet algorithms, transformers, The Accelerate library from the GitHub repository : A library for high-performance deep learning network. Additionally, the code installs opencv-contrib-python and controlnet_aux packages, which are required for image processing and ControlNet auxiliary functionalities.

In [1]:

!pip install -q diffusers==0.14.0 transformers xformers git+https://github.com/huggingface/accelerate.git

  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 737.4/737.4 kB 10.9 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.2/7.2 MB 73.5 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.1/109.1 MB 3.1 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 236.8/236.8 kB 16.9 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 88.3 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 34.5 MB/s eta 0:00:00
  Building wheel for accelerate (pyproject.toml) ... done

In [2]:

!pip install -q opencv-contrib-python
!pip install -q controlnet_aux

     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 261.9/261.9 kB 5.3 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.2/42.2 kB 3.1 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 39.7 MB/s eta 0:00:00

In [3]:

from diffusers import StableDiffusionControlNetPipeline
from diffusers.utils import load_image

The code initializes an instance of the HEDdetector class using the pretrained 'lllyasviel/Annotators' model. This model is used for edge detection in the subsequent steps. It is trained on a dataset of annotated images.

In [17]:

from controlnet_aux import HEDdetector
from diffusers.utils import load_image

#hed = HEDdetector.from_pretrained('lllyasviel/ControlNet',filename="./models/control_sd15_depth.pth")
hed = HEDdetector.from_pretrained('lllyasviel/Annotators')

Let us do an example now.

In [ ]:

image = load_image("https://huggingface.co/lllyasviel/sd-controlnet-scribble/resolve/main/images/bag.png")
image_scribble = hed(image, scribble=True)

In [18]:

image.save("bag.png")
image_scribble.save("bag_scribble.png")

In [20]:

from PIL import Image
import IPython.display as display

display.display(Image.open("bag.png"))
display.display(Image.open("bag_scribble.png"))

Generating Image Scribbles with Stable Diffusion and ControlNet

Related Notebooks