infer_face_inpainting

Replace faces using diffusion inpainting. This algorithm uses Segformer for segmentation and RealVisXL V4.0 for inpainting.

This algorithm requires at least 13GB GPU VRAM to run.

🚀 Use with Ikomia API

1. Install Ikomia API

We strongly recommend using a virtual environment. If you're not sure where to start, we offer a tutorial here.

pip install ikomia

2. Create your workflow

from ikomia.dataprocess.workflow import Workflow
from ikomia.utils.displayIO import display

# Init your workflow
wf = Workflow()

# Add algorithm
algo = wf.add_task(name="infer_face_inpainting", auto_connect=True)

# Run on your image  
wf.run_on(url="https://images.pexels.com/photos/774909/pexels-photo-774909.jpeg?cs=srgb&dl=pexels-andrea-piacquadio-774909.jpg&fm=jpg&w=640&h=960")

# Inpect your result
display(algo.get_image_with_mask())
display(algo.get_output(2).get_image())
display(algo.get_output(3).get_image())

☀️ Use with Ikomia Studio

Ikomia Studio offers a friendly UI with the same features as the API.

• If you haven't started using Ikomia Studio yet, download and install it from this page.
• For additional guidance on getting started with Ikomia Studio, check out this blog post.

📝 Set algorithm parameters

• model_name_seg (str) - default 'matei-dorian/segformer-b5-finetuned-human-parsing': Name of the segmentation model. Other model available:
  • mattmdjaga/segformer_b2_clothes
• dilatation_percent_face (float) - default '0.001': Dilation percentage of the face mask.
• dilatation_percent_hair (float) - default '0.03': Dilation percentage of the hair mask.
• crop_percent_bottom_face (float) - default '0.05': The mask is generated accross the hair, face and neck. In case you don't want the neck to be segmented you crop this part by increasing the percentage.
• mask_only (bool) - default 'False': If True, only the segmentation step will be done. This allows for quick segmentation mask adjustement before doing the inpainting.
• model_name_diff (str) - default 'SG161222/RealVisXL_V4.0': Name of the stable diffusion model.

Note: for faster inference with less than 10 steps, use 'SG161222/RealVisXL_V4.0_Lightning' with guidance_scale=1-2

• prompt (str): Text prompt to guide the image generation.
• negative_prompt (str, optional): The prompt not to guide the image generation. Ignored when not using guidance (i.e., ignored if guidance_scale is less than 1).
• num_inference_steps (int) - default '50': Number of denoising steps (minimum: 1; maximum: 500). For 'RealVisXL_V4.0_Lightning' we recommend using between 5 and 10 steps.
• guidance_scale (float) - default '7.5': Scale for classifier-free guidance (minimum: 1; maximum: 20). For 'RealVisXL_V4.0_Lightning' we recommend using 0 and 2.
• strength (int) - default '0.75': Conceptually, indicates how much to transform the reference image. Must be between 0 and 1. image will be used as a starting point, adding more noise to it the larger the strength. The number of denoising steps depends on the amount of noise initially added. When strength is 1, added noise will be maximum and the denoising process will run for the full number of iterations specified in num_inference_steps. A value of 1, therefore, essentially ignores image.
• seed (int) - default '-1': Seed value. '-1' generates a random number between 0 and 191965535.

Parameters should be in strings format when added to the dictionary.

from ikomia.dataprocess.workflow import Workflow
from ikomia.utils.displayIO import display

# Init your workflow
wf = Workflow()

# Add algorithm
algo = wf.add_task(name="infer_face_inpainting", auto_connect=True)

algo.set_parameters({
        "model_name_seg":"matei-dorian/segformer-b5-finetuned-human-parsing",
        "dilatation_percent_face":"0.0",
        "dilatation_percent_hair":"0.03",
        "crop_percent_bottom_face":"0.22",
        "mask_only":"False",
        "model_name_diff": "SG161222/RealVisXL_V4.0",
        "prompt":"high quality, portrait photo, blond hair, detailed face, skin pores, no makeup",
        "negative_prompt":"(face asymmetry, eyes asymmetry, deformed eyes, open mouth)",
        "guidance_scale":"7.5",
        "num_inference_steps":"50",
        "strength":"0.75",
        "seed":"-1",
})


# Run on your image  
wf.run_on(url="https://images.pexels.com/photos/774909/pexels-photo-774909.jpeg?cs=srgb&dl=pexels-andrea-piacquadio-774909.jpg&fm=jpg&w=640&h=960")

# Inpect your result
display(algo.get_image_with_mask())
display(algo.get_output(2).get_image())
display(algo.get_output(3).get_image())

🔍 Explore algorithm outputs

Every algorithm produces specific outputs, yet they can be explored them the same way using the Ikomia API. For a more in-depth understanding of managing algorithm outputs, please refer to the documentation.

from ikomia.dataprocess.workflow import Workflow

# Init your workflow
wf = Workflow()

# Add algorithm
algo = wf.add_task(name="infer_face_inpainting", auto_connect=True)

# Run on your image  
wf.run_on(url="https://images.pexels.com/photos/774909/pexels-photo-774909.jpeg?cs=srgb&dl=pexels-andrea-piacquadio-774909.jpg&fm=jpg&w=640&h=960")

# Iterate over outputs
for output in algo.get_outputs():
    # Print information
    print(output)
    # Export it to JSON
    output.to_json()

📃 Citation

• Segformer
  • Documentation
  • Code source

@inproceedings{xie2021segformer,
  title={SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers},
  author={Xie, Enze and Wang, Wenhai and Yu, Zhiding and Anandkumar, Anima and Alvarez, Jose M and Luo, Ping},
  booktitle={Neural Information Processing Systems (NeurIPS)},
  year={2021}
}

• RealVisXL v4.0
  • Documentation