# Marigold Computer Vision

![marigold](https://marigoldmonodepth.github.io/images/teaser_collage_compressed.jpg)

Marigold was proposed in 
[Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation](https://huggingface.co/papers/2312.02145), 
a CVPR 2024 Oral paper by 
[Bingxin Ke](http://www.kebingxin.com/), 
[Anton Obukhov](https://www.obukhov.ai/), 
[Shengyu Huang](https://shengyuh.github.io/), 
[Nando Metzger](https://nandometzger.github.io/), 
[Rodrigo Caye Daudt](https://rcdaudt.github.io/), and 
[Konrad Schindler](https://scholar.google.com/citations?user=FZuNgqIAAAAJ&hl=en).
The core idea is to **repurpose the generative prior of Text-to-Image Latent Diffusion Models (LDMs) for traditional 
computer vision tasks**.
This approach was explored by fine-tuning Stable Diffusion for **Monocular Depth Estimation**, as demonstrated in the 
teaser above.

Marigold was later extended in the follow-up paper, 
[Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis](https://huggingface.co/papers/2312.02145), 
authored by 
[Bingxin Ke](http://www.kebingxin.com/), 
[Kevin Qu](https://www.linkedin.com/in/kevin-qu-b3417621b/?locale=en_US), 
[Tianfu Wang](https://tianfwang.github.io/), 
[Nando Metzger](https://nandometzger.github.io/), 
[Shengyu Huang](https://shengyuh.github.io/), 
[Bo Li](https://www.linkedin.com/in/bobboli0202/), 
[Anton Obukhov](https://www.obukhov.ai/), and 
[Konrad Schindler](https://scholar.google.com/citations?user=FZuNgqIAAAAJ&hl=en).
This work expanded Marigold to support new modalities such as **Surface Normals** and **Intrinsic Image Decomposition** 
(IID), introduced a training protocol for **Latent Consistency Models** (LCM), and demonstrated **High-Resolution** (HR) 
processing capability.

> [!TIP]
> The early Marigold models (`v1-0` and earlier) were optimized for best results with at least 10 inference steps.
> LCM models were later developed to enable high-quality inference in just 1 to 4 steps.
> Marigold models `v1-1` and later use the DDIM scheduler to achieve optimal 
> results in as few as 1 to 4 steps.

## Available Pipelines

Each pipeline is tailored for a specific computer vision task, processing an input RGB image and generating a 
corresponding prediction.
Currently, the following computer vision tasks are implemented:

| Pipeline                                                                                                                                          | Recommended Model Checkpoints                                                                                                                                                                           |                              Spaces (Interactive Apps)                               | Predicted Modalities                                                                                                                                                               |
|---------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------:|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [MarigoldDepthPipeline](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_depth.py)           | [prs-eth/marigold-depth-v1-1](https://huggingface.co/prs-eth/marigold-depth-v1-1)                                                                                                                       |          [Depth Estimation](https://huggingface.co/spaces/prs-eth/marigold)          | [Depth](https://en.wikipedia.org/wiki/Depth_map), [Disparity](https://en.wikipedia.org/wiki/Binocular_disparity)                                                                   |
| [MarigoldNormalsPipeline](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_normals.py)       | [prs-eth/marigold-normals-v1-1](https://huggingface.co/prs-eth/marigold-normals-v1-1)                                                                                                                   | [Surface Normals Estimation](https://huggingface.co/spaces/prs-eth/marigold-normals) | [Surface normals](https://en.wikipedia.org/wiki/Normal_mapping)                                                                                                                    |
| [MarigoldIntrinsicsPipeline](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_intrinsics.py) | [prs-eth/marigold-iid-appearance-v1-1](https://huggingface.co/prs-eth/marigold-iid-appearance-v1-1),[prs-eth/marigold-iid-lighting-v1-1](https://huggingface.co/prs-eth/marigold-iid-lighting-v1-1) | [Intrinsic Image Decomposition](https://huggingface.co/spaces/prs-eth/marigold-iid)  | [Albedo](https://en.wikipedia.org/wiki/Albedo), [Materials](https://www.n.aiq3d.com/wiki/roughnessmetalnessao-map), [Lighting](https://en.wikipedia.org/wiki/Diffuse_reflection)   |

## Available Checkpoints

All original checkpoints are available under the [PRS-ETH](https://huggingface.co/prs-eth/) organization on Hugging Face.
They are designed for use with diffusers pipelines and the [original codebase](https://github.com/prs-eth/marigold), which can also be used to train 
new model checkpoints.
The following is a summary of the recommended checkpoints, all of which produce reliable results with 1 to 4 steps. 

| Checkpoint                                                                                          | Modality     | Comment                                                                                                                                                                              |
|-----------------------------------------------------------------------------------------------------|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [prs-eth/marigold-depth-v1-1](https://huggingface.co/prs-eth/marigold-depth-v1-1)                   | Depth        | Affine-invariant depth prediction assigns each pixel a value between 0 (near plane) and 1 (far plane), with both planes determined by the model during inference.                    |
| [prs-eth/marigold-normals-v0-1](https://huggingface.co/prs-eth/marigold-normals-v0-1)               | Normals      | The surface normals predictions are unit-length 3D vectors in the screen space camera, with values in the range from -1 to 1.                                                        |
| [prs-eth/marigold-iid-appearance-v1-1](https://huggingface.co/prs-eth/marigold-iid-appearance-v1-1) | Intrinsics   | InteriorVerse decomposition is comprised of Albedo and two BRDF material properties: Roughness and Metallicity.                                                                      | 
| [prs-eth/marigold-iid-lighting-v1-1](https://huggingface.co/prs-eth/marigold-iid-lighting-v1-1)     | Intrinsics   | HyperSim decomposition of an image $I$ is comprised of Albedo $A$, Diffuse shading $S$, and Non-diffuse residual $R$: $I = A*S+R$. |

> [!TIP]
> Make sure to check out the Schedulers [guide](../../using-diffusers/schedulers) to learn how to explore the tradeoff 
> between scheduler speed and quality, and see the [reuse components across pipelines](../../using-diffusers/loading#reuse-a-pipeline) section to learn how to 
> efficiently load the same components into multiple pipelines. 
> Also, to know more about reducing the memory usage of this pipeline, refer to the ["Reduce memory usage"] section 
> [here](../../using-diffusers/svd#reduce-memory-usage).

> [!WARNING]
> Marigold pipelines were designed and tested with the scheduler embedded in the model checkpoint.
> The optimal number of inference steps varies by scheduler, with no universal value that works best across all cases.
> To accommodate this, the `num_inference_steps` parameter in the pipeline's `__call__` method defaults to `None` (see the 
> API reference).
> Unless set explicitly, it inherits the value from the `default_denoising_steps` field in the checkpoint configuration 
> file (`model_index.json`).
> This ensures high-quality predictions when invoking the pipeline with only the `image` argument.

See also Marigold [usage examples](../../using-diffusers/marigold_usage).

## Marigold Depth Prediction API[[diffusers.MarigoldDepthPipeline]]

#### diffusers.MarigoldDepthPipeline[[diffusers.MarigoldDepthPipeline]]

[Source](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_depth.py#L104)

Pipeline for monocular depth estimation using the Marigold method: https://marigoldmonodepth.github.io.

This model inherits from [DiffusionPipeline](/docs/diffusers/main/en/api/pipelines/overview#diffusers.DiffusionPipeline). Check the superclass documentation for the generic methods the
library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)

__call__diffusers.MarigoldDepthPipeline.__call__https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_depth.py#L347[{"name": "image", "val": ": typing.Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, typing.List[PIL.Image.Image], typing.List[numpy.ndarray], typing.List[torch.Tensor]]"}, {"name": "num_inference_steps", "val": ": typing.Optional[int] = None"}, {"name": "ensemble_size", "val": ": int = 1"}, {"name": "processing_resolution", "val": ": typing.Optional[int] = None"}, {"name": "match_input_resolution", "val": ": bool = True"}, {"name": "resample_method_input", "val": ": str = 'bilinear'"}, {"name": "resample_method_output", "val": ": str = 'bilinear'"}, {"name": "batch_size", "val": ": int = 1"}, {"name": "ensembling_kwargs", "val": ": typing.Optional[typing.Dict[str, typing.Any]] = None"}, {"name": "latents", "val": ": typing.Union[torch.Tensor, typing.List[torch.Tensor], NoneType] = None"}, {"name": "generator", "val": ": typing.Union[torch._C.Generator, typing.List[torch._C.Generator], NoneType] = None"}, {"name": "output_type", "val": ": str = 'np'"}, {"name": "output_uncertainty", "val": ": bool = False"}, {"name": "output_latent", "val": ": bool = False"}, {"name": "return_dict", "val": ": bool = True"}]- **image** (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`), --
  `List[torch.Tensor]`: An input image or images used as an input for the depth estimation task. For
  arrays and tensors, the expected value range is between `[0, 1]`. Passing a batch of images is possible
  by providing a four-dimensional array or a tensor. Additionally, a list of images of two- or
  three-dimensional arrays or tensors can be passed. In the latter case, all list elements must have the
  same width and height.
- **num_inference_steps** (`int`, *optional*, defaults to `None`) --
  Number of denoising diffusion steps during inference. The default value `None` results in automatic
  selection.
- **ensemble_size** (`int`, defaults to `1`) --
  Number of ensemble predictions. Higher values result in measurable improvements and visual degradation.
- **processing_resolution** (`int`, *optional*, defaults to `None`) --
  Effective processing resolution. When set to `0`, matches the larger input image dimension. This
  produces crisper predictions, but may also lead to the overall loss of global context. The default
  value `None` resolves to the optimal value from the model config.
- **match_input_resolution** (`bool`, *optional*, defaults to `True`) --
  When enabled, the output prediction is resized to match the input dimensions. When disabled, the longer
  side of the output will equal to `processing_resolution`.
- **resample_method_input** (`str`, *optional*, defaults to `"bilinear"`) --
  Resampling method used to resize input images to `processing_resolution`. The accepted values are:
  `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`.
- **resample_method_output** (`str`, *optional*, defaults to `"bilinear"`) --
  Resampling method used to resize output predictions to match the input resolution. The accepted values
  are `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`.
- **batch_size** (`int`, *optional*, defaults to `1`) --
  Batch size; only matters when setting `ensemble_size` or passing a tensor of images.
- **ensembling_kwargs** (`dict`, *optional*, defaults to `None`) --
  Extra dictionary with arguments for precise ensembling control. The following options are available:
  - reduction (`str`, *optional*, defaults to `"median"`): Defines the ensembling function applied in
    every pixel location, can be either `"median"` or `"mean"`.
  - regularizer_strength (`float`, *optional*, defaults to `0.02`): Strength of the regularizer that
    pulls the aligned predictions to the unit range from 0 to 1.
  - max_iter (`int`, *optional*, defaults to `2`): Maximum number of the alignment solver steps. Refer to
    `scipy.optimize.minimize` function, `options` argument.
  - tol (`float`, *optional*, defaults to `1e-3`): Alignment solver tolerance. The solver stops when the
    tolerance is reached.
  - max_res (`int`, *optional*, defaults to `None`): Resolution at which the alignment is performed;
    `None` matches the `processing_resolution`.
- **latents** (`torch.Tensor`, or `List[torch.Tensor]`, *optional*, defaults to `None`) --
  Latent noise tensors to replace the random initialization. These can be taken from the previous
  function call's output.
- **generator** (`torch.Generator`, or `List[torch.Generator]`, *optional*, defaults to `None`) --
  Random number generator object to ensure reproducibility.
- **output_type** (`str`, *optional*, defaults to `"np"`) --
  Preferred format of the output's `prediction` and the optional `uncertainty` fields. The accepted
  values are: `"np"` (numpy array) or `"pt"` (torch tensor).
- **output_uncertainty** (`bool`, *optional*, defaults to `False`) --
  When enabled, the output's `uncertainty` field contains the predictive uncertainty map, provided that
  the `ensemble_size` argument is set to a value above 2.
- **output_latent** (`bool`, *optional*, defaults to `False`) --
  When enabled, the output's `latent` field contains the latent codes corresponding to the predictions
  within the ensemble. These codes can be saved, modified, and used for subsequent calls with the
  `latents` argument.
- **return_dict** (`bool`, *optional*, defaults to `True`) --
  Whether or not to return a [MarigoldDepthOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldDepthOutput) instead of a plain tuple.0[MarigoldDepthOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldDepthOutput) or `tuple`If `return_dict` is `True`, [MarigoldDepthOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldDepthOutput) is returned, otherwise a
`tuple` is returned where the first element is the prediction, the second element is the uncertainty
(or `None`), and the third is the latent (or `None`).

Function invoked when calling the pipeline.

Examples:
```py
>>> import diffusers
>>> import torch

>>> pipe = diffusers.MarigoldDepthPipeline.from_pretrained(
...     "prs-eth/marigold-depth-v1-1", variant="fp16", torch_dtype=torch.float16
... ).to("cuda")

>>> image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")
>>> depth = pipe(image)

>>> vis = pipe.image_processor.visualize_depth(depth.prediction)
>>> vis[0].save("einstein_depth.png")

>>> depth_16bit = pipe.image_processor.export_depth_to_16bit_png(depth.prediction)
>>> depth_16bit[0].save("einstein_depth_16bit.png")
```

**Parameters:**

unet (`UNet2DConditionModel`) : Conditional U-Net to denoise the depth latent, conditioned on image latent.

vae (`AutoencoderKL`) : Variational Auto-Encoder (VAE) Model to encode and decode images and predictions to and from latent representations.

scheduler (`DDIMScheduler` or `LCMScheduler`) : A scheduler to be used in combination with `unet` to denoise the encoded image latents.

text_encoder (`CLIPTextModel`) : Text-encoder, for empty text embedding.

tokenizer (`CLIPTokenizer`) : CLIP tokenizer.

prediction_type (`str`, *optional*) : Type of predictions made by the model.

scale_invariant (`bool`, *optional*) : A model property specifying whether the predicted depth maps are scale-invariant. This value must be set in the model config. When used together with the `shift_invariant=True` flag, the model is also called "affine-invariant". NB: overriding this value is not supported.

shift_invariant (`bool`, *optional*) : A model property specifying whether the predicted depth maps are shift-invariant. This value must be set in the model config. When used together with the `scale_invariant=True` flag, the model is also called "affine-invariant". NB: overriding this value is not supported.

default_denoising_steps (`int`, *optional*) : The minimum number of denoising diffusion steps that are required to produce a prediction of reasonable quality with the given model. This value must be set in the model config. When the pipeline is called without explicitly setting `num_inference_steps`, the default value is used. This is required to ensure reasonable results with various model flavors compatible with the pipeline, such as those relying on very short denoising schedules (`LCMScheduler`) and those with full diffusion schedules (`DDIMScheduler`).

default_processing_resolution (`int`, *optional*) : The recommended value of the `processing_resolution` parameter of the pipeline. This value must be set in the model config. When the pipeline is called without explicitly setting `processing_resolution`, the default value is used. This is required to ensure reasonable results with various model flavors trained with varying optimal processing resolution values.

**Returns:**

`[MarigoldDepthOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldDepthOutput) or `tuple``

If `return_dict` is `True`, [MarigoldDepthOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldDepthOutput) is returned, otherwise a
`tuple` is returned where the first element is the prediction, the second element is the uncertainty
(or `None`), and the third is the latent (or `None`).

#### diffusers.pipelines.marigold.MarigoldDepthOutput[[diffusers.pipelines.marigold.MarigoldDepthOutput]]

[Source](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_depth.py#L83)

Output class for Marigold monocular depth prediction pipeline.

**Parameters:**

prediction (`np.ndarray`, `torch.Tensor`) : Predicted depth maps with values in the range [0, 1]. The shape is `numimages × 1 × height × width` for `torch.Tensor` or `numimages × height × width × 1` for `np.ndarray`.

uncertainty (`None`, `np.ndarray`, `torch.Tensor`) : Uncertainty maps computed from the ensemble, with values in the range [0, 1]. The shape is `numimages × 1 × height × width` for `torch.Tensor` or `numimages × height × width × 1` for `np.ndarray`.

latent (`None`, `torch.Tensor`) : Latent features corresponding to the predictions, compatible with the `latents` argument of the pipeline. The shape is `numimages * numensemble × 4 × latentheight × latentwidth`.

#### diffusers.pipelines.marigold.MarigoldImageProcessor.visualize_depth[[diffusers.pipelines.marigold.MarigoldImageProcessor.visualize_depth]]

[Source](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/marigold_image_processing.py#L387)

Visualizes depth maps, such as predictions of the `MarigoldDepthPipeline`.

Returns: `List[PIL.Image.Image]` with depth maps visualization.

**Parameters:**

depth (`Union[PIL.Image.Image, np.ndarray, torch.Tensor, List[PIL.Image.Image], List[np.ndarray], : List[torch.Tensor]]`): Depth maps.

val_min (`float`, *optional*, defaults to `0.0`) : Minimum value of the visualized depth range.

val_max (`float`, *optional*, defaults to `1.0`) : Maximum value of the visualized depth range.

color_map (`str`, *optional*, defaults to `"Spectral"`) : Color map used to convert a single-channel depth prediction into colored representation.

## Marigold Normals Estimation API[[diffusers.MarigoldNormalsPipeline]]
#### diffusers.MarigoldNormalsPipeline[[diffusers.MarigoldNormalsPipeline]]

[Source](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_normals.py#L99)

Pipeline for monocular normals estimation using the Marigold method: https://marigoldmonodepth.github.io.

This model inherits from [DiffusionPipeline](/docs/diffusers/main/en/api/pipelines/overview#diffusers.DiffusionPipeline). Check the superclass documentation for the generic methods the
library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)

__call__diffusers.MarigoldNormalsPipeline.__call__https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_normals.py#L332[{"name": "image", "val": ": typing.Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, typing.List[PIL.Image.Image], typing.List[numpy.ndarray], typing.List[torch.Tensor]]"}, {"name": "num_inference_steps", "val": ": typing.Optional[int] = None"}, {"name": "ensemble_size", "val": ": int = 1"}, {"name": "processing_resolution", "val": ": typing.Optional[int] = None"}, {"name": "match_input_resolution", "val": ": bool = True"}, {"name": "resample_method_input", "val": ": str = 'bilinear'"}, {"name": "resample_method_output", "val": ": str = 'bilinear'"}, {"name": "batch_size", "val": ": int = 1"}, {"name": "ensembling_kwargs", "val": ": typing.Optional[typing.Dict[str, typing.Any]] = None"}, {"name": "latents", "val": ": typing.Union[torch.Tensor, typing.List[torch.Tensor], NoneType] = None"}, {"name": "generator", "val": ": typing.Union[torch._C.Generator, typing.List[torch._C.Generator], NoneType] = None"}, {"name": "output_type", "val": ": str = 'np'"}, {"name": "output_uncertainty", "val": ": bool = False"}, {"name": "output_latent", "val": ": bool = False"}, {"name": "return_dict", "val": ": bool = True"}]- **image** (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`), --
  `List[torch.Tensor]`: An input image or images used as an input for the normals estimation task. For
  arrays and tensors, the expected value range is between `[0, 1]`. Passing a batch of images is possible
  by providing a four-dimensional array or a tensor. Additionally, a list of images of two- or
  three-dimensional arrays or tensors can be passed. In the latter case, all list elements must have the
  same width and height.
- **num_inference_steps** (`int`, *optional*, defaults to `None`) --
  Number of denoising diffusion steps during inference. The default value `None` results in automatic
  selection.
- **ensemble_size** (`int`, defaults to `1`) --
  Number of ensemble predictions. Higher values result in measurable improvements and visual degradation.
- **processing_resolution** (`int`, *optional*, defaults to `None`) --
  Effective processing resolution. When set to `0`, matches the larger input image dimension. This
  produces crisper predictions, but may also lead to the overall loss of global context. The default
  value `None` resolves to the optimal value from the model config.
- **match_input_resolution** (`bool`, *optional*, defaults to `True`) --
  When enabled, the output prediction is resized to match the input dimensions. When disabled, the longer
  side of the output will equal to `processing_resolution`.
- **resample_method_input** (`str`, *optional*, defaults to `"bilinear"`) --
  Resampling method used to resize input images to `processing_resolution`. The accepted values are:
  `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`.
- **resample_method_output** (`str`, *optional*, defaults to `"bilinear"`) --
  Resampling method used to resize output predictions to match the input resolution. The accepted values
  are `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`.
- **batch_size** (`int`, *optional*, defaults to `1`) --
  Batch size; only matters when setting `ensemble_size` or passing a tensor of images.
- **ensembling_kwargs** (`dict`, *optional*, defaults to `None`) --
  Extra dictionary with arguments for precise ensembling control. The following options are available:
  - reduction (`str`, *optional*, defaults to `"closest"`): Defines the ensembling function applied in
    every pixel location, can be either `"closest"` or `"mean"`.
- **latents** (`torch.Tensor`, *optional*, defaults to `None`) --
  Latent noise tensors to replace the random initialization. These can be taken from the previous
  function call's output.
- **generator** (`torch.Generator`, or `List[torch.Generator]`, *optional*, defaults to `None`) --
  Random number generator object to ensure reproducibility.
- **output_type** (`str`, *optional*, defaults to `"np"`) --
  Preferred format of the output's `prediction` and the optional `uncertainty` fields. The accepted
  values are: `"np"` (numpy array) or `"pt"` (torch tensor).
- **output_uncertainty** (`bool`, *optional*, defaults to `False`) --
  When enabled, the output's `uncertainty` field contains the predictive uncertainty map, provided that
  the `ensemble_size` argument is set to a value above 2.
- **output_latent** (`bool`, *optional*, defaults to `False`) --
  When enabled, the output's `latent` field contains the latent codes corresponding to the predictions
  within the ensemble. These codes can be saved, modified, and used for subsequent calls with the
  `latents` argument.
- **return_dict** (`bool`, *optional*, defaults to `True`) --
  Whether or not to return a [MarigoldNormalsOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldNormalsOutput) instead of a plain tuple.0[MarigoldNormalsOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldNormalsOutput) or `tuple`If `return_dict` is `True`, [MarigoldNormalsOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldNormalsOutput) is returned, otherwise a
`tuple` is returned where the first element is the prediction, the second element is the uncertainty
(or `None`), and the third is the latent (or `None`).

Function invoked when calling the pipeline.

Examples:
```py
>>> import diffusers
>>> import torch

>>> pipe = diffusers.MarigoldNormalsPipeline.from_pretrained(
...     "prs-eth/marigold-normals-v1-1", variant="fp16", torch_dtype=torch.float16
... ).to("cuda")

>>> image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")
>>> normals = pipe(image)

>>> vis = pipe.image_processor.visualize_normals(normals.prediction)
>>> vis[0].save("einstein_normals.png")
```

**Parameters:**

unet (`UNet2DConditionModel`) : Conditional U-Net to denoise the normals latent, conditioned on image latent.

vae (`AutoencoderKL`) : Variational Auto-Encoder (VAE) Model to encode and decode images and predictions to and from latent representations.

scheduler (`DDIMScheduler` or `LCMScheduler`) : A scheduler to be used in combination with `unet` to denoise the encoded image latents.

text_encoder (`CLIPTextModel`) : Text-encoder, for empty text embedding.

tokenizer (`CLIPTokenizer`) : CLIP tokenizer.

prediction_type (`str`, *optional*) : Type of predictions made by the model.

use_full_z_range (`bool`, *optional*) : Whether the normals predicted by this model utilize the full range of the Z dimension, or only its positive half.

default_denoising_steps (`int`, *optional*) : The minimum number of denoising diffusion steps that are required to produce a prediction of reasonable quality with the given model. This value must be set in the model config. When the pipeline is called without explicitly setting `num_inference_steps`, the default value is used. This is required to ensure reasonable results with various model flavors compatible with the pipeline, such as those relying on very short denoising schedules (`LCMScheduler`) and those with full diffusion schedules (`DDIMScheduler`).

default_processing_resolution (`int`, *optional*) : The recommended value of the `processing_resolution` parameter of the pipeline. This value must be set in the model config. When the pipeline is called without explicitly setting `processing_resolution`, the default value is used. This is required to ensure reasonable results with various model flavors trained with varying optimal processing resolution values.

**Returns:**

`[MarigoldNormalsOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldNormalsOutput) or `tuple``

If `return_dict` is `True`, [MarigoldNormalsOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldNormalsOutput) is returned, otherwise a
`tuple` is returned where the first element is the prediction, the second element is the uncertainty
(or `None`), and the third is the latent (or `None`).

#### diffusers.pipelines.marigold.MarigoldNormalsOutput[[diffusers.pipelines.marigold.MarigoldNormalsOutput]]

[Source](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_normals.py#L78)

Output class for Marigold monocular normals prediction pipeline.

**Parameters:**

prediction (`np.ndarray`, `torch.Tensor`) : Predicted normals with values in the range [-1, 1]. The shape is `numimages × 3 × height × width` for `torch.Tensor` or `numimages × height × width × 3` for `np.ndarray`.

uncertainty (`None`, `np.ndarray`, `torch.Tensor`) : Uncertainty maps computed from the ensemble, with values in the range [0, 1]. The shape is `numimages × 1 × height × width` for `torch.Tensor` or `numimages × height × width × 1` for `np.ndarray`.

latent (`None`, `torch.Tensor`) : Latent features corresponding to the predictions, compatible with the `latents` argument of the pipeline. The shape is `numimages * numensemble × 4 × latentheight × latentwidth`.

#### diffusers.pipelines.marigold.MarigoldImageProcessor.visualize_normals[[diffusers.pipelines.marigold.MarigoldImageProcessor.visualize_normals]]

[Source](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/marigold_image_processing.py#L488)

Visualizes surface normals, such as predictions of the `MarigoldNormalsPipeline`.

Returns: `List[PIL.Image.Image]` with surface normals visualization.

**Parameters:**

normals (`Union[np.ndarray, torch.Tensor, List[np.ndarray], List[torch.Tensor]]`) : Surface normals.

flip_x (`bool`, *optional*, defaults to `False`) : Flips the X axis of the normals frame of reference. Default direction is right.

flip_y (`bool`, *optional*, defaults to `False`) : Flips the Y axis of the normals frame of reference. Default direction is top.

flip_z (`bool`, *optional*, defaults to `False`) : Flips the Z axis of the normals frame of reference. Default direction is facing the observer.

## Marigold Intrinsic Image Decomposition API[[diffusers.MarigoldIntrinsicsPipeline]]

#### diffusers.MarigoldIntrinsicsPipeline[[diffusers.MarigoldIntrinsicsPipeline]]

[Source](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_intrinsics.py#L120)

Pipeline for Intrinsic Image Decomposition (IID) using the Marigold method:
https://marigoldcomputervision.github.io.

This model inherits from [DiffusionPipeline](/docs/diffusers/main/en/api/pipelines/overview#diffusers.DiffusionPipeline). Check the superclass documentation for the generic methods the
library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)

__call__diffusers.MarigoldIntrinsicsPipeline.__call__https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_intrinsics.py#L359[{"name": "image", "val": ": typing.Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, typing.List[PIL.Image.Image], typing.List[numpy.ndarray], typing.List[torch.Tensor]]"}, {"name": "num_inference_steps", "val": ": typing.Optional[int] = None"}, {"name": "ensemble_size", "val": ": int = 1"}, {"name": "processing_resolution", "val": ": typing.Optional[int] = None"}, {"name": "match_input_resolution", "val": ": bool = True"}, {"name": "resample_method_input", "val": ": str = 'bilinear'"}, {"name": "resample_method_output", "val": ": str = 'bilinear'"}, {"name": "batch_size", "val": ": int = 1"}, {"name": "ensembling_kwargs", "val": ": typing.Optional[typing.Dict[str, typing.Any]] = None"}, {"name": "latents", "val": ": typing.Union[torch.Tensor, typing.List[torch.Tensor], NoneType] = None"}, {"name": "generator", "val": ": typing.Union[torch._C.Generator, typing.List[torch._C.Generator], NoneType] = None"}, {"name": "output_type", "val": ": str = 'np'"}, {"name": "output_uncertainty", "val": ": bool = False"}, {"name": "output_latent", "val": ": bool = False"}, {"name": "return_dict", "val": ": bool = True"}]- **image** (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`), --
  `List[torch.Tensor]`: An input image or images used as an input for the intrinsic decomposition task.
  For arrays and tensors, the expected value range is between `[0, 1]`. Passing a batch of images is
  possible by providing a four-dimensional array or a tensor. Additionally, a list of images of two- or
  three-dimensional arrays or tensors can be passed. In the latter case, all list elements must have the
  same width and height.
- **num_inference_steps** (`int`, *optional*, defaults to `None`) --
  Number of denoising diffusion steps during inference. The default value `None` results in automatic
  selection.
- **ensemble_size** (`int`, defaults to `1`) --
  Number of ensemble predictions. Higher values result in measurable improvements and visual degradation.
- **processing_resolution** (`int`, *optional*, defaults to `None`) --
  Effective processing resolution. When set to `0`, matches the larger input image dimension. This
  produces crisper predictions, but may also lead to the overall loss of global context. The default
  value `None` resolves to the optimal value from the model config.
- **match_input_resolution** (`bool`, *optional*, defaults to `True`) --
  When enabled, the output prediction is resized to match the input dimensions. When disabled, the longer
  side of the output will equal to `processing_resolution`.
- **resample_method_input** (`str`, *optional*, defaults to `"bilinear"`) --
  Resampling method used to resize input images to `processing_resolution`. The accepted values are:
  `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`.
- **resample_method_output** (`str`, *optional*, defaults to `"bilinear"`) --
  Resampling method used to resize output predictions to match the input resolution. The accepted values
  are `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`.
- **batch_size** (`int`, *optional*, defaults to `1`) --
  Batch size; only matters when setting `ensemble_size` or passing a tensor of images.
- **ensembling_kwargs** (`dict`, *optional*, defaults to `None`) --
  Extra dictionary with arguments for precise ensembling control. The following options are available:
  - reduction (`str`, *optional*, defaults to `"median"`): Defines the ensembling function applied in
    every pixel location, can be either `"median"` or `"mean"`.
- **latents** (`torch.Tensor`, *optional*, defaults to `None`) --
  Latent noise tensors to replace the random initialization. These can be taken from the previous
  function call's output.
- **generator** (`torch.Generator`, or `List[torch.Generator]`, *optional*, defaults to `None`) --
  Random number generator object to ensure reproducibility.
- **output_type** (`str`, *optional*, defaults to `"np"`) --
  Preferred format of the output's `prediction` and the optional `uncertainty` fields. The accepted
  values are: `"np"` (numpy array) or `"pt"` (torch tensor).
- **output_uncertainty** (`bool`, *optional*, defaults to `False`) --
  When enabled, the output's `uncertainty` field contains the predictive uncertainty map, provided that
  the `ensemble_size` argument is set to a value above 2.
- **output_latent** (`bool`, *optional*, defaults to `False`) --
  When enabled, the output's `latent` field contains the latent codes corresponding to the predictions
  within the ensemble. These codes can be saved, modified, and used for subsequent calls with the
  `latents` argument.
- **return_dict** (`bool`, *optional*, defaults to `True`) --
  Whether or not to return a [MarigoldIntrinsicsOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldIntrinsicsOutput) instead of a plain tuple.0[MarigoldIntrinsicsOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldIntrinsicsOutput) or `tuple`If `return_dict` is `True`, [MarigoldIntrinsicsOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldIntrinsicsOutput) is returned, otherwise a
`tuple` is returned where the first element is the prediction, the second element is the uncertainty
(or `None`), and the third is the latent (or `None`).

Function invoked when calling the pipeline.

Examples:
```py
>>> import diffusers
>>> import torch

>>> pipe = diffusers.MarigoldIntrinsicsPipeline.from_pretrained(
...     "prs-eth/marigold-iid-appearance-v1-1", variant="fp16", torch_dtype=torch.float16
... ).to("cuda")

>>> image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")
>>> intrinsics = pipe(image)

>>> vis = pipe.image_processor.visualize_intrinsics(intrinsics.prediction, pipe.target_properties)
>>> vis[0]["albedo"].save("einstein_albedo.png")
>>> vis[0]["roughness"].save("einstein_roughness.png")
>>> vis[0]["metallicity"].save("einstein_metallicity.png")
```

```py
>>> import diffusers
>>> import torch

>>> pipe = diffusers.MarigoldIntrinsicsPipeline.from_pretrained(
...     "prs-eth/marigold-iid-lighting-v1-1", variant="fp16", torch_dtype=torch.float16
... ).to("cuda")

>>> image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg")
>>> intrinsics = pipe(image)

>>> vis = pipe.image_processor.visualize_intrinsics(intrinsics.prediction, pipe.target_properties)
>>> vis[0]["albedo"].save("einstein_albedo.png")
>>> vis[0]["shading"].save("einstein_shading.png")
>>> vis[0]["residual"].save("einstein_residual.png")
```

**Parameters:**

unet (`UNet2DConditionModel`) : Conditional U-Net to denoise the targets latent, conditioned on image latent.

vae (`AutoencoderKL`) : Variational Auto-Encoder (VAE) Model to encode and decode images and predictions to and from latent representations.

scheduler (`DDIMScheduler` or `LCMScheduler`) : A scheduler to be used in combination with `unet` to denoise the encoded image latents.

text_encoder (`CLIPTextModel`) : Text-encoder, for empty text embedding.

tokenizer (`CLIPTokenizer`) : CLIP tokenizer.

prediction_type (`str`, *optional*) : Type of predictions made by the model.

target_properties (`Dict[str, Any]`, *optional*) : Properties of the predicted modalities, such as `target_names`, a `List[str]` used to define the number, order and names of the predicted modalities, and any other metadata that may be required to interpret the predictions.

default_denoising_steps (`int`, *optional*) : The minimum number of denoising diffusion steps that are required to produce a prediction of reasonable quality with the given model. This value must be set in the model config. When the pipeline is called without explicitly setting `num_inference_steps`, the default value is used. This is required to ensure reasonable results with various model flavors compatible with the pipeline, such as those relying on very short denoising schedules (`LCMScheduler`) and those with full diffusion schedules (`DDIMScheduler`).

default_processing_resolution (`int`, *optional*) : The recommended value of the `processing_resolution` parameter of the pipeline. This value must be set in the model config. When the pipeline is called without explicitly setting `processing_resolution`, the default value is used. This is required to ensure reasonable results with various model flavors trained with varying optimal processing resolution values.

**Returns:**

`[MarigoldIntrinsicsOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldIntrinsicsOutput) or `tuple``

If `return_dict` is `True`, [MarigoldIntrinsicsOutput](/docs/diffusers/main/en/api/pipelines/marigold#diffusers.pipelines.marigold.MarigoldIntrinsicsOutput) is returned, otherwise a
`tuple` is returned where the first element is the prediction, the second element is the uncertainty
(or `None`), and the third is the latent (or `None`).

#### diffusers.pipelines.marigold.MarigoldIntrinsicsOutput[[diffusers.pipelines.marigold.MarigoldIntrinsicsOutput]]

[Source](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/pipeline_marigold_intrinsics.py#L96)

Output class for Marigold Intrinsic Image Decomposition pipeline.

**Parameters:**

prediction (`np.ndarray`, `torch.Tensor`) : Predicted image intrinsics with values in the range [0, 1]. The shape is `(numimages * numtargets) × 3 × height × width` for `torch.Tensor` or `(numimages * numtargets) × height × width × 3` for `np.ndarray`, where `numtargets` corresponds to the number of predicted target modalities of the intrinsic image decomposition.

uncertainty (`None`, `np.ndarray`, `torch.Tensor`) : Uncertainty maps computed from the ensemble, with values in the range [0, 1]. The shape is `(numimages * numtargets) × 3 × height × width` for `torch.Tensor` or `(numimages * numtargets) × height × width × 3` for `np.ndarray`.

latent (`None`, `torch.Tensor`) : Latent features corresponding to the predictions, compatible with the `latents` argument of the pipeline. The shape is `(numimages * numensemble) × (numtargets * 4) × latentheight × latentwidth`.

#### diffusers.pipelines.marigold.MarigoldImageProcessor.visualize_intrinsics[[diffusers.pipelines.marigold.MarigoldImageProcessor.visualize_intrinsics]]

[Source](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/marigold/marigold_image_processing.py#L549)

Visualizes intrinsic image decomposition, such as predictions of the `MarigoldIntrinsicsPipeline`.

Returns: `List[Dict[str, PIL.Image.Image]]` with intrinsic image decomposition visualization.

**Parameters:**

prediction (`Union[np.ndarray, torch.Tensor, List[np.ndarray], List[torch.Tensor]]`) : Intrinsic image decomposition.

target_properties (`Dict[str, Any]`) : Decomposition properties. Expected entries: `target_names: List[str]` and a dictionary with keys `prediction_space: str`, `sub_target_names: List[Union[str, Null]]` (must have 3 entries, null for missing modalities), `up_to_scale: bool`, one for each target and sub-target.

color_map (`Union[str, Dict[str, str]]`, *optional*, defaults to `"Spectral"`) : Color map used to convert a single-channel predictions into colored representations. When a dictionary is passed, each modality can be colored with its own color map.

