Metadata-Version: 2.4
Name: pixeloe
Version: 1.0.0
Summary: Detail-Oriented Pixelization based on Contrast-Aware Outline Expansion.
Author-email: "Shih-Ying Yeh(KohakuBlueLeaf)" <apolloyeh0123@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/KohakuBlueleaf/PixelOE
Project-URL: Repository, https://github.com/KohakuBlueleaf/PixelOE
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: opencv-python
Requires-Dist: numpy
Requires-Dist: pillow
Requires-Dist: torch
Requires-Dist: torchvision
Requires-Dist: kornia
Requires-Dist: slangpy>=0.43.1; (sys_platform == "win32" and platform_machine == "AMD64") or (sys_platform == "linux" and (platform_machine == "x86_64" or platform_machine == "aarch64")) or (sys_platform == "darwin" and platform_machine == "arm64" and platform_release >= "25")
Provides-Extra: slang
Requires-Dist: slangpy>=0.43.1; extra == "slang"
Provides-Extra: gradio
Requires-Dist: gradio>=4.0.0; extra == "gradio"
Requires-Dist: toml; extra == "gradio"
Provides-Extra: standalone
Requires-Dist: gradio>=4.0.0; extra == "standalone"
Requires-Dist: toml; extra == "standalone"
Requires-Dist: pywebview; extra == "standalone"
Provides-Extra: all
Requires-Dist: gradio>=4.0.0; extra == "all"
Requires-Dist: toml; extra == "all"
Requires-Dist: pywebview; extra == "all"
Provides-Extra: dev
Requires-Dist: black; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# PixelOE: Detail-Oriented ***Pixel***ization based on Contrast-Aware ***O***utline ***E***xpansion.

**Create stunning pixel art from high-resolution images without AI or complex networks.**

[![PyPI version](https://badge.fury.io/py/pixeloe.svg)](https://badge.fury.io/py/pixeloe) [![License](https://img.shields.io/github/license/KohakuBlueleaf/PixelOE)](https://github.com/KohakuBlueleaf/PixelOE/blob/main/LICENSE)

![](img/snow-leopard-pixel-lg-256c-d.webp)

PixelOE is a Python library that generates high-quality pixel art from standard images using a novel, **contrast-aware** approach. This method focuses on preserving crucial visual details by first **expanding outlines** of key features, then applying an intelligent downscaling to the target resolution.

**Key Features:**

* **No AI/NN Required:**  Utilizes classical image processing methods for efficient and predictable results.
* **Detail-Oriented:**  Emphasizes preservation of fine details and sharp edges.
* **Contrast-Aware:**  Adapts downsampling based on local image contrast.
* **Outline Expansion:**  Broadens important features before downscaling to prevent loss of detail.
* **Flexible Downscale Modes:** Offers various downsampling methods for different styles (center, contrast, k-centroid, bicubic, nearest).
* **Color Palette Optimization:** Option to quantize color palettes for a classic pixel art aesthetic.
* **Fast Pure Pytorch implementation** Achieving over 180img/sec (bs1) on RTX4090 with 1920x1080 input and 480x270 output
* **GPU Free**: All the core logic can be used without GPU.
* **Slang Compute-Shader Backend:** The whole pipeline is also implemented as [Slang](https://shader-slang.org) compute kernels (CUDA / Vulkan / D3D12 / native CPU). It gives the same results as the torch pipeline and is 5-10x faster on GPU and ~3x faster on CPU. `pixelize()` picks the fastest backend that works on your machine automatically.

## Usage

### ComfyUI

![1739714420116](image/README/1739714420116.png)

We provide ComfyUI Custom Node implementation in this repository directly, you just need to clone this repo into the custom_node directory and it will work!

```
cd custom_nodes
git clone https://github.com/KohakuBlueleaf/PixelOE
```

There are 3 nodes are provided in this extensions:

* PixelOE: The pixelization Nodes
* OutlineExpansion: Only utilize the OutlineExpansion algorithm
* PreResize: Resize based on target pixel image resolution and each pixel's size.

### Installation

To utilize this package in your code or utilize its CLI implementation, you need to install it with pip first:

```bash
pip install pixeloe
```

This also installs [slangpy](https://github.com/shader-slang/slangpy) (the Slang backend) on Windows x64, Linux x64/arm64 and macOS 26+ on Apple Silicon. On other platforms PixelOE runs the torch pipeline.

### Command Line Interface (CLI)

**Note**: CLI in PixelOE is currently using legacy API which will be deprecated in near future.

The `pixeloe` package provides two primary commands:

* `pixeloe.pixelize`: Full pixelization process
* `pixeloe.outline`: Outline expansion

To view command usage, use the `--help` flag:

```bash
pixeloe.pixelize --help
```

```
usage: pixeloe.pixelize [-h] [--output_img OUTPUT_IMG] [--mode {center,contrast,k-centroid,bicubic,nearest}] [--target_size TARGET_SIZE]
                        [--patch_size PATCH_SIZE] [--thickness THICKNESS] [--no_color_matching] [--contrast CONTRAST]
                        [--saturation SATURATION] [--colors COLORS] [--no_upscale] [--no_downscale]
                        input_img

positional arguments:
  input_img

options:
  -h, --help            show this help message and exit
  --output_img OUTPUT_IMG, -O OUTPUT_IMG
  --mode {center,contrast,k-centroid,bicubic,nearest}, -M {center,contrast,k-centroid,bicubic,nearest}
  --target_size TARGET_SIZE, -S TARGET_SIZE
  --patch_size PATCH_SIZE, -P PATCH_SIZE
  --thickness THICKNESS, -T THICKNESS
  --no_color_matching
  --contrast CONTRAST
  --saturation SATURATION
  --colors COLORS
  --no_upscale
  --no_downscale
```

**Example:**

```bash
pixeloe.pixelize img/test.webp --output_img img/test2.webp --target_size 256 --patch_size 8
```

### Gradio

![1737572469521](image/README/1737572469521.png)
If you want to an interface to easily test it, you can install it with gradio:

```bash
pip install -r requirements-gradio.txt
```

Then run the demo:

```bash
python ./client/demo-gr.py
```

After startup, the app should be available at http://localhost:7860
    
**With Standalone Window (Desktop app):**

```bash
pip install -r requirements-standalone.txt   # With standalone window
```

Note that the standalone window may have additional requirements to run.
For example, on Linux, you must have either QT or GTK with Python extensions installed in order to use pywebview.

### Python API

You can integrate PixelOE directly into your Python/Pytorch projects:

legacy API (numpy/cv2 based, slow)

```python
import cv2
from pixeloe.pixelize import pixelize

img = cv2.imread("img/test.webp")
img = pixelize(img, target_size=256, patch_size=8)
cv2.imwrite("img/test2.webp", img)
```

torch API (Fast, GPU supported)

```python
import torch
from PIL import Image

from pixeloe.torch.pixelize import pixelize
from pixeloe.torch.utils import to_numpy, pre_resize

img = Image.open("img/snow-leopard.webp")
img = pre_resize(img, target_size=256, patch_size=4).cuda().half()
result = pixelize(img, pixel_size=4, thickness=3)
result_img = Image.fromarray(to_numpy(result)[0])
result_img.save("img/snow-leopard-pixel.webp")
```

Backend selection (torch API): `pixelize()` runs on the fastest [Slang backend](#slang-backend) that works for the input tensor's device, and falls back to the torch pipeline otherwise. To choose one explicitly, pass `backend=` or set the `PIXELOE_BACKEND` environment variable:

```python
result = pixelize(img, pixel_size=4, thickness=3)                   # "auto"
result = pixelize(img, pixel_size=4, thickness=3, backend="torch")  # torch pipeline
result = pixelize(img, pixel_size=4, thickness=3, backend="vulkan") # cuda / vulkan / d3d12 / cpu
```

Slang API (backend contexts, extra options such as sliding-window outline statistics):

```python
from pixeloe.slang import pixelize, get_context

result = pixelize(img, pixel_size=4, thickness=3, context=get_context("cuda"))
result = pixelize(img, pixel_size=4, thickness=3, local_stats="sliding")
```

Slang CLI:

```bash
pixeloe.slang img/snow-leopard.webp out.png --backend cuda --target-size 256 --pixel-size 4
```

## Example

### Outline Expansion

| Original                 | Expanded                         |
| ------------------------ | -------------------------------- |
| ![](img/snow-leopard.webp) | ![](img/snow-leopard-oe-orig.webp) |

With this outline expansion method, you can obtain descent pixelization through some naive downsampling method:

| Expanded                    | Dowsampled                     |
| --------------------------- | ------------------------------ |
| ![](img/snow-leopard-oe.webp) | ![](img/snow-leopard-pixel.webp) |

### Pixelization

![house-grid](demo/house-grid.webp)

![horse-girl-grid](demo/horse-girl-grid.webp)

![dragon-girl-grid](demo/dragon-girl-grid.webp)

#### Use outline expansion to improve existing method

Use the outline expansion method can improve lot of existing pixelization method.
Even the Neural Network based method can also be improved:

Here is the example of using outline expansion to improve "Make Your Own Sprites: Aliasing-Aware and Cell-Controllable Pixelization"(SIGGRAPH Asia 2022)
![make-your-own-sprites](demo/house-make-your-own-sprites.webp)

## Algorithms Implemented in This Project

* Contrast-Awared Outline Expansion
* Contrast-Awared Down Sampling
* Deterministic K-Means with image-specialized initial centroids
* Weighed/Repeat K-Means for smoother results

## How It Works

The PixelOE algorithm has two key stages:

### 1. Contrast-Aware Outline Expansion:

This step is designed to make sure that fine details and high-contrast edges will survive after the downscaling step. Here is the process:

1. **Weight Map Generation:**

   * The input image is converted to grayscale.
   * Local median brightness is calculated.
   * Local max and min brightness values within each patch are found.
   * "Bright" and "dark" distances are calculated using local max/min and median.
   * Two weights are combined:
     * The first weight is prioritize brighter details in darker median area
     * The second weight is based on the distance between the brighter/darker details.
   * The combined weight is normalized to 0-1.
2. **Selective Morphological Operations:**

   * The input image is eroded to shrink bright regions
   * The input image is dilated to expand bright regions
   * The eroded and dilated results are blended together based on the generated weight map.
   * Morphological closing and opening are then applied to clean up edge artifacts.

| Dilation                        | Erosion                        |
| ------------------------------- | ------------------------------ |
| ![](img/snow-leopard-dilate.webp) | ![](img/snow-leopard-erode.webp) |

| Blended                     | Weight                     |
| --------------------------- | -------------------------- |
| ![](img/snow-leopard-oe.webp) | ![](img/snow-leopard-w.webp) |

### 2. Contrast-Based Downsampling:

This method reduces the image resolution while maintaining important luminance details.

1. **LAB Color Space Conversion:** The image is converted to the LAB color space to process luminance (L) and color (A, B) channels separately.
2. **Luminance Channel (L) Processing:**
   * A sliding window with `find_pixel` function is used, and each patch is processed independently.
   * Inside each patch, the center pixel is selected based on its relationship with the median, mean, min, max value within the patch.
   * If a patch has skewed distribution on low value, the minimum value will be selected to keep the dark detail and vice versa.
   * Otherwise, keep the center value.
3. **Color Channel (A and B) Processing:** A simple median filter is applied to the A and B channels
4. **Convert back to RGB** The processed LAB channels are combined and converted back to the RGB color space.

By adaptively selecting the most representative pixel for each local area, the downscaling method preserves important luminance details and edges to maintain the artistic style in pixel art.

### Optional Enhancements

* **Color Palette Optimization:** You can reduce the number of colors using k-means clustering or maxcover method for a more classic pixel art look.
* **Color Matching:** Optionally transfer the color palette from the original image.

## Slang Backend

`pixeloe.slang` implements the full pipeline as Slang compute kernels, written against the torch pipeline and tested against it. It covers outline expansion, every downscale mode, color matching, sharpening, k-means quantization and ordered / error-diffusion dithering. The same shader source runs on:

| backend | device | notes |
| --- | --- | --- |
| `cuda` | NVIDIA GPU | shares torch's CUDA context and stream, zero-copy; compiles through NVRTC (needs the CUDA toolkit) |
| `vulkan` / `d3d12` | any GPU | CUDA-shared buffers on NVIDIA; other adapters (e.g. Intel Arc: `"d3d12:B50"`) go through host staging |
| `cpu` | CPU | Slang-generated C++ kernels on an OpenMP thread pool (Windows, needs MSVC) |

In `"auto"` mode, CUDA tensors try `cuda` → `vulkan` → `d3d12` and CPU tensors try `cpu`. Each backend is checked once per process with a small test image. If none works, the torch pipeline runs.

How it matches the torch pipeline:

* Order-independent integer (fixed-point) sums make k-means and k-centroid deterministic on every backend.
* Exact median selection, and the torch pipeline's padding and rounding behavior.
* Differences are limited to floating-point rounding, plus k-centroid blocks whose two clusters are the same size. There the Slang backend deterministically picks the first cluster.

Speed-up over the fastest torch variant (fp16 / fp32, eager / `torch.compile`) on an RTX 4090, over 5 sizes × 18 settings:

| Slang backend | median | range |
| --- | --- | --- |
| cuda | 8.5x | 2.2x - 20.6x |
| vulkan | 7.0x | 2.2x - 20.9x |
| d3d12 | 6.6x | 1.7x - 18.2x |
| cpu (vs torch CPU) | 3.0x | 1.8x - 6.2x |

![slang vs torch](benchmarks/slang_vs_torch/figures/fig_overview.png)

Full benchmark scripts and figures (scaling, memory, CPU thread count, other GPUs) are in [benchmarks/slang_vs_torch](benchmarks/slang_vs_torch).

## Acknowledgement

* Astropulse
  * k-centroid downscaling algorithm.
* Claude 3 opus:
  * Convert some matlab code to python.
* Gemini 2.0 Flash
  * Refine this README

## Citation

```bibtex
@misc{PixelOE,
    title={Detail-Oriented Pixelization based on Contrast-Aware Outline Expansion.}, 
    author={Shin-Ying Yeh},
    year={2024},
    month={March},
    howpublished=\url{https://github.com/KohakuBlueleaf/PixelOE},
}
```

## Star History

<a href="https://star-history.com/#KohakuBlueleaf/PixelOE&Date">
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=KohakuBlueleaf/PixelOE&type=Date&theme=dark" />
   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=KohakuBlueleaf/PixelOE&type=Date" />
   <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=KohakuBlueleaf/PixelOE&type=Date" />
 </picture>
</a>
