Palette Matching¶

In [1]:

from phomo import Pool, Master, Mosaic
from phomo.utils import rainbow_of_squares

from phomo import Pool, Master, Mosaic from phomo.utils import rainbow_of_squares

WARNING: CPU random generator seem to be failing, disabling hardware random number generation
WARNING: RDRND generated: 0xffffffff 0xffffffff 0xffffffff 0xffffffff
/home/lcoyle/.cache/pypoetry/virtualenvs/phomo-pX3Qwu7w-py3.12/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

In [2]:

import matplotlib.pyplot as plt
import numpy as np

import matplotlib.pyplot as plt import numpy as np

In [3]:

from pathlib import Path

from pathlib import Path

We are once again using the UTKFaces dataset. As you would expect, is contains a lot of flesh tone colours.

In [4]:

pool = Pool.from_dir("faces/", tile_size=(10, 10))

pool = Pool.from_dir("faces/", tile_size=(10, 10))

Loading tiles: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 9780/9780 [00:03<00:00, 2478.07it/s]

In [5]:

pool.plot()

pool.plot()

As a master image let's use Stary Night which mostly has blues/greens.

Due to the big difference in colour distribution between our master image and the tile images, we can't expect to get a great photo mosaic straight away.

In [6]:

master = Master.from_file("master.jpg")
master.img

master = Master.from_file("master.jpg") master.img

Out[6]:

In [7]:

master.plot()

master.plot()

In [8]:

def build_mosaic(master, pool):
    mosaic = Mosaic(master, pool)
    mosaic_img = mosaic.build(mosaic.d_matrix_cuda())
    distance = np.linalg.norm(mosaic.master.array.astype(float) - np.array(mosaic_img).astype(float))
    num_pixels = np.prod(mosaic.master.array.shape)
    loss =  distance / num_pixels
    print(f"Loss: {loss}")
    return mosaic_img

def build_mosaic(master, pool): mosaic = Mosaic(master, pool) mosaic_img = mosaic.build(mosaic.d_matrix_cuda()) distance = np.linalg.norm(mosaic.master.array.astype(float) - np.array(mosaic_img).astype(float)) num_pixels = np.prod(mosaic.master.array.shape) loss = distance / num_pixels print(f"Loss: {loss}") return mosaic_img

Let's see what the photo mosaic looks like:

In [9]:

build_mosaic(master, pool)

build_mosaic(master, pool)

Building mosaic: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1911/1911 [00:00<00:00, 324388.48it/s]

Loss: 0.06740223794924285

Out[9]:

Yeah not great, clearly the flesh tones don't really match well with the blue/green Stary Night.

We have a few ways to get around this.

Equalizing the colour distributions¶

We can spread the colour distributions of both the Master and the Pool to cover the full RGB space.

This will modify the appearance of the master and pool images but lead to better mosaics.

In [10]:

master_eq = master.equalize()
pool_eq = pool.equalize()
master_eq.img

master_eq = master.equalize() pool_eq = pool.equalize() master_eq.img

Out[10]:

In [11]:

pool_eq.plot();

pool_eq.plot();

In [12]:

master_eq.plot();

master_eq.plot();

In [13]:

build_mosaic(master_eq, pool_eq)

build_mosaic(master_eq, pool_eq)

Building mosaic: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1911/1911 [00:00<00:00, 385518.49it/s]

Loss: 0.06615275570074403

Out[13]:

This is already much better, but, seeing as we modified the colour distribution of both the Master and the Pool, neither the tile images or the master image are representative of the original images.

This is made obvious, by the eerily white faces which make up the clouds of the night sky.

Palette transfer¶

Master -> Pool¶

Another approach is to cast the Master's colour distribution to match the tile pool.

This will modify the master image to be closer to the colours in the tile images while leaving the tile images unchanged.

In [14]:

matched_master = master.match(pool)
matched_master.img

matched_master = master.match(pool) matched_master.img

Out[14]:

In [15]:

matched_master.plot();

matched_master.plot();

In [16]:

build_mosaic(matched_master, pool)

build_mosaic(matched_master, pool)

Building mosaic: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1911/1911 [00:00<00:00, 388226.05it/s]

Loss: 0.04494942425162315

Out[16]:

Ok so we get a nicer flesh toned Stary Night, where the tile images are the unmodified images we provided.

Pool -> Master¶

We can also do the opposite, and modify the tile image pool to better match the master image.

In [17]:

pool_matched = pool.match(master)

pool_matched = pool.match(master)

In [18]:

pool_matched.plot();

pool_matched.plot();

In [19]:

build_mosaic(master, pool_matched)

build_mosaic(master, pool_matched)

Building mosaic: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1911/1911 [00:00<00:00, 375283.97it/s]

Loss: 0.03631232331544263

Out[19]:

This is the mosaic closest to the original Stary Night, but the tile images have been severly shifted towards the blue/green.