---
title: "How do you grade a forged painting? Scoring a hand-painted copy without comparing pixels"
description: "How the paint rating system in Counterfeit Gallery works: four components, a geometric mean, the bugs that taught me the most, and a benchmark I could run without opening the game."
date: "2026-10-12T10:21:05.226Z"
author: "Zsombor Rózsa"
tags: ["game-dev", "unity", "scoring", "counterfeit-gallery"]
canonical: "https://codescrambler.eu/blog/how-do-you-grade-a-forged-painting"
lang: "en"
---

In *Counterfeit Gallery* you run a shop that sells forgeries, and every forgery is a painting you copied by hand, with a finger, a mouse or a stylus, from a famous work. That means the game has to look at a wobbly, imperfect copy and decide how good it is. This is how that grading works, and mostly how it went wrong along the way.

![The result screen for a copy of The Starry Night: composition 100%, colour 93%, shape 90%, detail 79%, overall 90%.](/images/blog/cg-scoring/game-appraisal.webp)

## The test I set

The bar I wrote down was simple: **would a normal person look at their result and agree that a better copy deserved a better score?**

That is a question about *ranking*, not about similarity. A scorer can be mathematically accurate and still feel unfair if it ranks a worse-looking copy above a better one. So I never tuned it against a similarity number. I wrote a benchmark of fake player attempts (blurry copies, shifted copies, washed-out copies, greyscale copies, scribbles, a flat fill of the average colour), ranked them by hand *before* running anything, and counted how often the scorer disagreed with me.

Per-pixel difference was out from the start: it punishes a copy shifted by three pixels as hard as one in the wrong colours, and no player would accept that.

![Six versions of The Starry Night at 128 by 128: the original, a light blur, a 5% shift, a wobbly hand, a greyscale copy and a wrong-hue copy, with their benchmark scores.](/images/blog/cg-scoring/attempts-good-to-bad.webp)

*Scores are the benchmark average over all 57 reference paintings for that kind of attempt, shown on one painting. The images are the 128×128 versions the scorer actually sees.*

## What it measures

Both images are first reduced to a 128×128 working image, so the device's resolution never touches the score. Then four things are graded:

- **Composition:** are the darks and lights in the right places? A correlation of a 16×16 brightness map, so it ignores how bright the whole thing is and only cares about placement.
- **Colour:** average Lab colour difference over a grid, plus a second term for how much colour you used and whether your hues agree with the original.
- **Shape:** SSIM on a 64×64 map, checked at the best of a few small shifts, so a copy that sits slightly to one side is not treated like a bad one.
- **Detail:** Sobel edges turned into a 32×32 density map and compared, multiplied by a symmetric term for the total amount of edges. "No detail" and "scribbled everywhere" both lose.

The four numbers are the ones the player sees on the result screen, and the total has to agree with them or it feels broken.

```tikz
% libraries: positioning, arrows.meta, calc
\begin{tikzpicture}[
  font=\sffamily\small,
  box/.style={draw=teal!60!black, fill=teal!10, rounded corners=3pt, minimum height=9mm, align=center, inner sep=4pt},
  big/.style={box, fill=amber!25, draw=orange!70!black},
  >={Stealth[length=2.2mm]}, node distance=7mm]
\definecolor{amber}{HTML}{FBBF24}
\node[box] (p) {Player's\\copy};
\node[box, below=3mm of p] (r) {Reference\\painting};
\node[box, right=10mm of $(p)!0.5!(r)$] (w) {$128{\times}128$\\working image\\(margins masked)};
\node[box, right=10mm of w, yshift=21mm] (c1) {Composition};
\node[box, right=10mm of w, yshift=7mm]  (c2) {Colour};
\node[box, right=10mm of w, yshift=-7mm] (c3) {Shape};
\node[box, right=10mm of w, yshift=-21mm] (c4) {Detail};
\node[big, right=12mm of c2, yshift=-7mm] (g) {Weighted\\geometric mean};
\node[box, right=8mm of g] (s) {Difficulty\\curve};
\node[big, right=8mm of s] (o) {Score};
\draw[->] (p) -- (w); \draw[->] (r) -- (w);
\foreach \c in {c1,c2,c3,c4} {\draw[->] (w) -- (\c); \draw[->] (\c) -- (g);}
\draw[->] (g) -- (s); \draw[->] (s) -- (o);
\end{tikzpicture}
```

## The arithmetic mean was a mistake

I averaged the four components at first. Three of them (composition, shape, detail) are computed on brightness, so a copy can get three out of four right and still be the wrong painting. A perfect greyscale copy scored 0.84, above a genuinely good full-colour copy at 0.79.

```tikz
\begin{tikzpicture}[font=\sffamily\small, x=1cm, y=4.2cm]
\definecolor{teal}{HTML}{2DD4BF}\definecolor{amber}{HTML}{FBBF24}
\draw[->] (0,0) -- (0,1.02) node[above, font=\sffamily\footnotesize] {average score};
\foreach \v in {0.25,0.5,0.75,1.0} {\draw[gray!40] (0,\v) -- (6.2,\v); \node[left, font=\sffamily\footnotesize] at (0,\v) {\v};}
\fill[amber!80!black] (1,0) rectangle (2.2,0.84); \node[above] at (1.6,0.84) {0.84};
\fill[teal!70!black]  (3.6,0) rectangle (4.8,0.79); \node[above] at (4.2,0.79) {0.79};
\node[align=center, below, font=\sffamily\footnotesize] at (1.6,0) {perfect greyscale\\copy};
\node[align=center, below, font=\sffamily\footnotesize] at (4.2,0) {good full-colour\\copy};
\node[red!60!black, font=\sffamily\footnotesize\bfseries] at (3.0,0.95) {wrong order};
\end{tikzpicture}
```

A weighted **geometric** mean fixed it, because every component becomes something you have to earn. It also collapsed most of the cheats: a flat fill of the average colour dropped from 0.245 to 0.086. Each component gets a tiny floor before the log, so one zeroed component drags the score down hard without producing minus infinity, and a blank canvas still lands on exactly zero.

## Smaller things that didn't work

- **Colour difference alone is too forgiving.** With only Lab delta-E, a hue-swapped copy scored 0.92 and a greyscale copy 0.95, because the painting was already low in chroma and everything looked "close enough". The chroma half of the colour score brought those down to 0.33 and 0.37 on the colour axis.
- **An 8×8 composition grid told me nothing.** It sat at 1.000 for nearly every test case. 16×16 discriminates.
- **Structure without alignment.** A 5% offset hurt the shape score far more than a person would think it should. A small search over shifts fixed it, and scribbles gained nothing from it because they have no structure to line up.

## A curve that capped perfect copies

Every painting has a "how forgiving is this one" tolerance, and my first curve simply multiplied the score by it. Any painting with a tolerance below 1 could never reach 100%. I found it by resampling a reference onto the canvas and scoring it against itself: it came out at 0.897 to 0.959 instead of 1.000, tracking each painting's tolerance exactly.

The replacement has fixed endpoints. Zero maps to zero, one maps to one, and the tolerance only changes how forgiving the middle is:

$$\text{displayed} = \frac{t\,x}{1 + (t-1)\,x}, \qquad x = \text{raw}^{\,\gamma}$$

```tikz
\begin{tikzpicture}[font=\sffamily\small, scale=5.2]
\definecolor{teal}{HTML}{2DD4BF}\definecolor{amber}{HTML}{FBBF24}
\draw[gray!40] (0,0) grid[step=0.25] (1,1);
\draw[->] (0,0) -- (1.08,0) node[right, font=\sffamily\footnotesize] {raw};
\draw[->] (0,0) -- (0,1.08) node[above, font=\sffamily\footnotesize] {displayed};
\draw[red!60!black, thick, dashed, domain=0:1, samples=2] plot (\x, {min(1,0.9*\x)});
\foreach \t/\col in {0.9/teal!50!black, 1.32/blue!60!black} {
  \draw[\col, very thick, domain=0:1, samples=40] plot (\x, {\t*\x/(1+(\t-1)*\x)});
}
\draw[amber!80!black, very thick, domain=0:1, samples=2] plot (\x, \x);
\node[red!60!black, font=\sffamily\footnotesize, anchor=west] at (1.03,0.9) {old curve, tol 0.9};
\node[font=\sffamily\footnotesize, anchor=west, blue!60!black] at (1.03,1.0) {tol 1.32};
\node[font=\sffamily\footnotesize, anchor=west, teal!50!black] at (1.03,0.78) {tol 0.9};
\fill (1,1) circle (0.012); \fill[red!60!black] (1,0.9) circle (0.012);
\end{tikzpicture}
```

I also stopped guessing tolerances. Each one is solved by bisection against the real evaluator, aiming for a mean of 0.80 on three block-copy efforts.

## The bug I'm proudest of finding

My benchmark said everything was fine. A separate progression simulator, which scores the same painting recreated with bigger and bigger blocks, said otherwise. The score should fall smoothly as strokes get coarser. Instead, detail collapsed to exactly 0.000 at block sizes 9, 10 and 11 while the other components degraded gently. On *The Starry Night* the total went from **0.767 at 8 px to 0.341 at 9 px**. A 0.43 cliff for a one-pixel change in brush size is not defensible.

![The Starry Night as block mosaics at 8 px and 9 px.](/images/blog/cg-scoring/stroke-size-mosaics.webp)

I suspected the shift search first. Turning it off changed nothing. The cause was **aliasing**: a regular grid of strokes beats against the cell size of the edge-density map and produces a moiré pattern, which pushed the correlation to zero or below. The geometric mean then did exactly its job and wiped out the total.

```tikz
\begin{tikzpicture}[font=\sffamily\footnotesize, x=0.5mm]
\foreach \i in {0,...,17} {\draw[thick, blue!60!black] ({\i*9},1.5) -- ({\i*9},2.3);}
\foreach \i in {0,...,19} {\draw[thick, orange!80!black] ({\i*8},0.4) -- ({\i*8},1.2);}
\node[left, blue!60!black] at (-3,1.9) {9 px strokes};
\node[left, orange!80!black] at (-3,0.8) {8 px strokes};
\foreach \x in {0,72,144} {\draw[dashed, teal!60!black] (\x,-0.4) -- (\x,2.9); \node[below, teal!60!black] at (\x,-0.4) {in step};}
\foreach \x in {36,108} {\node[below, red!60!black] at (\x,-0.4) {out of step};}
\end{tikzpicture}
```

The fix was a 3×3 blur on both edge maps before comparing them, which is also closer to what the component should mean: *roughly where* the fine detail sits, not its exact pixel layout.

| | before | after |
|---|---|---|
| Starry Night, 8 px strokes | 0.767 | 0.767 |
| Starry Night, 9 px strokes | **0.341** | 0.701 |
| Starry Night, 11 px strokes | **0.322** | 0.551 |
| Benchmark ranking inversions | 4 | 2 |

The mean error got slightly worse (0.0389 to 0.0452) but the ranking, the thing I was actually optimising, improved. I'd take that trade again. The lesson I keep repeating to myself: keep two different kinds of test, because the benchmark would never have caught this one.

Here is the same sweep on today's build, after later changes: a smooth fall-off, no cliff.

```tikz
\begin{tikzpicture}[font=\sffamily\footnotesize, x=0.2cm, y=5cm]
\foreach \v in {0,0.25,0.5,0.75,1.0} {\draw[gray!35] (0,\v) -- (33,\v); \node[left] at (0,\v) {\v};}
\foreach \b in {4,8,12,16,20,24,28,32} {\node[below] at (\b,0) {\b};}
\node[below, yshift=-3.5mm] at (16,0) {stroke size, px};
\draw[very thick, teal!60!black] plot[mark=*, mark size=1.4pt] coordinates {(4,0.888) (6,0.805) (8,0.738) (9,0.656) (10,0.650) (11,0.595) (12,0.594) (14,0.640) (16,0.619) (20,0.430) (24,0.243) (32,0.186)};
\node[align=center, anchor=west] at (13,0.88) {smooth fall-off,\\no cliff at 9 px};
\end{tikzpicture}
```

## The margin that was giving out free points

Most paintings aren't square, so both images are letterboxed onto the square working image with flat bars. A player once filled the *Water Lilies* canvas with one flat colour and scored 49%, "passable", for a canvas with no picture on it.

The flat margins on both sides "agreed" perfectly no matter what the player painted, which handed free credit to every component. The fix was to score only the real content rectangle:

```tikz
\begin{tikzpicture}[font=\sffamily\footnotesize]
\fill[gray!25] (0,0) rectangle (4,4);
\fill[blue!25] (0,0.9) rectangle (4,3.1);
\draw[thick] (0,0) rectangle (4,4);
\draw[thick, blue!60!black] (0,0.9) rectangle (4,3.1);
\node at (2,2) {painting};
\node at (2,0.45) {letterbox margin};
\node at (2,3.55) {letterbox margin};
\draw[<->] (-0.3,0) -- node[left] {128 px} (-0.3,4);
\draw[<->] (4.3,0.9) -- node[right, xshift=1mm, align=left, text=blue!60!black] {content rect:\\the only part\\that is scored} (4.3,3.1);
\end{tikzpicture}
```

Getting there turned up three sub-bugs, each of which would reopen the leak if reintroduced: a tie in the alignment search silently picked a shift that dragged margin back in, rounding the mask boundary to the *nearest* grid cell leaked margin about half the time (it has to round inward), and my own test harness forgot that its loader flips images vertically. After the fix, composition for that flat fill is exactly 0.000 on all 57 references.

## Resisting cheats

Players find the cheapest path, so I measured the cheapest paths. Current numbers, averaged over the catalogue:

```tikz
\begin{tikzpicture}[font=\sffamily\small, x=7cm, y=0.62cm]
\definecolor{teal}{HTML}{2DD4BF}
\foreach \v in {0,0.1,0.2,0.3,0.4} {\draw[gray!35] (\v,0.4) -- (\v,-9.6); \node[above, font=\sffamily\footnotesize] at (\v,0.4) {\v};}
\foreach \n/\s/\i/\c in {Blank canvas/0.000/0/gray, Random noise/0.056/1/red!60!black, Scribbles in right colours/0.100/2/red!60!black, Average-colour fill/0.106/3/red!60!black, Wrong painting/0.117/4/red!60!black, Half finished/0.315/5/orange!80!black} {
  \node[left, font=\sffamily\footnotesize] at (0,-\i*1.5) {\n};
  \fill[\c] (0,-\i*1.5-0.4) rectangle (\s,-\i*1.5+0.4);
  \node[right, font=\sffamily\footnotesize] at (\s,-\i*1.5) {\s};
}
\end{tikzpicture}
```

On average, even a half-finished copy (0.32) scores about three times what the best cheat does (0.12). A different painting entirely is the highest of the cheats, which seems right: it is at least a painting.

![The painting screen in Counterfeit Gallery.](/images/blog/cg-scoring/game-painting.webp)

## Tuning without opening the game

The scoring files have no dependency on Unity, so I compile the exact same sources into a console app and run the benchmark, the stroke-size sweep, a per-component breakdown and a regression gate from the command line. Any parameter can be swept from the command line too. Everything above came out of that loop, at about a millisecond per evaluation.

Today's benchmark: what I wanted for each kind of attempt next to what the scorer gives.

```tikz
\begin{tikzpicture}[font=\sffamily\footnotesize, x=6.2cm, y=0.5cm]
\definecolor{teal}{HTML}{2DD4BF}
\foreach \v in {0,0.25,0.5,0.75,1.0} {\draw[gray!35] (\v,0.7) -- (\v,-15.7); \node[above] at (\v,0.7) {\v};}
\foreach \n/\m/\w [count=\i from 0] in {exact/1.000/1.000, blur light/0.853/0.900, blur heavy/0.649/0.720, wobble/0.725/0.780, posterize/0.774/0.800, shift 5\%/0.760/0.820, low detail/0.652/0.620, greyscale/0.691/0.680, washed out/0.781/0.750, wrong hues/0.628/0.580, half done/0.315/0.420, other painting/0.117/0.200, flat fill/0.106/0.120, scribble/0.100/0.100, noise/0.056/0.050, blank/0.000/0.000} {
  \node[left] at (-0.02,-\i) {\n};
  \draw[gray!60] (\m,-\i) -- (\w,-\i);
  \draw[thick, orange!80!black, fill=white] (\w,-\i) circle (1.3mm);
  \fill[teal!60!black] (\m,-\i) circle (1.4mm);
}
\fill[teal!60!black] (0.45,-17.2) circle (1.4mm) node[right, black] {\ scorer};
\draw[thick, orange!80!black, fill=white] (0.75,-17.2) circle (1.3mm) node[right, black] {\ what I wanted};
\end{tikzpicture}
```

## Where it is still soft

- A washed-out, low-contrast copy scores a bit higher than a person would give it, since composition is contrast-blind by design and only colour notices.
- A copy drawn well but in the wrong colours still lands around 0.69. That is deliberate: the screen shows composition 99, colour 33, shape 99, detail 100, and a much lower total would look like a bug.
- The benchmark still has a few ranking inversions, all between cases within about 0.1 of each other, which is inside the range where two people would also disagree.

The wet-paint and blending side of the game (subtractive pigment mixing, brush load, wet-in-wet) is a separate story, and I'll write that up separately.

*Counterfeit Gallery is on [itch.io](https://code-scrambler.itch.io) and Google Play.*