Building a Licensed Anime Upscaling Test Set in Public

Anime upscaling is often demonstrated with one dramatic wallpaper enlargement. That makes a good screenshot and a weak acceptance test. A model can sharpen a character outline beautifully while destroying the flat color beside it, turning a screentone pattern into mud, or painting a dark fringe around a transparent PNG. We wanted a test set that could catch those failures before we built an anime landing page.

So we started with sources and failure modes, not a new model. We collected 18 real, externally authored images with licenses that permit testing and publication. The first formal suite uses 15 modern anime and illustration sources across clean, JPEG70, and JPEG40 variants—45 deterministic runs. A cosplay photograph, a photo of a printed calendar, and a historic woodblock print stay out of the modern-anime aggregate. They are useful boundary material, but calling them modern anime would make the result easier to market and less honest.

What the set actually contains

The first two collection rounds produced a deliberately mixed set:

• cel-shaded and flat-color characters, including low-resolution JPEGs;
• fine and faint line art;
• pencil and grayscale texture;
• transparent PNGs;
• detailed digital wallpapers;
• painterly anime-style illustration;
• a manga portrait in grayscale, screentone, and colored variants.

The three manga variants share one composition. That is intentional: they let us compare what the same pipeline does to ordinary grayscale shading, a regular dot pattern, and color highlights. They do not count as three independent topics in our coverage claims.

We also know what is still missing: licensed modern animation frames, more complex natural backgrounds, and explicit gradient banding cases. A test suite with a documented gap is more useful than a claim of universal coverage.

Licensing is part of the test fixture

Every source has an entry in our manifest with its author, source page, original file, downloaded version, dimensions, SHA-256, and license. The accepted licenses are public domain, CC0, CC BY, and CC BY-SA. Source review is not a legal warranty, but an image with no traceable origin does not belong in our public benchmark.

That metadata cannot collapse into one dataset-level license. Different cases carry different authors and terms. Share-alike derivatives need to remain under the same license when we publish them. Resized Wikimedia thumbnails are identified as thumbnails rather than described as original files, and the SVG manga variants are identified as raster renditions.

This is tedious work. It is also what allows the Quality page, downloadable JSON, CSV, and blog posts to show real inputs instead of mysteriously sourced anime art.

The degradation protocol

For the formal suite, each modern source is first resized to fit within 960 pixels on its longest side. That keeps the browser benchmark practical and gives us a consistent reference. We then create a 4× bicubic downsample, optionally encode it as JPEG at quality 70 or 40, decode it, and run the production upscaler back to 4×.

The clean and JPEG paths let us separate two different questions:

1. Can the model enlarge clean line art and flat color without inventing damage?
2. Does it treat compression artifacts as texture that should be amplified?

PSNR and SSIM compare the result with the resized source. They do not prove that an image is pleasant to look at. A model can score well and still break a line in a place a viewer notices immediately. That is why every case also has fixed crops and a human review state: pass, warn, fail, or unreviewed.

The 2× path is evaluated separately. ClearPix currently produces 2× by running 4× super-resolution and downsampling the result; a fast path that threw away that extra work has already failed our quality gate. We are not going to hide that trade-off in an anime-specific report.

The failure checklist

Our review is organized around the things anime viewers actually notice:

• broken, thickened, or doubled line art;
• bright or dark halos around clean edges;
• synthetic texture in flat colors;
• screentone that disappears, sticks together, or becomes a different pattern;
• banding in gradients;
• JPEG blocks and mosquito noise made more visible;
• shifts in skin, hair, or highlight color;
• transparent pixels turning opaque, or dark and light fringes around alpha edges.

During implementation we confirmed that the shared upscaler was writing alpha 255 for every output pixel. An anime model cannot fix that for us. Alpha now has to be reconstructed from the source at the output size and checked separately from RGB quality.

Why publish the set before claiming a model win

Our production model is realesr-general-x4v3, a compact general-purpose model. The Real-ESRGAN project also distributes an anime-specific model. The existence of that model is not evidence that we need it, and the word “anime” in a filename is not evidence that it will be better in a browser.

The general model gets the first run. If it handles line art, flat color, screentone, transparency, and compressed inputs well enough, we will keep the simpler pipeline. If it fails the core cases, any anime-specific alternative will have to beat it on the same inputs, crops, runtime, and download budget.

We are publishing the Quality data and real output crops so that decision can be inspected. The companion baseline report records what happened, including the cases that did not pass. The eventual anime image upscaler will use the same production path, not a separate demo pipeline.

That is the point of building the test in public: the page can describe the evidence, rather than asking the evidence to support the page.