format zoo for videos - a bad idea in digital preservation

Background 

In an article on https://axfelix.github.io/ffv1, reasons are given not to apply the existing normalization of born-digital videos to FFV1, but to convert to lossy codecs instead. Elsewhere I even heard that normalization is not applied at all because it requires so many resources.

Why is normalization a good idea after all?

Normalization ensures that a manageable set of file formats remains from the huge format zoo, which can be handled well in the future. Normalization therefore reduces the organizational complexity above all.

And why should you use Matroska/FFV1?

FFV1 has the disadvantage of imposing higher storage requirements on its users, but in my opinion, the following points outweigh it:

• FFV1 is much less complex than h264 (read "reduced technical complexity")
• FFV1 (like other lossless codecs) allows automatic format migration (see also RAWcooked) — this reduces organizational complexity
• FFV1 is freely available, widely used, well documented and standardized

The point that FFV1 is also more resistant to bit rot is just the icing on the cake.

Summary

Incidentally, personnel cost is the cost driver in digital preservation, as opposed to the pure storage cost.

Hence, the ultimate question is: how expensive is storage capacity in relation to the reduced technical and organizational complexity?

Hinweis auf interessantes Interview zu FFV1

Ein äußerst interessantes Interview von Jürgen Keiper mit Peter Bubestinger zur Entstehung und Motivation von Matroska/FFV1 als langzeitarchivfähiges Datenformat für audiovisuelle Medien.

Es ist besonders interessant für Archivare, die wissen wollen, warum FFV1/Matroska ihre Probleme lösen kann. Peter schafft es Sachverhalte einfach und anschaulich zu erklären und kommt (fast) ohne technisches Vokabular aus.

Prädikat: Sehenswert!

Hier der Link zum Video:

https://www.memento-movie.de/2017/08/die-geschichte-eines-codecs-ffv1-in-der-archivwelt/

FFV1 - some compression results

In a pilot we got some retrodigitized films and videos in Matroska/FFV1 format. In the following table I summarized the results:


n/a

film/video	1	2	3	4	5
description	8mm, positive, b/w	8mm, positiv, b/w	16mm, positive, b/w	35mm, combined, color	35mm, combined, color
width	2500	2500	2048	4096	4096
height	1524	1524	1520	3460	2976
bits per pixel	48	48	48	48	48
pxfmt	gbrp16le	gbrp16le	gbrp16le	gbrp16le	gbrp16le
duration in s	12	12	11,459	2,5	2,5
fps	24	24	24	24	24
frames	288	288	275	60	60
original size	6583680000	6583680000	5136682844,16	5101977600	4388290560
compressed size	3861943880	3790690517	3680779719	3908475344	3576745774
compression ratio	1,704	1,736	1,395	1,305	1,226
(DPX size)	6584159232	6584159232	5136841600	5102077440	4388390400
(h264 lossless)	n/a	n/a	n/a	n/a	n/a
(h265 lossless)	3573420309	3559442475	2756504247	3015053822	2992764833
(jp2k lossless)	4589886341	4534014321	3732555539	3869665916	3514687046
with audio	n	n	n	y	n

n/a

film/video	6	7	8	9	10
description	35mm, combined, color	vhs, color	betacam, color	betacam, color	Digi-beta, color
width	4096	720	720	720	720
height	3200	576	576	576	576
bits per pixel	48	20	20	20	20
pxfmt	gbrp16le	yuv422p10le	yuv422p10le	yuv422p10le	yuv422p10le
duration in s	1088,042	280	280	280	280
fps	24	25	25	25	25
frames	26113	7000	7000	7000	7000
original size	2053610510746	7257600000	7257600000	7257600000	7257600000
compressed size	1575415175611	3565437155	3838500934	3449372280	4451325952
compression ratio	1,303	2,035	1,890	2,104	1,630
(DPX size)	2053653333632	17472217728	17429888000	17429888000	17429888000
(h264 lossless)	n/a	n/a	n/a	n/a	n/a
(h265 lossless)	1248031292634	3659828688	3772522257	3442739259	4323623225
(jp2k lossless)	1517117560575	3300899483	3470434177	3150727081	4022908822
with audio	n	y	y	y	y

All files are encoded with FFV1v3 with slices, slice-crc, GOP=1. If audio exists, it is (lin. PCM 48kHz, 16bit) included in compression-size, but not in original size, because original size is calculated by width*height*pits_per_pixel*frames and compression-size is equivalent to filesize. The count of frames is calculated with the duration value of the MKV-files. The files 1 to 5, and 7-10 are first parts of the movies (each 4GB splits).

Hint: Once the project is completed, rights must be clarified. If possible, I will publish the sources.

Update 2017-06-09

• added file size for DPX after using "ffmpeg -i input.mkv DPX/frame_%06d.dpx"
• added file size for h264 after using "ffmpeg -i input.mkv -c:v libx264 -g 1 -qp 0 -crf 0 output.mkv" (RGB without lossy conversion to YUV not supported yet)
• added file size for h265 after using "ffmpeg -i input.mkv -c:v libx265 -preset veryslow -x265-params lossless=1 output.mkv"
• added file size for openjpeg2000 after using "ffmpeg -i input.mkv -c:v libopenjpeg output.mkv" 

Update 2017-06-29

• added sizes for film no 6
• in general, the processing time of h265 and jp2k is one magnitude greater than for ffv1

Interpretation

The files 1-3 are all originally b/w. It seems to be that the codec does not decorrelate the color channels. Also the material 1-6 is retrodigitized from film and are noisy. The file 1 is very special. In decoding the FFV1 produces a very high load on the CPU (eight cores at 100%). The most decoding time is spent in method get_rac(). The original film has the highest noise level in contrast to the other files.

I think the compression-ratio difference between video- and film files comes from the different pixel format. A ratio between 1,5 - 2 was expected, but 1,3 is a surprise.

Update 2017-06-09

The reason for high CPU load was, that the digitization service provider has created a file with a framerate of 1000 fps, but the scanner has provided 24 or 25 fps. Therefore 42-40 equal frames  was encoded on block.