FFmpeg

(Stand H16, BVE, London)  Open Broadcast Systems is pleased to announce its development and open-sourcing of a VC-2 HQ encoder and decoder. VC-2, formerly known as Dirac Pro, is ideally suited for mezzanine compression of video allowing HD to be transported using Gigabit Ethernet or 4K down Ten-Gigabit Ethernet at an ultra-low latency. As a core component in all of our products, FFmpeg 3.0 is the destination for this code, allowing any manufacturer to use and improve VC-2 encoding and decoding.

At the Broadcast Video Expo in London, we will be showing a low-latency encoder and decoder running on a wide range of hardware, from a software encoder you can fit in the palm of your hand, to high density blades, ideal for remote production.

Apart from being patent-free VC-2 has another trick up it’s sleeve compared to codecs not designed for video like JPEG2000. It doesn’t just have low generation loss, it has zero generation loss (thanks to its symmetric quantisation). Provided the downstream encoders are configured correctly (same bitrate, same slice-size etc.), there will be only a single encode generation, a unique feature in the industry.

VC-2 is also much more lightweight allowing reduced power consumption and encoding of higher resolutions without tiling and only requiring a modest bitrate penalty. However, in our implementation we’re going much further than many hardware manufacturers who are implementing only the simplest variant of VC-2.

Credit must go to BBC R&D for implementing high-quality, freely available, reference implementations and writing a clear specification.

Our implementation can be found in FFmpeg 3.0 which is available at http://www.ffmpeg.org

One of the great things about having rack-space in our new office is that we can now support open source projects using our equipment such as FFmpeg and Libav. They are critical parts of our software as well as underpin much of multimedia processing in the world today.

Fuzzing, is one of the ways in which we can improve the quality of the decoders when exposed to corrupted input. It involves randomly or systematically corrupting the input of a program in order to make it crash. The heartbleed vulnerability was one of the most famous bugs found via fuzzing [1].

Google, notably fuzzed FFmpeg and Libav at a relatively large scale, leading to a thousand fixes. But after seeing crashes in the H264 decoder earlier in the year, with real-world events such as packet loss and video splices, it was clear that something was wrong. One possibility is that Google only fuzzed progressive H264 content using frame threads and didn’t include interlaced content nor tried decoding in the lower-latency sliced-threads mode. Or that the codebase changed significantly enough to introduce new bugs.

Using basic tools like zzuf and later on the more advanced american fuzzy lop and a single quad-core server (in contrast to Google’s 2000 cores), the following unique bugs were found, a few of which caused easily-triggerable, real-world crashes.

H264 Frame Threads

https://trac.ffmpeg.org/ticket/4428

H264 Sliced Threads

https://trac.ffmpeg.org/ticket/4440

https://trac.ffmpeg.org/ticket/4438

https://trac.ffmpeg.org/ticket/4431

https://trac.ffmpeg.org/ticket/4408

https://trac.ffmpeg.org/ticket/4977

FFv1

https://trac.ffmpeg.org/ticket/4931

https://trac.ffmpeg.org/ticket/4932

https://trac.ffmpeg.org/ticket/4939

VP9

https://trac.ffmpeg.org/ticket/4935

Opus

https://bugzilla.libav.org/show_bug.cgi?id=876

https://bugzilla.libav.org/show_bug.cgi?id=909

Thanks to @rilian for providing fuzzing scripts and thanks to those who investigated and fixed the bugs, Michael Niedermayer in particular.

[1] http://www.codenomicon.com/files/pdf/Heartbleed-Story.pdf

 

Note: This is a more technical post than usual, and about 5 months late.

The decoding in the OBE C-100 decoder was optimised to make use of instructions in modern CPUs and this blog post explains how we did it:

HD-SDI video uses 10-bit pixels but computers operate in bytes (8-bits). However, 10-bit professional video doesn’t fit nicely into bytes. Instead, 10-bit video on a computer is stored in memory like this:



The X represents an unused bit - note how in total 12 out of 32 of the bits are unused (that’s 37.5%). It’s very wasteful if the data needs to be transferred to a piece of hardware like a Blackmagic SDI card. Virtually all professional SDI cards use the ‘v210’ format that was first introduced by Apple in the 90s [1] and v210 improves the efficiency of 10-bit storage by packing the 10-bit video samples as follows:

(adapted from [1])

Now only 2 out of the 32-bits are unused, a major improvement. Using the old v210 encoder in FFmpeg, each pixel is loaded from memory, shifted to the correct position and “inserted” using the OR operation. When doing this on 1920x1080 material, this involves about 250 million of these operations every second. More CPU time is spent packing the pixels for display than actually decompressing them from the encoded video!

Clearly, we’ve got to do something about this - Thanks to the magic of SIMD instructions (in this case SSSE3 and AVX) we can instead process 12 pixels in one go [2]: 

  1. Load luma pixels from memory
  2. Make sure they are within the v210 range
  3. Shift each pixel (if necessary) to appropriate position
  4. Shuffle pixels to rearrange them to v210 order
  5. Repeat 1-4 for chroma
  6. OR the luma and chroma registers together
  7. Store in memory

This can be (unscientifically) benchmarked with the command:

ffmpeg -pix_fmt yuv422p10 -s 1920x1080 -f rawvideo -i /dev/zero -f rawvideo -vcodec v210 -y /dev/null

Before: 168fps

After: 480fps

A 3x speed boost.

But, a lot of content that the decoder receives is 8-bit which has this packing format:



In existing software decoders, this needs to be converted to the 10-bit samples in the first picture and then packed into v210, a two step process. But, we can now just do this in a single step.

ffmpeg -pix_fmt yuv422p -s 1920x1080 -f rawvideo -i /dev/zero -f rawvideo -vcodec v210 -y /dev/null

Before: 95fps

After: 620fps

Now 6.5x faster!

What more could be done: 

  • Allow the decoder to decode straight to v210 using FFmpeg's draw_horiz_band capability. 
  • Try using AVX2 on newer Haswell CPUs - should provide a small speed increase but with an increased complexity.
  • Use multiple CPU cores on the conversion - this isn’t really useful for OBE but people creating v210 files may find it useful (especially UHD content).

Thanks must go to those who helped review this code.

[1] https://developer.apple.com/library/mac/technotes/tn2162/_index.html#//apple_ref/doc/uid/DTS40013070-CH1-TNTAG8-V210__4_2_2_COMPRESSION_TYPE

(This is from Apple’s venerable Letters from the Ice Floe)

[2] http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavcodec/x86/v210enc.asm