One of the great things about having rack-space in our new office is that we can now support open source projects using our equipment such as FFmpeg and Libav. They are critical parts of our software as well as underpin much of multimedia processing in the world today.

Fuzzing, is one of the ways in which we can improve the quality of the decoders when exposed to corrupted input. It involves randomly or systematically corrupting the input of a program in order to make it crash. The heartbleed vulnerability was one of the most famous bugs found via fuzzing [1].

Google, notably fuzzed FFmpeg and Libav at a relatively large scale, leading to a thousand fixes. But after seeing crashes in the H264 decoder earlier in the year, with real-world events such as packet loss and video splices, it was clear that something was wrong. One possibility is that Google only fuzzed progressive H264 content using frame threads and didn’t include interlaced content nor tried decoding in the lower-latency sliced-threads mode. Or that the codebase changed significantly enough to introduce new bugs.

Using basic tools like zzuf and later on the more advanced american fuzzy lop and a single quad-core server (in contrast to Google’s 2000 cores), the following unique bugs were found, a few of which caused easily-triggerable, real-world crashes.

H264 Frame Threads

https://trac.ffmpeg.org/ticket/4428

H264 Sliced Threads

https://trac.ffmpeg.org/ticket/4440

https://trac.ffmpeg.org/ticket/4438

https://trac.ffmpeg.org/ticket/4431

https://trac.ffmpeg.org/ticket/4408

https://trac.ffmpeg.org/ticket/4977

FFv1

https://trac.ffmpeg.org/ticket/4931

https://trac.ffmpeg.org/ticket/4932

https://trac.ffmpeg.org/ticket/4939

VP9

https://trac.ffmpeg.org/ticket/4935

Opus

https://bugzilla.libav.org/show_bug.cgi?id=876

https://bugzilla.libav.org/show_bug.cgi?id=909

Thanks to @rilian for providing fuzzing scripts and thanks to those who investigated and fixed the bugs, Michael Niedermayer in particular.

[1] http://www.codenomicon.com/files/pdf/Heartbleed-Story.pdf