Video Codecs: Everything You Need To Know About

During the pandemic when most of the world was under lockdown and more people were online than ever, major video streaming platforms took a collective decision to limit video content to 480p to relieve strain on global mobile networks. Did you know YouTube (35%) and Netflix (15%) account for 50% of all global mobile internet traffic?  The report also suggests 79% of global mobile data traffic will be video by 2022. With this growth rate, video codecs will play an even bigger role in the years to come.

Statista provides information on the level of global mobile video traffic from 2017 to 2022.

In 2018, global mobile video traffic amounted to 12,051 petabytes per month and is expected to multiply to 60,889 PB per month in 2022. Mobile traffic accounts for only 60% of total traffic. It is safe to predict video content consumption will reach 100, 000 PB/month in 2022.

What is a Video Codec?

A video codec is a technology that compresses and decompresses digital video files. A video codec compresses a raw video file so that it can be easily streamed online. It consists of an encoder and a decoder. An encoder compresses an analog video file, whereas a decoder decompresses the video file at the device to prepare the video for playback. 

The term “codec” stands for “coder-decoder,” emphasizing its role in both shrinking video data for storage or transmission and restoring it for playback. Raw video files are typically large, demanding significant storage and making online streaming or sharing difficult.

To make videos easier to stream, codecs compress the original file using an encoder, which reduces the size of the video. The corresponding decoder on the playback device then decompresses the video, making it ready for viewing.

Video codecs operate by applying specific algorithms to reduce the file size, ensuring that it is easier to store and transfer. Upon reaching the destination device, the codec decompresses the file so the video can be played.

This compression process involves eliminating unnecessary data while maintaining video quality as much as possible. This is crucial for efficient storage and transmission, especially in streaming scenarios. Popular codecs like H.264, H.265 (HEVC), VP9, and AV1 are tailored to different needs, balancing video quality with file size to suit various applications.

Codecs essentially work in the way that they apply algorithms to compress the video file into the relevant container format. This makes storage and transfer of the file easier as the compressed file is of significantly low size than the raw format. When the video file is at the targeted device, codec then the video file is decompressed, so that it can be viewed by the user. 

Why Do You Need A Video Codec?

Raw video and audio files are very high in size. This makes it difficult to stream over the internet, as it will end up taking a lot of bandwidth and storage to play a video. Without a codec, it’ll be difficult to stream videos online. As for a full HD file, i.e. 1080p file, the bandwidth consumption can come up to 7GB/ minute. It’ll take forever to stream a video even with a decent network. 

Video codecs make it much easier to stream the content online. As they drastically decrease the video file size and optimize it for online streaming. Which in turn gives a smooth and high video frame rate. 

A video codec is essential for managing digital video files efficiently. Here are the key reasons why you need a video codec:

• File Size Reduction: Raw video files are often extremely large, making them difficult to store and manage. A video codec compresses these files, significantly reducing their size without severely compromising quality. This allows you to save space on your devices and make video storage more efficient.
• Efficient Streaming: Large video files can be challenging to stream over the internet, especially on slower connections. Video codecs reduce the file size, making it easier to stream videos smoothly without buffering or interruptions. This is particularly important for platforms like YouTube, Netflix, and other streaming services, where quick and seamless video delivery is crucial.
• Faster Transmission: Smaller video files are quicker to upload, download, and share over the internet. Whether you’re sending videos via email, uploading them to social media, or sharing them on a cloud platform, a codec ensures that the process is faster and more efficient.
• Compatibility Across Devices: Different devices and platforms support different video formats. Video codecs ensure that your video files are compatible with a wide range of devices, from smartphones and tablets to computers and smart TVs. This compatibility is vital for ensuring that your videos can be viewed without issues on various platforms.
• Preserving Video Quality: While reducing file size, video codecs are designed to maintain the best possible video quality. Advanced codecs like H.265 (HEVC) and AV1 are particularly effective at balancing compression with quality, ensuring that viewers still enjoy a high-quality viewing experience even with reduced file sizes.
• Enabling High-Quality Video Production: In professional video production, codecs allow for the efficient handling of high-resolution video, such as 4K or 8K footage. Without codecs, managing and editing such large files would be incredibly cumbersome and time-consuming.

How Do Video Codecs Work?

For any video codec to work it needs to compress the frames. There are two types of frame compressions – inter-frame and intra-frame compression. In an intra-frame compression, each frame is compressed independently of the adjacent frames. It is therefore essentially image-compression applied to video.

As discussed above, video codecs essentially compress the video file. They do it by compressing the individual frames in the video. This type of compression can be further divided into two parts. They are intra-frame compression and inter-frame compression. 

With intra-frame compression, every single frame is compressed individually. It can also be stated as image compression applied to each and every frame of the video. 

Inter-frame compression identifies all the static parts in the frames and uses them again in the consequent frames. For example, if there’s a video of a single person speaking in the stationary background, it’ll use all the pixels which are stationary in the next frames, while only adding the moving pixels from the subsequent frames.  It is much more efficient than intra-frame, this is why most of the video codecs are optimized to identify redundancies across frames.

Video codecs are crucial to the video streaming process, enabling efficient transmission of high-quality video content over the internet. Here’s how they work in relation to video streaming:

1. Video Capture and Encoding:
   • Raw Video Capture: When content is created, whether it’s a movie, a live stream, or a recorded video, it starts as raw, uncompressed data. This raw data is extremely large and unsuitable for streaming due to its size.
   • Compression and Encoding: Before the video can be streamed, it must be compressed using a video codec. The encoder part of the codec analyzes the video, identifies redundant or unnecessary information, and compresses the file into a smaller size. This process reduces the data rate (bitrate) of the video, making it easier to transmit over the internet without consuming excessive bandwidth.
2. Streaming Server Preparation:
   • Transcoding: Often, the video content is transcoded into multiple formats and bitrates on the server side. This allows the same video to be streamed to different devices and internet connections with varying quality. For example, a video might be encoded at 1080p for high-speed connections and at 480p for slower connections.
   • Packaging: The compressed video is then packaged into a format suitable for streaming, such as HLS (HTTP Live Streaming) or DASH (Dynamic Adaptive Streaming over HTTP). These formats allow for the smooth delivery of video over the internet.
3. Delivery and Adaptive Bitrate Streaming:
   • Adaptive Bitrate Streaming: One of the key roles of a video codec in streaming is enabling adaptive bitrate streaming (ABR). ABR technology monitors the viewer’s internet connection in real-time and dynamically adjusts the quality of the video stream accordingly.
     • For example, if a viewer’s connection is strong, the stream may be delivered in higher quality (e.g., 1080p). If the connection weakens, the stream can automatically switch to a lower quality (e.g., 720p or 480p) to avoid buffering.
     • This ensures a smooth viewing experience, even if the viewer’s internet connection fluctuates.
   • Buffering and Latency: Efficient codecs are designed to minimize buffering and latency. They compress video data in a way that allows it to be streamed in smaller chunks, reducing the time it takes to load and play on the viewer’s device.
4. Decompression and Playback:
   • Real-Time Decompression: As the compressed video data reaches the viewer’s device, the codec’s decoder processes it in real-time. The decoder decompresses the video, reconstructing the frames for playback. This allows viewers to watch the video smoothly, as if it were a locally stored file, despite it being streamed from a remote server.
   • Device Compatibility: The efficiency and compatibility of codecs are critical for video streaming. Different devices (smartphones, tablets, smart TVs, etc.) may support different codecs. For instance, H.264 and H.265 (HEVC) are widely supported across most modern devices, ensuring that streamed videos can be played back without compatibility issues.
5. Optimizing for Bandwidth and Quality:
   • Balancing Quality and Bandwidth: In streaming, the balance between video quality and bandwidth usage is key. Video codecs like H.264, H.265, VP9, and AV1 are designed to provide high-quality video at lower bitrates. This balance is crucial because it allows streaming platforms to deliver high-definition content without requiring excessive bandwidth, making it accessible to users with varying internet speeds.
   • Efficient Streaming: By reducing the file size while maintaining video quality, codecs enable streaming services to deliver content quickly and efficiently. This is especially important for live streaming, where minimal delay (latency) is crucial for real-time interaction.

 

The Relevance Of Video Codec In The Post-Streaming World

Every time we binge a series on Netflix, make a Facetime call on our iPhones, or share a video on Telegram, we are streaming video from one computing device to another. It could be from the Netflix data center to your laptop or your iPhone to your manager’s iPad via Apple data centers.

Since bandwidth is limited from server to users and there is only so much traffic a data center can handle, especially if it is a colocation data center, streaming video is all about delivering the best quality videos to users in the smallest of bytes The question is how can you compress a video without losing its quality?

The answer is video codecs.  Video Codecs are the reason we could stream 1.336 billion minutes of Mandalorian in 4K in a matter of weeks.

Netflix, Apple, Telegram, Disney Plus all employ a bunch of video codecs to ensure smooth delivery of video content from their data centers to your device.

What’s The Difference Between Video Codecs and Containers

It can be pretty easy to confuse between the two, as a beginner to the topic.

Video container is essentially a package which contains the metadata for the file, along with video and audio codec. 

A single video container can contain different types of video codec. They are more or less, a storage unit. What they essentially do is, open and allow the codecs to work on the video. But for this to work, a codec should essentially be supported by the device used to stream the video. 

Not all devices support all types of video containers and codecs. This is why it is generally advised to encode video in different formats. Based on what devices you want your video to be streamed. 

What Are Some Of The Best Video Codecs for Streaming?

The video codec, you should eventually go for, essentially depends on what devices you want your videos to be streamed on. Here are some popular video codecs out there. 

• H.264/AVC
• H.265/HEVC
• AV1
• VP9

H.264/AVC

H.264/AVC is the most widely used video codec for video streaming. It is supported by every single device out there. At a similar file size, it gives much better video quality than the video codecs before it. 

H.265 or HVEC

H.265 is better than H.264 in terms that it gives half the bitrate than it and is good for playback high-resolution videos. But it has a downside, H.265 uses about three times more resources than H.264. H.265, the successor to near-universal codec H.264, HEVC’s loyalty model is too complicated. Hence, major tech companies are giving it a pass to circumvent the loyalty issues

Indeed, there are some issues to iron out. For example, if we talk about Apple’s beloved codec HEVC, or and working together to build a loyalty-free, open codec to replace H264 and VP8 for 4K streaming.

AV1

AV1 is coming as an immense success with many device makers, smartphone platforms, desktop operating systems, and browsers supporting it. The Alliance for Open Media (AOMedia), which counts Apple, Google, Amazon, Facebook, Mozilla, and Microsoft as its members, sees AV1 as a successor to VP9 and a royalty-free alternative to HEVC. AV1 uses elements from Daala, Thor, and VP10, three preceding open video codecs for backward compatibility.

VP9

VP9 is a free video code developed by Google. It is supported by all of the Google products, i.e.Android, Chrome, and YouTube. In terms of performance, it provides better video quality at the same bitrate than H.265. But, it is not supported by Apple devices. 

Uncompressed RAW and Consumer Codecs

Outside a video codec, a video is uncompressed, unprocessed, lossless, and enormous in size.

 

A raw, uncompressed video is perfect for storing and editing purposes but transmitting such a large video over a network is impractical.

Codecs compress a video before sending it over a network through a process of encoding. Each video codec has a different method for video encoding and decoding an uncompressed RAW video. Video decoders for common consumer video codecs are generally baked into operating systems such as Windows, iOS, Android, etc. although that might not be the case all the time.

Mac and iPhone users were unable to stream YouTube in 4K until recently because Apple wouldn’t add a VP9 decoder to its platforms. The codec feud between Google and Apple remained unresolved until the release of macOS Big Sur and iOS 14 and came at the expense of millions of users’ experience.

Compressed RAW vs Uncompressed RAW

A raw uncompressed video is compressed using one of the available video codecs before transmitting to the network and making it available across the users.

The noir backlit looks are some of the signature elements of the often very violent period drama and have helped make the BBC series an international hit.

Uncompressed RAW Video Encoding

The first four seasons of Peaky Blinder were shot on an ARRI camera in ARRIRAW format. ARRIRAW, unlike Apple’s ProRes RAW or Google’s VP9, is not a video codec. Rather, it is a format that literally returns uncompressed, unencrypted, and uncompromised sensor data. In a conventional sense, ARRIRAW can be considered a digital version of the camera negative. Most of our favorite movies including the Snyder Cut of 2017 Justice league were shot on an ARRI camera in ARRIRAW format.

Compressed RAW Video Transcoding

For later seasons, however, cinematographer SI Bell shifted to Red cameras and production went into Red’s proprietary Redcode RAW format. Redcode RAW is a professional-grade, intermediate CODEC, which is intended for post-production use. Redcode RAW along with Apple’s ProRes RAW, Blackmagic RAW, Hasselblad RAW are some of the examples of compressed RAW codecs. In fact, pretty much every major camera-maker and video-editing software company has a proprietary RAW format to its credit.

The Tradeoff Between Professional and Consumer Video Codecs

Professional codecs, while doing a great job for post-production, aren’t intended for end-users consumption. Professional codes need specialized software to run, and the encoded video is comparatively huge in size to transmit over a network, making them impractical for streaming partners or P2P transfers. Therefore, videos in professional video codecs are transcoded to consumer codes such as H264 or MPEG4 before transmission.

When Warner Bros picked HBO Max to release the Snyder’s Cut of Justice League, the team at HBO had to encode the movie from ARRIRAW to tens of consumer codecs to support every device and browser HBO Max is available on, including Apple iPhones and Google Chrome.

To give you an idea, an HD movie in uncompressed RAW format could go more than a Terabyte in size.

It was a huge task for the team to retain the video quality of the theatric masterpiece in 4:3 aspect ratio on our small screens. Consumer video codecs have a considerably higher compression ratio than professional codecs, but they are essential for consistent delivery of video content over streaming and P2P channels.

Compressed RAW codec is smaller in size and faster to transcode

However, for the fifth season of Peaky Blinders, Netflix did not have to encode the uncompressed video since it was shot in Redcode RAW, which is a compressed RAW codec itself. Rather the team just had to transcode the video from a professional video codec to various consumer codecs such as H265, MPEG4 upon user requests. Video transcoding is a simpler and less time-consuming job than encoding. Therefore, video content intended for small screens is shot in one of the professional RAW codecs rather than uncompressed RAW.

The Anatomy Of Video Transcoding From Server To Client

Every time you’re converting a movie from H264 to MPEG4 or exporting a Hasselblad RAW footage to HEVC, you’re running a transcoding job. In a post-streaming world, however, transcoding is more about compressing video files till the point they start showing a significant quality loss. A smaller video file is easier on streaming platforms’ data centers and end-users limited bandwidth. If you want to know more about, what is transcoding, you can visit the blog linked.

The various parameters to video transcoding

While video compression is a trade-off between video bitrate (or size of the video) and video quality, there are many more parameters to consider: the resolution, bit-depth, and chroma-subsampling, encoding time, hardware, etc.

Bitstream and transmission of video content

A compressed video is transmitted in bitstreams over a network. Bitstream for each video codec is different in format and can only be decoded by a designated decoder.

The job of the decoder is to reproduce the video at the receiver’s end.

Quality loss is inevitable!

Since algorithms dictating video codecs compress the video before transmission, some quality loss is evident at the receiver’s end. However, video quality loss from modern coding algorithms is insignificant to average users.

The Lifecycle Of A Video: From production houses, streaming platforms to your living room

Most of the production-grade video contents start life as a series of compressed or uncompressed RAW clips. These clips are merged to create the end content during post-production. The end-content is then transferred to the data center of the streaming partner.

User-agent detection

Now every time a user requests to stream that video on an app or browser, the application server will run a script to gather metadata on his device, application, or browser. The video metadata contains valuable information about the user’s device: user-agent, version, operating system, screen resolution, supported codecs, etc. Based on the information, the streaming server can determine the best video codec for content delivery.

For example, if the user requests Mulan in 4K on his latest Chromebook, Disney Plus will probably deliver the content in the VP9 codec.

Identification of the best codec for content delivery

Once the best video codec is determined for content delivery over the streaming channel, the server will compress the RAW video in bitstreams and transmit the bitstreams to the user over a secure HTTP connection. The format of a bitstream varies from one video codec to another.

Reproduction of the video at the receiver’s end

Once the streaming server establishes a link with the user’s device, the codec decoder on the user’s device will start reproducing the video from the bitstreams. This how a video travels from video production houses to your living room.

Conclusion

Today, far more people than ever rely on their devices for entertainment, socializing, learning, communication, and work. Without codecs, a seamlessly connected world that we live in and take for granted would be a distant dream in a post-apocalyptic world. The idea that we can call a Zoom meeting and connect to all our colleagues at the same time in a matter of seconds regardless of the device we are on sounds surreal unless we consider codecs.

Codecs are, literally, the engines driving modern web applications. Ideally, we need a universal video codec that constantly improves to fulfill our insatiable demand for running video-centric applications. Learn more about encoding, transcoding, and metadata in the blogs linked.

FAQs

• What are the most popular video codecs?
  • The most popular video codecs include H.264 (AVC), H.265 (HEVC), VP9, and AV1. H.264 is widely used for its balance of compression and quality, H.265 offers improved compression rates, VP9 is an open-source alternative supporting high-quality video, and AV1 aims to provide high compression efficiency with royalty-free licensing.
• Why is H.265 considered better than H.264?
  • H.265, or High Efficiency Video Coding (HEVC), offers up to 50% better data compression at the same level of video quality, or significantly improved video quality at the same bit rate as H.264. This makes it more efficient for streaming high-definition video and for use in areas with limited bandwidth.
• What is the significance of AV1?
  • AV1 is a cutting-edge video codec developed by the Alliance for Open Media. It is designed to be more efficient than its predecessors, allowing for better compression and improved quality at lower bitrates. Its open-source, royalty-free nature makes it an attractive option for web video and streaming platforms aiming to reduce bandwidth usage and improve accessibility.
• Can different video codecs affect playback compatibility?
  • Yes, the choice of video codec can affect playback compatibility across various devices and browsers. Older devices and browsers may not support newer codecs like H.265 or AV1, leading to playback issues. Ensuring compatibility often involves encoding videos in multiple codecs or using fallback options for broader support.