When developing streaming pipelines with GStreamer, ffmpeg, or similar frameworks, it is important to keep track of your pipeline’s latency. In live control systems such as FPV drones and PTZ camera controllers, even small increases in latency lengthen the feedback loop and degrade the user experience. Therefore it is crucial to keep the latency in check.

“If you cannot measure it, you cannot improve it.”

misquote from Lord Kelvin

The above quote is often attributed to Lord Kelvin, which is not quite accurate. He actually said something similar, just in more words. Still, I like the misquote, so I’m going with it. Now let’s get to measuring.

1. Glass-to-glass (g2g) Link to heading

Usually the fastest way to measure a video streaming pipeline’s latency is by measuring the glass-to-glass latency. Most video streaming pipelines have a producer somewhere that is generating a stream, like a camera, and a consumer that wants to display the stream, like a monitor. In such a case, the most accurate and usually easiest way to measure the streaming pipeline’s latency is by displaying a clock on the monitor and pointing the camera at the clock. As a result, you get two clocks on the stream. By taking a screenshot and subtracting the smaller time from the larger time, you get the glass-to-glass latency of the pipeline.

In order to get an accurate reading, you must make sure that your measuring equipment is precise enough. You must use a high-refresh-rate timer and a similar refresh rate monitor. A good timer is the Linux date command executed like this:

while true; do echo -ne "`date +%H:%M:%S:%N`\r"; done

This timer’s refresh rate is CPU-bound. On my 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz processor the timer refreshes at around 650 Hz, but since I only have a 60 Hz monitor, then going above it doesn’t give us any more precision. This can be validated with a slow-motion camera that has at least 2x the speed of your monitor’s refresh rate. My phone camera has a 240 Hz slow-motion mode, and by recording the timestamp changes of two timer frames, I was able to validate that two timer draws were 15,678,490 ns = 15.68 ms apart, which converts to 63.7 Hz. That matches my 60 Hz monitor frequency well.

A simple, sample, unoptimized GStreamer/DeepStream pipeline you can use to stream a CSI camera to an HDMI-connected monitor for basic g2g latency measurements is the following:

gst-launch-1.0 nvarguscamerasrc sensor_id=0 ! 'video/x-raw(memory:NVMM),width=1280 height=720, framerate=60/1' ! nv3dsink

Keep in mind that with this g2g technique, your measurement uncertainty is usually one frame, so if you have a 30 fps pipeline, then the uncertainty is 33 ms. If you need to get sub-frame precision, then check out chapter 3. Specialized equipment.

Why the g2g method is good Link to heading

  1. If you can live with the 1-frame accuracy, then measuring latency with the glass-to-glass method is easy and accurate enough. This is because you do not need to change your streaming pipeline in any way to do this; you just point and shoot your existing hardware. You are not altering the latency by measuring it.
  2. It is easy to set up. It can be done with any camera and monitor setup.

Why the g2g method is bad Link to heading

  1. If you are streaming from a file or to a file, then this method does not work at all, because for the glass-to-glass method to work, you need, surprise surprise, two glasses: the lens and the monitor glass.
  2. Sometimes your producer and consumer are thousands of kilometers apart, making bringing them together impractical and inaccurate. By bringing the two ends of the pipeline together, we are changing the pipeline by reducing many network hops in between, thus rendering the result inaccurate.
  3. There is no granularity. You only get one measurement: the whole latency. What is slowing down my pipeline? Is it my network or my encoder? There is no way of knowing from g2g.
  4. The latency measurement uncertainty is +/- 1 frame, which can be high or low depending on the application.

2. Latency tracers Link to heading

Measuring latency with tracers involves exporting the timestamps of a packet as it passes through the pipeline. In GStreamer, this tracing logic is enabled with the built-in latency tracers. They enable you to measure the entire pipeline latency or the element latencies individually. To enable them, you can set the following environment variables:

export GST_DEBUG="GST_TRACER:7"
export GST_TRACERS="latency(flags=pipeline+element+reported)"
export GST_DEBUG_FILE="/app/latency.log"

This will generate a very large amount of logs in the latency.log file, on the order of hundreds to thousands of lines per second. Parsing this by hand is impractical. You can either pass this file into the gst-stats-1.0 command-line tool, which will output the min/mean/max element latencies, or you can write your own custom parser to plot the data however you like. I made a small Python script to plot the data with matplotlib which you can find here: GStreamer Latency Plotter. For an example let us measure the latency with tracers of the following GStreamer pipeline:

gst-launch-1.0 videotestsrc num-buffers=150 ! video/x-raw,format=I420,width=1280,height=720,framerate=30/1 ! videoconvert ! fakesink sync=false

This is your basic, run of the mill GStreamer test pipeline. Generate some HD buffers with videotestsrc at 30 frames per second and send them to fakesink, which just drops the buffers. Using the GStreamer Latency Plotter which I linked above, we can export the logfile into the following graphs:

Latency plot comparing pipeline element latencies over time
Example latency plot generated from GStreamer tracer logs.

You must keep in mind, that logging tracers is slow. You won’t get an accurate end-to-end latency measurement using this method - this is what g2g does best. However since tracers affect the latency of all the elements in the pipeline in the same way, they allow you to compare different elements between each other. You will get a reading on what are the slowest elements in your pipeline and where to direct your optimization efforts. For example: your RTP payloader takes as long as your software encoder? That is not right, something is probably misconfigured on the payloader.

Why the latency tracer method is good Link to heading

  1. Latency tracers give you the most granular option. You can determine exactly which element is holding up your pipeline.
  2. It applies to every streaming pipeline, even when you are streaming from a file or to a file.
  3. You can measure the latency of remote streams without physically pointing the camera at a clock, which is useful for edge devices that are hard to access.

Why the latency tracer method is bad Link to heading

  1. Tracers are computationally expensive. You change the output by measuring it, meaning you do not get an absolutely correct measurement.

3. Specialized equipment Link to heading

This measurement technique is reserved for the hard-core sub-frame latency people, like FPV camera system developers. I have not done this myself, but there is a detailed article about this method published on fpv.blue, which you can check out.