Measuring time in video

In trampoline jumping, air time is a part of final score. Jump air time is normally measured by photo diode positioned under bounce mat. Jumper crosses the infra red beam after landing till next bounce. Air time is sum of intervals when the beam is not crossed. Judge has to start measuring device just before the first move.

Sometimes measuring device didn’t work or air time judge didn’t start the device in the right time. Jumping routines had been recorded for possible later protests. When the air time measuring device didn’t give result, the air time had to be measured from the recorded video.

Video was recorded using USB web camera and encoded on PC. We found that time measured from video was significantly shorter than time measured by air time measuring device. The problem was in PC used as video encoder.

To clarify the problem, next chapter describe important video recording aspects.

Video is recorded by taking pictures in regular interval. Number of pictures taken in second is called frames per second (FPS). The FPS information is stored together with captured pictures and video player uses FPS information to determine time when to show next picture. When video recorder doesn’t keep declared FPS, time in video is damaged and there is no way to identify and correct the problem from video itself.

Figure Damaged time in video show video time damages caused by inaccurate FPS on recorder side. First column is ideal case when recorder keeps declared FPS and player show pictures in the right time. In the second column, video recorder take pictures later than declared. This causes shorter video duration in player. In the third column recorder takes pictures faster than declared FPS and video in player seems to be longer.

As demonstrated in previous chapter, it is necessary to keep FPS as precise as possible. Today’s PC include powerful processors that can encode video clip in shorter time than the clip lasts. PCs run multitasking operating systems like MS Windows. There are several tasks running on PC in the same time like video encoding, anti virus check, windows update, …​ Try to run task manager to see running processes when PC "does nothing". All running tasks share single processor or few processor cores. Operating system pretends that all tasks are running simultaneously by switching between them in short intervals. Running task receive processor for a while and then the processor is provided to another task. Ordinary multitasking operating systems don’t guarantee time when the task receives processor again.

So it is hard to keep FPS on ordinary multitasking operating system and encode live video even for powerful processors. This was our problem. Video recorder didn’t keep FPS and the video time was skewed. There are real time operating systems that guarantee maximal time interval when the task receives processor again, but they are not commonly installed on PCs.

Camcorders can record and encode live video without problem, because they do the recording only using specialized hardware.

There is an application that can display time for each picture (frame) in video clip and allow easily measure time. The problem was the application required AVI container. Camcorders don’t produce required AVI container. Video from camcorders was transcoded to AVI container required by the measuring application, but the transcoders reduced FPS so time measuring was less precise. Transcoding significantly slowed work.

I decided to create similar time measuring application that can handle video clips directly from camcorders.

Following text goes deeper into application architecture and it is hidden to end user.

There are wrappers over FFMpeg and QT for higher level languages, but rather incomplete, so backward moving picture by picture is difficult.

Video periodically contain complete pictures called I-frames. Other pictures between I-frames are stored as ``difference'' to previous I-frame (P-frames) and next I-frame (B-frames). When decoding required frame, jump to nearest previous I-frame and decode sequentially until required picture is reached. Going sequentially backwards is not natively possible.

Jumping is in time units that makes harder to estimate jump length. Number of I-frames per time unit (GOP) is not always available. Several tries to reach nearest I-frame is sometimes necessary. FFMpeg assign presentation timestamp to each picture and automatically decodes all pictures necessary to show required picture when going forward. Pictures are not always stored in the same order as they are captured.

Decoded pictures are saved into buffer to reduce jumping when going several pictures backward. The buffer takes some RAM.

Thanks for nice FFMpeg tutorial.