Measuring time in video
Measuring time from video is easy with suitable application, but special care has to be taken to how the video is recorded. Improper recording can make video useless. This article explain problems with recording video on ordinary PC and describe application for measuring time in video. The application is ready to download and try.
Updated synchronized trampoline script according to rules since 2017
Air time
In trampoline jumping, air time is a part of final score. Jump air time is normally measured by photo diode positioned under bounce mat. Jumper crosses the infra red beam after landing till next bounce. Air time is sum of intervals when the beam is not crossed. Judge has to start measuring device just before the first move.
![Measuring air time](/images/trampoline_jumping.gif)
Sometimes measuring device didn’t work or air time judge didn’t start the device in the right time. Jumping routines had been recorded for possible later protests. When the air time measuring device didn’t give result, the air time had to be measured from the recorded video.
Video was recorded using USB web camera and encoded on PC. We found that time measured from video was significantly shorter than time measured by air time measuring device. The problem was in PC used as video encoder.
To clarify the problem, next chapter describe important video recording aspects.
Video recording
Video is recorded by taking pictures in regular interval. Number of pictures taken in second is called frames per second (FPS). The FPS information is stored together with captured pictures and video player uses FPS information to determine time when to show next picture. When video recorder doesn’t keep declared FPS, time in video is damaged and there is no way to identify and correct the problem from video itself.
![Damaged time in video](/images/damaged_fps.png)
Figure Damaged time in video show video time damages caused by inaccurate FPS on recorder side. First column is ideal case when recorder keeps declared FPS and player show pictures in the right time. In the second column, video recorder take pictures later than declared. This causes shorter video duration in player. In the third column recorder takes pictures faster than declared FPS and video in player seems to be longer.
Video recording on PC
As demonstrated in previous chapter, it is necessary to keep FPS as precise as possible. Today’s PC include powerful processors that can encode video clip in shorter time than the clip lasts. PCs run multitasking operating systems like MS Windows. There are several tasks running on PC in the same time like video encoding, anti virus check, windows update, … Try to run task manager to see running processes when PC "does nothing". All running tasks share single processor or few processor cores. Operating system pretends that all tasks are running simultaneously by switching between them in short intervals. Running task receive processor for a while and then the processor is provided to another task. Ordinary multitasking operating systems don’t guarantee time when the task receives processor again.
So it is hard to keep FPS on ordinary multitasking operating system and encode live video even for powerful processors. This was our problem. Video recorder didn’t keep FPS and the video time was skewed. There are real time operating systems that guarantee maximal time interval when the task receives processor again, but they are not commonly installed on PCs.
Camcorders can record and encode live video without problem, because they do the recording only using specialized hardware.
Taking time from video
There is an application that can display time for each picture (frame) in video clip and allow easily measure time. The problem was the application required AVI container. Camcorders don’t produce required AVI container. Video from camcorders was transcoded to AVI container required by the measuring application, but the transcoders reduced FPS so time measuring was less precise. Transcoding significantly slowed work.
I decided to create similar time measuring application that can handle video clips directly from camcorders.
Application requirements
-
GUI application to measure several time intervals in video clip.
-
User marks begin and end pictures of intervals. Application shows total time as sum of measured intervals.
-
Save interval borders for later inspection.
-
Handle as much as possible video formats generated by camcorders.
-
Allow to move in video picture by picture forward and backward.
-
Use available libraries to reduce development time.
-
GUI design inspired by existing application.
-
Multiplatform at least for Windows and Linux (nice to have).
-
Use case:
-
Record video clips on camcorders recording to SD cards. The more FPS the more precise measurement.
-
In case of air time measuring problem, remove SD card from camcorder and insert another.
-
Measure time from video on removed SD card.
-
Following text goes deeper into application architecture and it is hidden to end user.
Selected libraries and programming language
-
FFMpeg - nice video library that can natively handle large amount of video format. Allows to move forward and backward in video.
-
QT library - nice multiplatform GUI library with designer.
-
C/C++ to glue FFMpeg with QT.
There are wrappers over FFMpeg and QT for higher level languages, but rather incomplete, so backward moving picture by picture is difficult.
Decoding pictures using FFMpeg library
Video periodically contain complete pictures called I-frames. Other pictures between I-frames are stored as ``difference'' to previous I-frame (P-frames) and next I-frame (B-frames). When decoding required frame, jump to nearest previous I-frame and decode sequentially until required picture is reached. Going sequentially backwards is not natively possible.
Jumping is in time units that makes harder to estimate jump length. Number of I-frames per time unit (GOP) is not always available. Several tries to reach nearest I-frame is sometimes necessary. FFMpeg assign presentation timestamp to each picture and automatically decodes all pictures necessary to show required picture when going forward. Pictures are not always stored in the same order as they are captured.
Decoded pictures are saved into buffer to reduce jumping when going several pictures backward. The buffer takes some RAM.
Thanks for nice FFMpeg tutorial.