Video for Linux Two - Devices

Video for Linux Two (V4L2) is a set of APIs and standards for handling video devices on Linux. Video for Linux Two is a replacement for the Video for Linux API that comes with the kernel.

This is the top level V4L2 device driver API document. This document descibes how to setup up V4L2 device nodes, and also specifies some data structures and ioctl codes common to different device types. Supplementing this file are specification documents for each of the different V4L2 device types. Links to these documents are the table in the next section.
 

Device Types

V4L2 devices are all Unix char type devices, with a major device number of 81. V4L2 devices have the following names in the /dev tree. (More device types can be added as needed.)
 

Device Name			Type of Device
/dev/video			Video capture interface
/dev/vfx			Video effects interface
/dev/codec			Video codec interface
/dev/vout			Video output interface
/dev/radio			AM/FM radio devices
/dev/vtx			Teletext interface chips
/dev/vbi			Data services interface

Device Minor Numbers

For example, a hypothetical driver for a video capture/compression card might export both video capture and video codec interfaces. If you had two such boards in the system, you would need two minors for the capture devices, and two for the codec devices. The parameters might be:

The device nodes might be:
 

node		major, minor
/dev/video0		81, 0
/dev/video1		81, 1
/dev/codec0		81, 10
/dev/codec1		81, 11

There is no specification for what minor numbers to use with what devices. It is entirely up to the administrator of a system. Obviously, all minor numbers for all devices for all V4L2 drivers must be unique. It is not specified how a driver will handle the case where a minor number is already in use. It may fail the module load. It is expected that driver writers will handle the failure gracefully, and only the offending device will be unavailable.

It is not specified what happens when a needed minor number is not on the command line. It is expected that driver writers will include sensible default values so if you have only one V4L2 driver, it will work without manually assigned minor numbers. However, administrators really should specify the minor numbers. In particular, if you have two V4L2 drivers, you will need to specify minors to guarantee that there are no conflicts.

Video for Linux Two applications open and scan the devices to find what they are looking for. Capability queries define what each interface supports.
 

Common V4L2 API Elements

Query Capabilities - VIDIOC_QUERYCAP

All V4L2 device types support this ioctl. This ioctl call is used to obtain the capability information for a device. The driver will fill in a struct v4l2_capability object.
 

The exact meaning of the flags will vary slightly depending on the type of device.

Note that the minimum and maximum image capture dimensions are for comparison purposes only. The actual maximum size you can capture may depend on the capture parameters, including the pixel format, compression (if any), the video standard (PAL is higher resolution than NTSC), and possibly other parameters such as the amount of on-board memory or bus bandwidth on your system. Same applies to maximum frame rate. The minimum and maximum sizes do not imply that all combinations of height/width within the range are possible.
 
 

Multiple Opens per Device

In general, V4L2 devices can be opened more than once simultaneously. Some devices may have limitations in that regard. See the documentation for the particular device type. Specifically, some devices will only be able to support I/O operations on one open at a time.

No-I/O Opens

However, it is still desirable to support additional simultaneous opens for control or administration purposes. So V4L2 has the concept of no-I/O opens. No-I/O opens can do many 'get' type ioctls, change UI controls, and do certain other non-disruptive ioctls depending on the type of device. The purpose is to be able to create standard video control panels that can run concurrently along side an application doing video input or output. Uses for no-I/O opens include UI control panels, changing channels, and performance monitoring. The application indicates an open will not be used for I/O by passing the O_NOIO flags to open().
 
 

The Stream Data Format Structure - struct v4l2_format

V4L2 devices handle multimedia data in the form of streams consisting of buffers containing formatted data. One device can have multiple simultaneous streams. Most streams are video images, but other types are possible. The v4l2_format structure incorporates a union to handle different format structures, and so that more can be added later. The application must always set the type field to indicate which type of format is being used, and which stream it applies to.
 

Video Image Format Structure - struct v4l2_pix_format

This structure completely defines the layout and format of an image or image buffer, including width, height, depth, pixel format, stride, and total size.
 

The depth is the amount of space allocated in the buffer per pixel, in bits. Each pixel format has a single corresponding depth value.

The pixel information may not fill all bits allocated, e.g. RGB555 and RGB32. Leftover bits are undefined. For planar YUV formats the depth is the average number of bits per pixel. For example, YUV420 is eight bits per component, but the U and V planes are 1/4 the size of the Y plane so the average bits per pixel is 12. The standard pixelformat values are listed in the table below. See the standard pixel format specification for more detailed information concerning pixel formats. Some drivers may support formats not listed here.

Bytesperline is the number of bytes of memory between two adjacent lines. Since most of the time it's not needed, bytesperline only applies if the FMT_FLAG_BYTESPERLINE flag is set. Otherwise the field is undefined and must be ignored. For YUV planar formats, it's the stride of the Y plane. When there is line padding, data begins at the start of the buffer, and pad bytes are at the end of lines. The values of pad bytes are undefined.

Sizeimage is usually (width*height*depth)/8 for uncompressed images, but it's different if bytesperline is used since there could be some padding between lines.
 

Memory-Mapping Device Buffers - VIDIOC_REQBUFS, VIDIOC_QUERYBUF

These ioctls implement a general-purpose protocol for using mmap() to make driver buffers or device memory buffers accessible to the application for I/O.

To map buffers, the application first calls VIDIOC_REQBUFS with a struct v4l2_requestbuffers filled in with the number and type of the buffers that it wants. Upon return the driver will fill in how many buffers it will allow to be mapped, and possibly modify the type field as well. The application should make sure the granted count and type are ok. Note that it is possible that the driver may have a lower limit on the number of buffers required for streaming data. If the driver returns a count value greater than the requested number then that is a lower limit and the application needs to allocate at least that many buffers.

In general, drivers can support many sets of buffers. Each set of buffers is identified by a unique buffer type value. The sets are independent and each set can hold a different type of data.

To map the buffers call VIDIOC_QUERYBUF for each buffer to get the details about the buffer, and call mmap() to map it. VIDIOC_QUERYBUF takes a struct v4l2_buffer object with the index and type fields filled in to indicate which buffer is being queried. The type field must be filled in with the value from the struct v4l2_requestbuffers object. Valid index values range from 0 to count - 1, inclusive. Upon return, the offset and length fields will be filled in with the values that must be passed to mmap() to map the buffer. Only pass offset and length values received from the VIDIOC_QUERYBUF ioctl; the driver needs the exact values it indicated to correctly map the requested buffer. Use MAP_SHARED if the buffer will be used by a forked process. When the buffer is no longer needed the application must call munmap(). The application must munmap() before closing the driver.

The driver will allocate buffers that will be located in system memory on the mmap() call, and free them on the munmap() call, if possible. These buffers are allocated in physical memory, so applications should always unmap buffers when they are not needed to free up system resources.

A common use for memory-mapped buffers is for streaming data to and from drivers with minimum overhead. Drivers will maintain internal queues of buffers and process them asynchonously. The ioctl commands for doing that are described in the device-specific documents. Memory-mapping can also be used for a variety of other purposes. Drivers can define hardware-specific memory buffer types if needed. Use V4L2_BUF_TYPE_PRIVATE_BASE and greater values for such buffer types.

The size of the buffers often depends on the format of the data. For example, the size of capture buffers is determined by the capture image format set by the VIDIOC_S_FMT ioctl. Therefore, once buffers are mapped, that can place a constraint on how the format can be changed. It is recommended to unmap related buffers before changing the format, and to re-request buffers afterward.
 

struct v4l2_requestbuffers

int count		The number of buffers requested or granted
__u32 type		The requested/granted buffer type
__u32 reserved[2]		reserved

struct v4l2_buffer

int index		Which buffer number this is or which to query
__u32 type		The buffer type
__u32 offset		Offset parameter to pass to mmap() to allocate this buffer
__u32 length		Length parameter to pass to mmap() to allocate this buffer and the physical length of the buffer
__u32 bytesused		The number of bytes of data in the buffer
__u32 flags		Flags concerning current status of the buffer or the data in the buffer
__s64 timestamp		Timestamp for the frame
struct v4l2_timecode timecode		The timecode for this frame, if any
__u32 sequence		This number of this frame in the sequence. Usually set only by capture drivers.
__u32 reserved[4]		reserved

Buffer type codes and flags for the type field of struct v4l2_buffer

V4L2_BUF_TYPE_field		A bitmask to isolate the buffer type field
V4L2_BUF_TYPE_CAPTURE		The buffer is for streaming capture
V4L2_BUF_TYPE_CODECIN		The buffer is an input buffer for an image transform operation
V4L2_BUF_TYPE_CODECOUT		The buffer is an output buffer for an image transform operation
V4L2_BUF_TYPE_EFFECTSIN		Input for an effects device operation
V4L2_BUF_TYPE_EFFECTSIN2		Input for an effects device operation
V4L2_BUF_TYPE_EFFECTSOUT		Output for an effects device operation
V4L2_BUF_TYPE_VIDEOOUT		The buffer is for streaming video output
V4L2_BUF_TYPE_PRIVATE_BASE		Starting value for driver private buffer types.
V4L2_BUF_ATTR_DEVICEMEM		(flag) The buffer is physically located in the device's on-board memory

Flags for the flags field of struct v4l2_buffer

V4L2_BUF_ATTR_MAPPED		The buffer is currently memory-mapped
V4L2_BUF_FLAG_QUEUED		The buffer is queued for processing (set by the driver on VIDIOC_QBUF)
V4L2_BUF_FLAG_DONE		The buffer has data in it (set by the driver when the frame is processed, cleared by the driver on VIDIOC_QBUF)
V4L2_BUF_FLAG_KEYFRAME		This frame is a keyframe or I frame (always set for uncompressed)
V4L2_BUF_FLAG_PFRAME		This frame is a predicted frame (only for some compressions)
V4L2_BUF_FLAG_BFRAME		This frame is a bidirectionally predicted frame (only for some compressions)
V4L2_BUF_FLAG_TOPFIELD		This image is a top (odd) field in a field-alternating stream
V4L2_BUF_FLAG_BOTFIELD		This image is a bottom (even) field in a field-alternating stream
V4L2_BUF_FLAG_TIMECODE		The timecode field is valid.

 

Timecodes

The struct v4l2_timecode structure is designed to hold an SMPTE timecode, or similar timecode. The hours, minutes, seconds and frames fields are normal binary values, not BCD.
 

Controls - VIDIOC_QUERYCTRL, VIDIOC_QUERYMENU, VIDIOC_G_CTRL, VIDIOC_S_CTRL

Devices typically have a number of user-settable controls such as brightness, saturation and so on, which would be presented to the user on a graphical user interface (GUI). But, different devices will have different controls available, and furthermore, the range of possible values, and the default value will vary from device to device. These ioctls provide the information and mechanism to create a nice user interface for these controls that will work correctly with any device.

All controls are accessed using an ID value. V4L2 defines a standard set of control IDs. Drivers can also implement their own device-specific controls using V4L2_CID_PRIVATE_BASE and increasing values. The pre-defined control IDs have the prefix V4L2_CID_, and are listed below. The ID is used when querying the properties of a control, and when getting or setting the current value.

The VIDIOC_QUERYCTRL ioctl fills in a struct v4l2_queryctrl object which describes the parameters of a control. The application must fill in the id field before making the call, and the ioctl will fill in the rest of the structure. If the specified control is not supported the ioctl returns the EINVAL error code. This interface allows for four types of controls: integer-valued, boolean-valued, menus, and buttons. An integer-valued control is usually represented by a slider or thumbwheel GUI element. A boolean-valued control is usually represented by a checkbox or toggle GUI element. A menu control is usually represented by a drop-down menu or radio buttons. A button control is represented by a button which performs some action when clicked. The type field indicates the data type of the control.

It is possible to enumerate the controls, call VIDIOC_QUERYCTRL with successive id values starting from V4L2_CID_BASE, and stop when the driver returns the EINVAL error code. After each call to VIDIOC_QUERYCTRL, check the flags field. If the control is supported by the driver then the V4L2_CTRL_FLAG_DISABLED bit will be zero. To enumerate the driver-private controls, use the same algorithm, but use successive id values starting from V4L2_CID_PRIVATE_BASE. On any control, if the V4L2_CTRL_FLAG_GRABBED flag is set, then the value of the control cannot be changed at this time, or using this file descriptor.

Control types:
 

To get and set the current value of a control, call VIDIOC_G_CTRL or VIDIOC_S_CTRL. These ioctls use a struct v4l2_control. Fill in the id field, and, if setting a value, the value field. All controls are changeable by a non-capturing open. If the id is not a supported control the driver returns EINVAL. If the value is not a legal value the driver may use the nearest legal value, or return ERANGE. If the control is read-only for some reason the driver will return EBUSY on a set attempt.
 

Device Performance - VIDIOC_G_PERF

The VIDIOC_G_PERF ioctl fills in a struct v4l2_performance object. Drivers will keep running tallies of the number of frames processed or dropped. Applications can compute derived values such as frames/second by polling VIDIOC_G_PERF at periodic intervals.