diff options
Diffstat (limited to 'Documentation/DocBook/media/v4l/dev-subdev.xml')
| -rw-r--r-- | Documentation/DocBook/media/v4l/dev-subdev.xml | 484 |
1 files changed, 484 insertions, 0 deletions
diff --git a/Documentation/DocBook/media/v4l/dev-subdev.xml b/Documentation/DocBook/media/v4l/dev-subdev.xml new file mode 100644 index 000000000..4f0ba58c9 --- /dev/null +++ b/Documentation/DocBook/media/v4l/dev-subdev.xml @@ -0,0 +1,484 @@ + <title>Sub-device Interface</title> + + <note> + <title>Experimental</title> + <para>This is an <link linkend="experimental">experimental</link> + interface and may change in the future.</para> + </note> + + <para>The complex nature of V4L2 devices, where hardware is often made of + several integrated circuits that need to interact with each other in a + controlled way, leads to complex V4L2 drivers. The drivers usually reflect + the hardware model in software, and model the different hardware components + as software blocks called sub-devices.</para> + + <para>V4L2 sub-devices are usually kernel-only objects. If the V4L2 driver + implements the media device API, they will automatically inherit from media + entities. Applications will be able to enumerate the sub-devices and discover + the hardware topology using the media entities, pads and links enumeration + API.</para> + + <para>In addition to make sub-devices discoverable, drivers can also choose + to make them directly configurable by applications. When both the sub-device + driver and the V4L2 device driver support this, sub-devices will feature a + character device node on which ioctls can be called to + <itemizedlist> + <listitem><para>query, read and write sub-devices controls</para></listitem> + <listitem><para>subscribe and unsubscribe to events and retrieve them</para></listitem> + <listitem><para>negotiate image formats on individual pads</para></listitem> + </itemizedlist> + </para> + + <para>Sub-device character device nodes, conventionally named + <filename>/dev/v4l-subdev*</filename>, use major number 81.</para> + + <section> + <title>Controls</title> + <para>Most V4L2 controls are implemented by sub-device hardware. Drivers + usually merge all controls and expose them through video device nodes. + Applications can control all sub-devices through a single interface.</para> + + <para>Complex devices sometimes implement the same control in different + pieces of hardware. This situation is common in embedded platforms, where + both sensors and image processing hardware implement identical functions, + such as contrast adjustment, white balance or faulty pixels correction. As + the V4L2 controls API doesn't support several identical controls in a single + device, all but one of the identical controls are hidden.</para> + + <para>Applications can access those hidden controls through the sub-device + node with the V4L2 control API described in <xref linkend="control" />. The + ioctls behave identically as when issued on V4L2 device nodes, with the + exception that they deal only with controls implemented in the sub-device. + </para> + + <para>Depending on the driver, those controls might also be exposed through + one (or several) V4L2 device nodes.</para> + </section> + + <section> + <title>Events</title> + <para>V4L2 sub-devices can notify applications of events as described in + <xref linkend="event" />. The API behaves identically as when used on V4L2 + device nodes, with the exception that it only deals with events generated by + the sub-device. Depending on the driver, those events might also be reported + on one (or several) V4L2 device nodes.</para> + </section> + + <section id="pad-level-formats"> + <title>Pad-level Formats</title> + + <warning><para>Pad-level formats are only applicable to very complex device that + need to expose low-level format configuration to user space. Generic V4L2 + applications do <emphasis>not</emphasis> need to use the API described in + this section.</para></warning> + + <note><para>For the purpose of this section, the term + <wordasword>format</wordasword> means the combination of media bus data + format, frame width and frame height.</para></note> + + <para>Image formats are typically negotiated on video capture and + output devices using the format and <link + linkend="vidioc-subdev-g-selection">selection</link> ioctls. The + driver is responsible for configuring every block in the video + pipeline according to the requested format at the pipeline input + and/or output.</para> + + <para>For complex devices, such as often found in embedded systems, + identical image sizes at the output of a pipeline can be achieved using + different hardware configurations. One such example is shown on + <xref linkend="pipeline-scaling" />, where + image scaling can be performed on both the video sensor and the host image + processing hardware.</para> + + <figure id="pipeline-scaling"> + <title>Image Format Negotiation on Pipelines</title> + <mediaobject> + <imageobject> + <imagedata fileref="pipeline.pdf" format="PS" /> + </imageobject> + <imageobject> + <imagedata fileref="pipeline.png" format="PNG" /> + </imageobject> + <textobject> + <phrase>High quality and high speed pipeline configuration</phrase> + </textobject> + </mediaobject> + </figure> + + <para>The sensor scaler is usually of less quality than the host scaler, but + scaling on the sensor is required to achieve higher frame rates. Depending + on the use case (quality vs. speed), the pipeline must be configured + differently. Applications need to configure the formats at every point in + the pipeline explicitly.</para> + + <para>Drivers that implement the <link linkend="media-controller-intro">media + API</link> can expose pad-level image format configuration to applications. + When they do, applications can use the &VIDIOC-SUBDEV-G-FMT; and + &VIDIOC-SUBDEV-S-FMT; ioctls. to negotiate formats on a per-pad basis.</para> + + <para>Applications are responsible for configuring coherent parameters on + the whole pipeline and making sure that connected pads have compatible + formats. The pipeline is checked for formats mismatch at &VIDIOC-STREAMON; + time, and an &EPIPE; is then returned if the configuration is + invalid.</para> + + <para>Pad-level image format configuration support can be tested by calling + the &VIDIOC-SUBDEV-G-FMT; ioctl on pad 0. If the driver returns an &EINVAL; + pad-level format configuration is not supported by the sub-device.</para> + + <section> + <title>Format Negotiation</title> + + <para>Acceptable formats on pads can (and usually do) depend on a number + of external parameters, such as formats on other pads, active links, or + even controls. Finding a combination of formats on all pads in a video + pipeline, acceptable to both application and driver, can't rely on formats + enumeration only. A format negotiation mechanism is required.</para> + + <para>Central to the format negotiation mechanism are the get/set format + operations. When called with the <structfield>which</structfield> argument + set to <constant>V4L2_SUBDEV_FORMAT_TRY</constant>, the + &VIDIOC-SUBDEV-G-FMT; and &VIDIOC-SUBDEV-S-FMT; ioctls operate on a set of + formats parameters that are not connected to the hardware configuration. + Modifying those 'try' formats leaves the device state untouched (this + applies to both the software state stored in the driver and the hardware + state stored in the device itself).</para> + + <para>While not kept as part of the device state, try formats are stored + in the sub-device file handles. A &VIDIOC-SUBDEV-G-FMT; call will return + the last try format set <emphasis>on the same sub-device file + handle</emphasis>. Several applications querying the same sub-device at + the same time will thus not interact with each other.</para> + + <para>To find out whether a particular format is supported by the device, + applications use the &VIDIOC-SUBDEV-S-FMT; ioctl. Drivers verify and, if + needed, change the requested <structfield>format</structfield> based on + device requirements and return the possibly modified value. Applications + can then choose to try a different format or accept the returned value and + continue.</para> + + <para>Formats returned by the driver during a negotiation iteration are + guaranteed to be supported by the device. In particular, drivers guarantee + that a returned format will not be further changed if passed to an + &VIDIOC-SUBDEV-S-FMT; call as-is (as long as external parameters, such as + formats on other pads or links' configuration are not changed).</para> + + <para>Drivers automatically propagate formats inside sub-devices. When a + try or active format is set on a pad, corresponding formats on other pads + of the same sub-device can be modified by the driver. Drivers are free to + modify formats as required by the device. However, they should comply with + the following rules when possible: + <itemizedlist> + <listitem><para>Formats should be propagated from sink pads to source pads. + Modifying a format on a source pad should not modify the format on any + sink pad.</para></listitem> + <listitem><para>Sub-devices that scale frames using variable scaling factors + should reset the scale factors to default values when sink pads formats + are modified. If the 1:1 scaling ratio is supported, this means that + source pads formats should be reset to the sink pads formats.</para></listitem> + </itemizedlist> + </para> + + <para>Formats are not propagated across links, as that would involve + propagating them from one sub-device file handle to another. Applications + must then take care to configure both ends of every link explicitly with + compatible formats. Identical formats on the two ends of a link are + guaranteed to be compatible. Drivers are free to accept different formats + matching device requirements as being compatible.</para> + + <para><xref linkend="sample-pipeline-config" /> + shows a sample configuration sequence for the pipeline described in + <xref linkend="pipeline-scaling" /> (table + columns list entity names and pad numbers).</para> + + <table pgwide="0" frame="none" id="sample-pipeline-config"> + <title>Sample Pipeline Configuration</title> + <tgroup cols="3"> + <colspec colname="what"/> + <colspec colname="sensor-0 format" /> + <colspec colname="frontend-0 format" /> + <colspec colname="frontend-1 format" /> + <colspec colname="scaler-0 format" /> + <colspec colname="scaler-0 compose" /> + <colspec colname="scaler-1 format" /> + <thead> + <row> + <entry></entry> + <entry>Sensor/0 format</entry> + <entry>Frontend/0 format</entry> + <entry>Frontend/1 format</entry> + <entry>Scaler/0 format</entry> + <entry>Scaler/0 compose selection rectangle</entry> + <entry>Scaler/1 format</entry> + </row> + </thead> + <tbody valign="top"> + <row> + <entry>Initial state</entry> + <entry>2048x1536/SGRBG8_1X8</entry> + <entry>(default)</entry> + <entry>(default)</entry> + <entry>(default)</entry> + <entry>(default)</entry> + <entry>(default)</entry> + </row> + <row> + <entry>Configure frontend sink format</entry> + <entry>2048x1536/SGRBG8_1X8</entry> + <entry><emphasis>2048x1536/SGRBG8_1X8</emphasis></entry> + <entry><emphasis>2046x1534/SGRBG8_1X8</emphasis></entry> + <entry>(default)</entry> + <entry>(default)</entry> + <entry>(default)</entry> + </row> + <row> + <entry>Configure scaler sink format</entry> + <entry>2048x1536/SGRBG8_1X8</entry> + <entry>2048x1536/SGRBG8_1X8</entry> + <entry>2046x1534/SGRBG8_1X8</entry> + <entry><emphasis>2046x1534/SGRBG8_1X8</emphasis></entry> + <entry><emphasis>0,0/2046x1534</emphasis></entry> + <entry><emphasis>2046x1534/SGRBG8_1X8</emphasis></entry> + </row> + <row> + <entry>Configure scaler sink compose selection</entry> + <entry>2048x1536/SGRBG8_1X8</entry> + <entry>2048x1536/SGRBG8_1X8</entry> + <entry>2046x1534/SGRBG8_1X8</entry> + <entry>2046x1534/SGRBG8_1X8</entry> + <entry><emphasis>0,0/1280x960</emphasis></entry> + <entry><emphasis>1280x960/SGRBG8_1X8</emphasis></entry> + </row> + </tbody> + </tgroup> + </table> + + <para> + <orderedlist> + <listitem><para>Initial state. The sensor source pad format is + set to its native 3MP size and V4L2_MBUS_FMT_SGRBG8_1X8 + media bus code. Formats on the host frontend and scaler sink + and source pads have the default values, as well as the + compose rectangle on the scaler's sink pad.</para></listitem> + + <listitem><para>The application configures the frontend sink + pad format's size to 2048x1536 and its media bus code to + V4L2_MBUS_FMT_SGRBG_1X8. The driver propagates the format to + the frontend source pad.</para></listitem> + + <listitem><para>The application configures the scaler sink pad + format's size to 2046x1534 and the media bus code to + V4L2_MBUS_FMT_SGRBG_1X8 to match the frontend source size and + media bus code. The media bus code on the sink pad is set to + V4L2_MBUS_FMT_SGRBG_1X8. The driver propagates the size to the + compose selection rectangle on the scaler's sink pad, and the + format to the scaler source pad.</para></listitem> + + <listitem><para>The application configures the size of the compose + selection rectangle of the scaler's sink pad 1280x960. The driver + propagates the size to the scaler's source pad + format.</para></listitem> + + </orderedlist> + </para> + + <para>When satisfied with the try results, applications can set the active + formats by setting the <structfield>which</structfield> argument to + <constant>V4L2_SUBDEV_FORMAT_ACTIVE</constant>. Active formats are changed + exactly as try formats by drivers. To avoid modifying the hardware state + during format negotiation, applications should negotiate try formats first + and then modify the active settings using the try formats returned during + the last negotiation iteration. This guarantees that the active format + will be applied as-is by the driver without being modified. + </para> + </section> + + <section id="v4l2-subdev-selections"> + <title>Selections: cropping, scaling and composition</title> + + <para>Many sub-devices support cropping frames on their input or output + pads (or possible even on both). Cropping is used to select the area of + interest in an image, typically on an image sensor or a video decoder. It can + also be used as part of digital zoom implementations to select the area of + the image that will be scaled up.</para> + + <para>Crop settings are defined by a crop rectangle and represented in a + &v4l2-rect; by the coordinates of the top left corner and the rectangle + size. Both the coordinates and sizes are expressed in pixels.</para> + + <para>As for pad formats, drivers store try and active + rectangles for the selection targets <xref + linkend="v4l2-selections-common" />.</para> + + <para>On sink pads, cropping is applied relative to the + current pad format. The pad format represents the image size as + received by the sub-device from the previous block in the + pipeline, and the crop rectangle represents the sub-image that + will be transmitted further inside the sub-device for + processing.</para> + + <para>The scaling operation changes the size of the image by + scaling it to new dimensions. The scaling ratio isn't specified + explicitly, but is implied from the original and scaled image + sizes. Both sizes are represented by &v4l2-rect;.</para> + + <para>Scaling support is optional. When supported by a subdev, + the crop rectangle on the subdev's sink pad is scaled to the + size configured using the &VIDIOC-SUBDEV-S-SELECTION; IOCTL + using <constant>V4L2_SEL_TGT_COMPOSE</constant> + selection target on the same pad. If the subdev supports scaling + but not composing, the top and left values are not used and must + always be set to zero.</para> + + <para>On source pads, cropping is similar to sink pads, with the + exception that the source size from which the cropping is + performed, is the COMPOSE rectangle on the sink pad. In both + sink and source pads, the crop rectangle must be entirely + contained inside the source image size for the crop + operation.</para> + + <para>The drivers should always use the closest possible + rectangle the user requests on all selection targets, unless + specifically told otherwise. + <constant>V4L2_SEL_FLAG_GE</constant> and + <constant>V4L2_SEL_FLAG_LE</constant> flags may be + used to round the image size either up or down. <xref + linkend="v4l2-selection-flags" /></para> + </section> + + <section> + <title>Types of selection targets</title> + + <section> + <title>Actual targets</title> + + <para>Actual targets (without a postfix) reflect the actual + hardware configuration at any point of time. There is a BOUNDS + target corresponding to every actual target.</para> + </section> + + <section> + <title>BOUNDS targets</title> + + <para>BOUNDS targets is the smallest rectangle that contains all + valid actual rectangles. It may not be possible to set the actual + rectangle as large as the BOUNDS rectangle, however. This may be + because e.g. a sensor's pixel array is not rectangular but + cross-shaped or round. The maximum size may also be smaller than the + BOUNDS rectangle.</para> + </section> + + </section> + + <section> + <title>Order of configuration and format propagation</title> + + <para>Inside subdevs, the order of image processing steps will + always be from the sink pad towards the source pad. This is also + reflected in the order in which the configuration must be + performed by the user: the changes made will be propagated to + any subsequent stages. If this behaviour is not desired, the + user must set + <constant>V4L2_SEL_FLAG_KEEP_CONFIG</constant> flag. This + flag causes no propagation of the changes are allowed in any + circumstances. This may also cause the accessed rectangle to be + adjusted by the driver, depending on the properties of the + underlying hardware.</para> + + <para>The coordinates to a step always refer to the actual size + of the previous step. The exception to this rule is the source + compose rectangle, which refers to the sink compose bounds + rectangle --- if it is supported by the hardware.</para> + + <orderedlist> + <listitem><para>Sink pad format. The user configures the sink pad + format. This format defines the parameters of the image the + entity receives through the pad for further processing.</para></listitem> + + <listitem><para>Sink pad actual crop selection. The sink pad crop + defines the crop performed to the sink pad format.</para></listitem> + + <listitem><para>Sink pad actual compose selection. The size of the + sink pad compose rectangle defines the scaling ratio compared + to the size of the sink pad crop rectangle. The location of + the compose rectangle specifies the location of the actual + sink compose rectangle in the sink compose bounds + rectangle.</para></listitem> + + <listitem><para>Source pad actual crop selection. Crop on the source + pad defines crop performed to the image in the sink compose + bounds rectangle.</para></listitem> + + <listitem><para>Source pad format. The source pad format defines the + output pixel format of the subdev, as well as the other + parameters with the exception of the image width and height. + Width and height are defined by the size of the source pad + actual crop selection.</para></listitem> + </orderedlist> + + <para>Accessing any of the above rectangles not supported by the + subdev will return <constant>EINVAL</constant>. Any rectangle + referring to a previous unsupported rectangle coordinates will + instead refer to the previous supported rectangle. For example, + if sink crop is not supported, the compose selection will refer + to the sink pad format dimensions instead.</para> + + <figure id="subdev-image-processing-crop"> + <title>Image processing in subdevs: simple crop example</title> + <mediaobject> + <imageobject> + <imagedata fileref="subdev-image-processing-crop.svg" + format="SVG" scale="200" /> + </imageobject> + </mediaobject> + </figure> + + <para>In the above example, the subdev supports cropping on its + sink pad. To configure it, the user sets the media bus format on + the subdev's sink pad. Now the actual crop rectangle can be set + on the sink pad --- the location and size of this rectangle + reflect the location and size of a rectangle to be cropped from + the sink format. The size of the sink crop rectangle will also + be the size of the format of the subdev's source pad.</para> + + <figure id="subdev-image-processing-scaling-multi-source"> + <title>Image processing in subdevs: scaling with multiple sources</title> + <mediaobject> + <imageobject> + <imagedata fileref="subdev-image-processing-scaling-multi-source.svg" + format="SVG" scale="200" /> + </imageobject> + </mediaobject> + </figure> + + <para>In this example, the subdev is capable of first cropping, + then scaling and finally cropping for two source pads + individually from the resulting scaled image. The location of + the scaled image in the cropped image is ignored in sink compose + target. Both of the locations of the source crop rectangles + refer to the sink scaling rectangle, independently cropping an + area at location specified by the source crop rectangle from + it.</para> + + <figure id="subdev-image-processing-full"> + <title>Image processing in subdevs: scaling and composition + with multiple sinks and sources</title> + <mediaobject> + <imageobject> + <imagedata fileref="subdev-image-processing-full.svg" + format="SVG" scale="200" /> + </imageobject> + </mediaobject> + </figure> + + <para>The subdev driver supports two sink pads and two source + pads. The images from both of the sink pads are individually + cropped, then scaled and further composed on the composition + bounds rectangle. From that, two independent streams are cropped + and sent out of the subdev from the source pads.</para> + + </section> + + </section> + + &sub-subdev-formats; |