1.. SPDX-License-Identifier: GPL-2.0 2 3.. _decoder: 4 5************************************************* 6Memory-to-Memory Stateful Video Decoder Interface 7************************************************* 8 9A stateful video decoder takes complete chunks of the bytestream (e.g. Annex-B 10H.264/HEVC stream, raw VP8/9 stream) and decodes them into raw video frames in 11display order. The decoder is expected not to require any additional information 12from the client to process these buffers. 13 14Performing software parsing, processing etc. of the stream in the driver in 15order to support this interface is strongly discouraged. In case such 16operations are needed, use of the Stateless Video Decoder Interface (in 17development) is strongly advised. 18 19Conventions and Notations Used in This Document 20=============================================== 21 221. The general V4L2 API rules apply if not specified in this document 23 otherwise. 24 252. The meaning of words "must", "may", "should", etc. is as per `RFC 26 2119 <https://tools.ietf.org/html/rfc2119>`_. 27 283. All steps not marked "optional" are required. 29 304. :c:func:`VIDIOC_G_EXT_CTRLS` and :c:func:`VIDIOC_S_EXT_CTRLS` may be used 31 interchangeably with :c:func:`VIDIOC_G_CTRL` and :c:func:`VIDIOC_S_CTRL`, 32 unless specified otherwise. 33 345. Single-planar API (see :ref:`planar-apis`) and applicable structures may be 35 used interchangeably with multi-planar API, unless specified otherwise, 36 depending on decoder capabilities and following the general V4L2 guidelines. 37 386. i = [a..b]: sequence of integers from a to b, inclusive, i.e. i = 39 [0..2]: i = 0, 1, 2. 40 417. Given an ``OUTPUT`` buffer A, then A' represents a buffer on the ``CAPTURE`` 42 queue containing data that resulted from processing buffer A. 43 44.. _decoder-glossary: 45 46Glossary 47======== 48 49CAPTURE 50 the destination buffer queue; for decoders, the queue of buffers containing 51 decoded frames; for encoders, the queue of buffers containing an encoded 52 bytestream; ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or 53 ``V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE``; data is captured from the hardware 54 into ``CAPTURE`` buffers. 55 56client 57 the application communicating with the decoder or encoder implementing 58 this interface. 59 60coded format 61 encoded/compressed video bytestream format (e.g. H.264, VP8, etc.); see 62 also: raw format. 63 64coded height 65 height for given coded resolution. 66 67coded resolution 68 stream resolution in pixels aligned to codec and hardware requirements; 69 typically visible resolution rounded up to full macroblocks; 70 see also: visible resolution. 71 72coded width 73 width for given coded resolution. 74 75decode order 76 the order in which frames are decoded; may differ from display order if the 77 coded format includes a feature of frame reordering; for decoders, 78 ``OUTPUT`` buffers must be queued by the client in decode order; for 79 encoders ``CAPTURE`` buffers must be returned by the encoder in decode order. 80 81destination 82 data resulting from the decode process; see ``CAPTURE``. 83 84display order 85 the order in which frames must be displayed; for encoders, ``OUTPUT`` 86 buffers must be queued by the client in display order; for decoders, 87 ``CAPTURE`` buffers must be returned by the decoder in display order. 88 89DPB 90 Decoded Picture Buffer; an H.264/HEVC term for a buffer that stores a decoded 91 raw frame available for reference in further decoding steps. 92 93EOS 94 end of stream. 95 96IDR 97 Instantaneous Decoder Refresh; a type of a keyframe in an H.264/HEVC-encoded 98 stream, which clears the list of earlier reference frames (DPBs). 99 100keyframe 101 an encoded frame that does not reference frames decoded earlier, i.e. 102 can be decoded fully on its own. 103 104macroblock 105 a processing unit in image and video compression formats based on linear 106 block transforms (e.g. H.264, VP8, VP9); codec-specific, but for most of 107 popular codecs the size is 16x16 samples (pixels). 108 109OUTPUT 110 the source buffer queue; for decoders, the queue of buffers containing 111 an encoded bytestream; for encoders, the queue of buffers containing raw 112 frames; ``V4L2_BUF_TYPE_VIDEO_OUTPUT`` or 113 ``V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE``; the hardware is fed with data 114 from ``OUTPUT`` buffers. 115 116PPS 117 Picture Parameter Set; a type of metadata entity in an H.264/HEVC bytestream. 118 119raw format 120 uncompressed format containing raw pixel data (e.g. YUV, RGB formats). 121 122resume point 123 a point in the bytestream from which decoding may start/continue, without 124 any previous state/data present, e.g.: a keyframe (VP8/VP9) or 125 SPS/PPS/IDR sequence (H.264/HEVC); a resume point is required to start decode 126 of a new stream, or to resume decoding after a seek. 127 128source 129 data fed to the decoder or encoder; see ``OUTPUT``. 130 131source height 132 height in pixels for given source resolution; relevant to encoders only. 133 134source resolution 135 resolution in pixels of source frames being source to the encoder and 136 subject to further cropping to the bounds of visible resolution; relevant to 137 encoders only. 138 139source width 140 width in pixels for given source resolution; relevant to encoders only. 141 142SPS 143 Sequence Parameter Set; a type of metadata entity in an H.264/HEVC bytestream. 144 145stream metadata 146 additional (non-visual) information contained inside encoded bytestream; 147 for example: coded resolution, visible resolution, codec profile. 148 149visible height 150 height for given visible resolution; display height. 151 152visible resolution 153 stream resolution of the visible picture, in pixels, to be used for 154 display purposes; must be smaller or equal to coded resolution; 155 display resolution. 156 157visible width 158 width for given visible resolution; display width. 159 160State Machine 161============= 162 163.. kernel-render:: DOT 164 :alt: DOT digraph of decoder state machine 165 :caption: Decoder State Machine 166 167 digraph decoder_state_machine { 168 node [shape = doublecircle, label="Decoding"] Decoding; 169 170 node [shape = circle, label="Initialization"] Initialization; 171 node [shape = circle, label="Capture\nsetup"] CaptureSetup; 172 node [shape = circle, label="Dynamic\nResolution\nChange"] ResChange; 173 node [shape = circle, label="Stopped"] Stopped; 174 node [shape = circle, label="Drain"] Drain; 175 node [shape = circle, label="Seek"] Seek; 176 node [shape = circle, label="End of Stream"] EoS; 177 178 node [shape = point]; qi 179 qi -> Initialization [ label = "open()" ]; 180 181 Initialization -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ]; 182 183 CaptureSetup -> Stopped [ label = "CAPTURE\nbuffers\nready" ]; 184 185 Decoding -> ResChange [ label = "Stream\nresolution\nchange" ]; 186 Decoding -> Drain [ label = "V4L2_DEC_CMD_STOP" ]; 187 Decoding -> EoS [ label = "EoS mark\nin the stream" ]; 188 Decoding -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ]; 189 Decoding -> Stopped [ label = "VIDIOC_STREAMOFF(CAPTURE)" ]; 190 Decoding -> Decoding; 191 192 ResChange -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ]; 193 ResChange -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ]; 194 195 EoS -> Drain [ label = "Implicit\ndrain" ]; 196 197 Drain -> Stopped [ label = "All CAPTURE\nbuffers dequeued\nor\nVIDIOC_STREAMOFF(CAPTURE)" ]; 198 Drain -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ]; 199 200 Seek -> Decoding [ label = "VIDIOC_STREAMON(OUTPUT)" ]; 201 Seek -> Initialization [ label = "VIDIOC_REQBUFS(OUTPUT, 0)" ]; 202 203 Stopped -> Decoding [ label = "V4L2_DEC_CMD_START\nor\nVIDIOC_STREAMON(CAPTURE)" ]; 204 Stopped -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ]; 205 } 206 207Querying Capabilities 208===================== 209 2101. To enumerate the set of coded formats supported by the decoder, the 211 client may call :c:func:`VIDIOC_ENUM_FMT` on ``OUTPUT``. 212 213 * The full set of supported formats will be returned, regardless of the 214 format set on ``CAPTURE``. 215 * Check the flags field of :c:type:`v4l2_fmtdesc` for more information 216 about the decoder's capabilities with respect to each coded format. 217 In particular whether or not the decoder has a full-fledged bytestream 218 parser and if the decoder supports dynamic resolution changes. 219 2202. To enumerate the set of supported raw formats, the client may call 221 :c:func:`VIDIOC_ENUM_FMT` on ``CAPTURE``. 222 223 * Only the formats supported for the format currently active on ``OUTPUT`` 224 will be returned. 225 226 * In order to enumerate raw formats supported by a given coded format, 227 the client must first set that coded format on ``OUTPUT`` and then 228 enumerate formats on ``CAPTURE``. 229 2303. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported 231 resolutions for a given format, passing desired pixel format in 232 :c:type:`v4l2_frmsizeenum` ``pixel_format``. 233 234 * Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a coded pixel 235 format will include all possible coded resolutions supported by the 236 decoder for given coded pixel format. 237 238 * Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a raw pixel format 239 will include all possible frame buffer resolutions supported by the 240 decoder for given raw pixel format and the coded format currently set on 241 ``OUTPUT``. 242 2434. Supported profiles and levels for the coded format currently set on 244 ``OUTPUT``, if applicable, may be queried using their respective controls 245 via :c:func:`VIDIOC_QUERYCTRL`. 246 247Initialization 248============== 249 2501. Set the coded format on ``OUTPUT`` via :c:func:`VIDIOC_S_FMT`. 251 252 * **Required fields:** 253 254 ``type`` 255 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. 256 257 ``pixelformat`` 258 a coded pixel format. 259 260 ``width``, ``height`` 261 coded resolution of the stream; required only if it cannot be parsed 262 from the stream for the given coded format; otherwise the decoder will 263 use this resolution as a placeholder resolution that will likely change 264 as soon as it can parse the actual coded resolution from the stream. 265 266 ``sizeimage`` 267 desired size of ``OUTPUT`` buffers; the decoder may adjust it to 268 match hardware requirements. 269 270 other fields 271 follow standard semantics. 272 273 * **Return fields:** 274 275 ``sizeimage`` 276 adjusted size of ``OUTPUT`` buffers. 277 278 * The ``CAPTURE`` format will be updated with an appropriate frame buffer 279 resolution instantly based on the width and height returned by 280 :c:func:`VIDIOC_S_FMT`. 281 However, for coded formats that include stream resolution information, 282 after the decoder is done parsing the information from the stream, it will 283 update the ``CAPTURE`` format with new values and signal a source change 284 event, regardless of whether they match the values set by the client or 285 not. 286 287 .. important:: 288 289 Changing the ``OUTPUT`` format may change the currently set ``CAPTURE`` 290 format. How the new ``CAPTURE`` format is determined is up to the decoder 291 and the client must ensure it matches its needs afterwards. 292 2932. Allocate source (bytestream) buffers via :c:func:`VIDIOC_REQBUFS` on 294 ``OUTPUT``. 295 296 * **Required fields:** 297 298 ``count`` 299 requested number of buffers to allocate; greater than zero. 300 301 ``type`` 302 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. 303 304 ``memory`` 305 follows standard semantics. 306 307 * **Return fields:** 308 309 ``count`` 310 the actual number of buffers allocated. 311 312 .. warning:: 313 314 The actual number of allocated buffers may differ from the ``count`` 315 given. The client must check the updated value of ``count`` after the 316 call returns. 317 318 Alternatively, :c:func:`VIDIOC_CREATE_BUFS` on the ``OUTPUT`` queue can be 319 used to have more control over buffer allocation. 320 321 * **Required fields:** 322 323 ``count`` 324 requested number of buffers to allocate; greater than zero. 325 326 ``type`` 327 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. 328 329 ``memory`` 330 follows standard semantics. 331 332 ``format`` 333 follows standard semantics. 334 335 * **Return fields:** 336 337 ``count`` 338 adjusted to the number of allocated buffers. 339 340 .. warning:: 341 342 The actual number of allocated buffers may differ from the ``count`` 343 given. The client must check the updated value of ``count`` after the 344 call returns. 345 3463. Start streaming on the ``OUTPUT`` queue via :c:func:`VIDIOC_STREAMON`. 347 3484. **This step only applies to coded formats that contain resolution information 349 in the stream.** Continue queuing/dequeuing bytestream buffers to/from the 350 ``OUTPUT`` queue via :c:func:`VIDIOC_QBUF` and :c:func:`VIDIOC_DQBUF`. The 351 buffers will be processed and returned to the client in order, until 352 required metadata to configure the ``CAPTURE`` queue are found. This is 353 indicated by the decoder sending a ``V4L2_EVENT_SOURCE_CHANGE`` event with 354 ``changes`` set to ``V4L2_EVENT_SRC_CH_RESOLUTION``. 355 356 * It is not an error if the first buffer does not contain enough data for 357 this to occur. Processing of the buffers will continue as long as more 358 data is needed. 359 360 * If data in a buffer that triggers the event is required to decode the 361 first frame, it will not be returned to the client, until the 362 initialization sequence completes and the frame is decoded. 363 364 * If the client has not set the coded resolution of the stream on its own, 365 calling :c:func:`VIDIOC_G_FMT`, :c:func:`VIDIOC_S_FMT`, 366 :c:func:`VIDIOC_TRY_FMT` or :c:func:`VIDIOC_REQBUFS` on the ``CAPTURE`` 367 queue will not return the real values for the stream until a 368 ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to 369 ``V4L2_EVENT_SRC_CH_RESOLUTION`` is signaled. 370 371 .. important:: 372 373 Any client query issued after the decoder queues the event will return 374 values applying to the just parsed stream, including queue formats, 375 selection rectangles and controls. 376 377 .. note:: 378 379 A client capable of acquiring stream parameters from the bytestream on 380 its own may attempt to set the width and height of the ``OUTPUT`` format 381 to non-zero values matching the coded size of the stream, skip this step 382 and continue with the `Capture Setup` sequence. However, it must not 383 rely on any driver queries regarding stream parameters, such as 384 selection rectangles and controls, since the decoder has not parsed them 385 from the stream yet. If the values configured by the client do not match 386 those parsed by the decoder, a `Dynamic Resolution Change` will be 387 triggered to reconfigure them. 388 389 .. note:: 390 391 No decoded frames are produced during this phase. 392 3935. Continue with the `Capture Setup` sequence. 394 395Capture Setup 396============= 397 3981. Call :c:func:`VIDIOC_G_FMT` on the ``CAPTURE`` queue to get format for the 399 destination buffers parsed/decoded from the bytestream. 400 401 * **Required fields:** 402 403 ``type`` 404 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 405 406 * **Return fields:** 407 408 ``width``, ``height`` 409 frame buffer resolution for the decoded frames. 410 411 ``pixelformat`` 412 pixel format for decoded frames. 413 414 ``num_planes`` (for _MPLANE ``type`` only) 415 number of planes for pixelformat. 416 417 ``sizeimage``, ``bytesperline`` 418 as per standard semantics; matching frame buffer format. 419 420 .. note:: 421 422 The value of ``pixelformat`` may be any pixel format supported by the 423 decoder for the current stream. The decoder should choose a 424 preferred/optimal format for the default configuration. For example, a 425 YUV format may be preferred over an RGB format if an additional 426 conversion step would be required for the latter. 427 4282. **Optional.** Acquire the visible resolution via 429 :c:func:`VIDIOC_G_SELECTION`. 430 431 * **Required fields:** 432 433 ``type`` 434 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 435 436 ``target`` 437 set to ``V4L2_SEL_TGT_COMPOSE``. 438 439 * **Return fields:** 440 441 ``r.left``, ``r.top``, ``r.width``, ``r.height`` 442 the visible rectangle; it must fit within the frame buffer resolution 443 returned by :c:func:`VIDIOC_G_FMT` on ``CAPTURE``. 444 445 * The following selection targets are supported on ``CAPTURE``: 446 447 ``V4L2_SEL_TGT_CROP_BOUNDS`` 448 corresponds to the coded resolution of the stream. 449 450 ``V4L2_SEL_TGT_CROP_DEFAULT`` 451 the rectangle covering the part of the ``CAPTURE`` buffer that 452 contains meaningful picture data (visible area); width and height 453 will be equal to the visible resolution of the stream. 454 455 ``V4L2_SEL_TGT_CROP`` 456 the rectangle within the coded resolution to be output to 457 ``CAPTURE``; defaults to ``V4L2_SEL_TGT_CROP_DEFAULT``; read-only on 458 hardware without additional compose/scaling capabilities. 459 460 ``V4L2_SEL_TGT_COMPOSE_BOUNDS`` 461 the maximum rectangle within a ``CAPTURE`` buffer, which the cropped 462 frame can be composed into; equal to ``V4L2_SEL_TGT_CROP`` if the 463 hardware does not support compose/scaling. 464 465 ``V4L2_SEL_TGT_COMPOSE_DEFAULT`` 466 equal to ``V4L2_SEL_TGT_CROP``. 467 468 ``V4L2_SEL_TGT_COMPOSE`` 469 the rectangle inside a ``CAPTURE`` buffer into which the cropped 470 frame is written; defaults to ``V4L2_SEL_TGT_COMPOSE_DEFAULT``; 471 read-only on hardware without additional compose/scaling capabilities. 472 473 ``V4L2_SEL_TGT_COMPOSE_PADDED`` 474 the rectangle inside a ``CAPTURE`` buffer which is overwritten by the 475 hardware; equal to ``V4L2_SEL_TGT_COMPOSE`` if the hardware does not 476 write padding pixels. 477 478 .. warning:: 479 480 The values are guaranteed to be meaningful only after the decoder 481 successfully parses the stream metadata. The client must not rely on the 482 query before that happens. 483 4843. **Optional.** Enumerate ``CAPTURE`` formats via :c:func:`VIDIOC_ENUM_FMT` on 485 the ``CAPTURE`` queue. Once the stream information is parsed and known, the 486 client may use this ioctl to discover which raw formats are supported for 487 given stream and select one of them via :c:func:`VIDIOC_S_FMT`. 488 489 .. important:: 490 491 The decoder will return only formats supported for the currently 492 established coded format, as per the ``OUTPUT`` format and/or stream 493 metadata parsed in this initialization sequence, even if more formats 494 may be supported by the decoder in general. In other words, the set 495 returned will be a subset of the initial query mentioned in the 496 `Querying Capabilities` section. 497 498 For example, a decoder may support YUV and RGB formats for resolutions 499 1920x1088 and lower, but only YUV for higher resolutions (due to 500 hardware limitations). After parsing a resolution of 1920x1088 or lower, 501 :c:func:`VIDIOC_ENUM_FMT` may return a set of YUV and RGB pixel formats, 502 but after parsing resolution higher than 1920x1088, the decoder will not 503 return RGB, unsupported for this resolution. 504 505 However, subsequent resolution change event triggered after 506 discovering a resolution change within the same stream may switch 507 the stream into a lower resolution and :c:func:`VIDIOC_ENUM_FMT` 508 would return RGB formats again in that case. 509 5104. **Optional.** Set the ``CAPTURE`` format via :c:func:`VIDIOC_S_FMT` on the 511 ``CAPTURE`` queue. The client may choose a different format than 512 selected/suggested by the decoder in :c:func:`VIDIOC_G_FMT`. 513 514 * **Required fields:** 515 516 ``type`` 517 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 518 519 ``pixelformat`` 520 a raw pixel format. 521 522 ``width``, ``height`` 523 frame buffer resolution of the decoded stream; typically unchanged from 524 what was returned with :c:func:`VIDIOC_G_FMT`, but it may be different 525 if the hardware supports composition and/or scaling. 526 527 * Setting the ``CAPTURE`` format will reset the compose selection rectangles 528 to their default values, based on the new resolution, as described in the 529 previous step. 530 5315. **Optional.** Set the compose rectangle via :c:func:`VIDIOC_S_SELECTION` on 532 the ``CAPTURE`` queue if it is desired and if the decoder has compose and/or 533 scaling capabilities. 534 535 * **Required fields:** 536 537 ``type`` 538 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 539 540 ``target`` 541 set to ``V4L2_SEL_TGT_COMPOSE``. 542 543 ``r.left``, ``r.top``, ``r.width``, ``r.height`` 544 the rectangle inside a ``CAPTURE`` buffer into which the cropped 545 frame is written; defaults to ``V4L2_SEL_TGT_COMPOSE_DEFAULT``; 546 read-only on hardware without additional compose/scaling capabilities. 547 548 * **Return fields:** 549 550 ``r.left``, ``r.top``, ``r.width``, ``r.height`` 551 the visible rectangle; it must fit within the frame buffer resolution 552 returned by :c:func:`VIDIOC_G_FMT` on ``CAPTURE``. 553 554 .. warning:: 555 556 The decoder may adjust the compose rectangle to the nearest 557 supported one to meet codec and hardware requirements. The client needs 558 to check the adjusted rectangle returned by :c:func:`VIDIOC_S_SELECTION`. 559 5606. If all the following conditions are met, the client may resume the decoding 561 instantly: 562 563 * ``sizeimage`` of the new format (determined in previous steps) is less 564 than or equal to the size of currently allocated buffers, 565 566 * the number of buffers currently allocated is greater than or equal to the 567 minimum number of buffers acquired in previous steps. To fulfill this 568 requirement, the client may use :c:func:`VIDIOC_CREATE_BUFS` to add new 569 buffers. 570 571 In that case, the remaining steps do not apply and the client may resume 572 the decoding by one of the following actions: 573 574 * if the ``CAPTURE`` queue is streaming, call :c:func:`VIDIOC_DECODER_CMD` 575 with the ``V4L2_DEC_CMD_START`` command, 576 577 * if the ``CAPTURE`` queue is not streaming, call :c:func:`VIDIOC_STREAMON` 578 on the ``CAPTURE`` queue. 579 580 However, if the client intends to change the buffer set, to lower 581 memory usage or for any other reasons, it may be achieved by following 582 the steps below. 583 5847. **If the** ``CAPTURE`` **queue is streaming,** keep queuing and dequeuing 585 buffers on the ``CAPTURE`` queue until a buffer marked with the 586 ``V4L2_BUF_FLAG_LAST`` flag is dequeued. 587 5888. **If the** ``CAPTURE`` **queue is streaming,** call :c:func:`VIDIOC_STREAMOFF` 589 on the ``CAPTURE`` queue to stop streaming. 590 591 .. warning:: 592 593 The ``OUTPUT`` queue must remain streaming. Calling 594 :c:func:`VIDIOC_STREAMOFF` on it would abort the sequence and trigger a 595 seek. 596 5979. **If the** ``CAPTURE`` **queue has buffers allocated,** free the ``CAPTURE`` 598 buffers using :c:func:`VIDIOC_REQBUFS`. 599 600 * **Required fields:** 601 602 ``count`` 603 set to 0. 604 605 ``type`` 606 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 607 608 ``memory`` 609 follows standard semantics. 610 61110. Allocate ``CAPTURE`` buffers via :c:func:`VIDIOC_REQBUFS` on the 612 ``CAPTURE`` queue. 613 614 * **Required fields:** 615 616 ``count`` 617 requested number of buffers to allocate; greater than zero. 618 619 ``type`` 620 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 621 622 ``memory`` 623 follows standard semantics. 624 625 * **Return fields:** 626 627 ``count`` 628 actual number of buffers allocated. 629 630 .. warning:: 631 632 The actual number of allocated buffers may differ from the ``count`` 633 given. The client must check the updated value of ``count`` after the 634 call returns. 635 636 .. note:: 637 638 To allocate more than the minimum number of buffers (for pipeline 639 depth), the client may query the ``V4L2_CID_MIN_BUFFERS_FOR_CAPTURE`` 640 control to get the minimum number of buffers required, and pass the 641 obtained value plus the number of additional buffers needed in the 642 ``count`` field to :c:func:`VIDIOC_REQBUFS`. 643 644 Alternatively, :c:func:`VIDIOC_CREATE_BUFS` on the ``CAPTURE`` queue can be 645 used to have more control over buffer allocation. For example, by 646 allocating buffers larger than the current ``CAPTURE`` format, future 647 resolution changes can be accommodated. 648 649 * **Required fields:** 650 651 ``count`` 652 requested number of buffers to allocate; greater than zero. 653 654 ``type`` 655 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. 656 657 ``memory`` 658 follows standard semantics. 659 660 ``format`` 661 a format representing the maximum framebuffer resolution to be 662 accommodated by newly allocated buffers. 663 664 * **Return fields:** 665 666 ``count`` 667 adjusted to the number of allocated buffers. 668 669 .. warning:: 670 671 The actual number of allocated buffers may differ from the ``count`` 672 given. The client must check the updated value of ``count`` after the 673 call returns. 674 675 .. note:: 676 677 To allocate buffers for a format different than parsed from the stream 678 metadata, the client must proceed as follows, before the metadata 679 parsing is initiated: 680 681 * set width and height of the ``OUTPUT`` format to desired coded resolution to 682 let the decoder configure the ``CAPTURE`` format appropriately, 683 684 * query the ``CAPTURE`` format using :c:func:`VIDIOC_G_FMT` and save it 685 until this step. 686 687 The format obtained in the query may be then used with 688 :c:func:`VIDIOC_CREATE_BUFS` in this step to allocate the buffers. 689 69011. Call :c:func:`VIDIOC_STREAMON` on the ``CAPTURE`` queue to start decoding 691 frames. 692 693Decoding 694======== 695 696This state is reached after the `Capture Setup` sequence finishes successfully. 697In this state, the client queues and dequeues buffers to both queues via 698:c:func:`VIDIOC_QBUF` and :c:func:`VIDIOC_DQBUF`, following the standard 699semantics. 700 701The content of the source ``OUTPUT`` buffers depends on the active coded pixel 702format and may be affected by codec-specific extended controls, as stated in 703the documentation of each format. 704 705Both queues operate independently, following the standard behavior of V4L2 706buffer queues and memory-to-memory devices. In addition, the order of decoded 707frames dequeued from the ``CAPTURE`` queue may differ from the order of queuing 708coded frames to the ``OUTPUT`` queue, due to properties of the selected coded 709format, e.g. frame reordering. 710 711The client must not assume any direct relationship between ``CAPTURE`` 712and ``OUTPUT`` buffers and any specific timing of buffers becoming 713available to dequeue. Specifically: 714 715* a buffer queued to ``OUTPUT`` may result in no buffers being produced 716 on ``CAPTURE`` (e.g. if it does not contain encoded data, or if only 717 metadata syntax structures are present in it), 718 719* a buffer queued to ``OUTPUT`` may result in more than one buffer produced 720 on ``CAPTURE`` (if the encoded data contained more than one frame, or if 721 returning a decoded frame allowed the decoder to return a frame that 722 preceded it in decode, but succeeded it in the display order), 723 724* a buffer queued to ``OUTPUT`` may result in a buffer being produced on 725 ``CAPTURE`` later into decode process, and/or after processing further 726 ``OUTPUT`` buffers, or be returned out of order, e.g. if display 727 reordering is used, 728 729* buffers may become available on the ``CAPTURE`` queue without additional 730 buffers queued to ``OUTPUT`` (e.g. during drain or ``EOS``), because of the 731 ``OUTPUT`` buffers queued in the past whose decoding results are only 732 available at later time, due to specifics of the decoding process. 733 734.. note:: 735 736 To allow matching decoded ``CAPTURE`` buffers with ``OUTPUT`` buffers they 737 originated from, the client can set the ``timestamp`` field of the 738 :c:type:`v4l2_buffer` struct when queuing an ``OUTPUT`` buffer. The 739 ``CAPTURE`` buffer(s), which resulted from decoding that ``OUTPUT`` buffer 740 will have their ``timestamp`` field set to the same value when dequeued. 741 742 In addition to the straightforward case of one ``OUTPUT`` buffer producing 743 one ``CAPTURE`` buffer, the following cases are defined: 744 745 * one ``OUTPUT`` buffer generates multiple ``CAPTURE`` buffers: the same 746 ``OUTPUT`` timestamp will be copied to multiple ``CAPTURE`` buffers. 747 748 * multiple ``OUTPUT`` buffers generate one ``CAPTURE`` buffer: timestamp of 749 the ``OUTPUT`` buffer queued first will be copied. 750 751 * the decoding order differs from the display order (i.e. the ``CAPTURE`` 752 buffers are out-of-order compared to the ``OUTPUT`` buffers): ``CAPTURE`` 753 timestamps will not retain the order of ``OUTPUT`` timestamps. 754 755.. note:: 756 757 The backing memory of ``CAPTURE`` buffers that are used as reference frames 758 by the stream may be read by the hardware even after they are dequeued. 759 Consequently, the client should avoid writing into this memory while the 760 ``CAPTURE`` queue is streaming. Failure to observe this may result in 761 corruption of decoded frames. 762 763 Similarly, when using a memory type other than ``V4L2_MEMORY_MMAP``, the 764 client should make sure that each ``CAPTURE`` buffer is always queued with 765 the same backing memory for as long as the ``CAPTURE`` queue is streaming. 766 The reason for this is that V4L2 buffer indices can be used by drivers to 767 identify frames. Thus, if the backing memory of a reference frame is 768 submitted under a different buffer ID, the driver may misidentify it and 769 decode a new frame into it while it is still in use, resulting in corruption 770 of the following frames. 771 772During the decoding, the decoder may initiate one of the special sequences, as 773listed below. The sequences will result in the decoder returning all the 774``CAPTURE`` buffers that originated from all the ``OUTPUT`` buffers processed 775before the sequence started. Last of the buffers will have the 776``V4L2_BUF_FLAG_LAST`` flag set. To determine the sequence to follow, the client 777must check if there is any pending event and: 778 779* if a ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to 780 ``V4L2_EVENT_SRC_CH_RESOLUTION`` is pending, the `Dynamic Resolution 781 Change` sequence needs to be followed, 782 783* if a ``V4L2_EVENT_EOS`` event is pending, the `End of Stream` sequence needs 784 to be followed. 785 786Some of the sequences can be intermixed with each other and need to be handled 787as they happen. The exact operation is documented for each sequence. 788 789Should a decoding error occur, it will be reported to the client with the level 790of details depending on the decoder capabilities. Specifically: 791 792* the CAPTURE buffer that contains the results of the failed decode operation 793 will be returned with the V4L2_BUF_FLAG_ERROR flag set, 794 795* if the decoder is able to precisely report the OUTPUT buffer that triggered 796 the error, such buffer will be returned with the V4L2_BUF_FLAG_ERROR flag 797 set. 798 799In case of a fatal failure that does not allow the decoding to continue, any 800further operations on corresponding decoder file handle will return the -EIO 801error code. The client may close the file handle and open a new one, or 802alternatively reinitialize the instance by stopping streaming on both queues, 803releasing all buffers and performing the Initialization sequence again. 804 805Seek 806==== 807 808Seek is controlled by the ``OUTPUT`` queue, as it is the source of coded data. 809The seek does not require any specific operation on the ``CAPTURE`` queue, but 810it may be affected as per normal decoder operation. 811 8121. Stop the ``OUTPUT`` queue to begin the seek sequence via 813 :c:func:`VIDIOC_STREAMOFF`. 814 815 * **Required fields:** 816 817 ``type`` 818 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. 819 820 * The decoder will drop all the pending ``OUTPUT`` buffers and they must be 821 treated as returned to the client (following standard semantics). 822 8232. Restart the ``OUTPUT`` queue via :c:func:`VIDIOC_STREAMON`. 824 825 * **Required fields:** 826 827 ``type`` 828 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. 829 830 * The decoder will start accepting new source bytestream buffers after the 831 call returns. 832 8333. Start queuing buffers containing coded data after the seek to the ``OUTPUT`` 834 queue until a suitable resume point is found. 835 836 .. note:: 837 838 There is no requirement to begin queuing coded data starting exactly 839 from a resume point (e.g. SPS or a keyframe). Any queued ``OUTPUT`` 840 buffers will be processed and returned to the client until a suitable 841 resume point is found. While looking for a resume point, the decoder 842 should not produce any decoded frames into ``CAPTURE`` buffers. 843 844 Some hardware is known to mishandle seeks to a non-resume point. Such an 845 operation may result in an unspecified number of corrupted decoded frames 846 being made available on the ``CAPTURE`` queue. Drivers must ensure that 847 no fatal decoding errors or crashes occur, and implement any necessary 848 handling and workarounds for hardware issues related to seek operations. 849 850 .. warning:: 851 852 In case of the H.264/HEVC codec, the client must take care not to seek 853 over a change of SPS/PPS. Even though the target frame could be a 854 keyframe, the stale SPS/PPS inside decoder state would lead to undefined 855 results when decoding. Although the decoder must handle that case without 856 a crash or a fatal decode error, the client must not expect a sensible 857 decode output. 858 859 If the hardware can detect such corrupted decoded frames, then 860 corresponding buffers will be returned to the client with the 861 V4L2_BUF_FLAG_ERROR set. See the `Decoding` section for further 862 description of decode error reporting. 863 8644. After a resume point is found, the decoder will start returning ``CAPTURE`` 865 buffers containing decoded frames. 866 867.. important:: 868 869 A seek may result in the `Dynamic Resolution Change` sequence being 870 initiated, due to the seek target having decoding parameters different from 871 the part of the stream decoded before the seek. The sequence must be handled 872 as per normal decoder operation. 873 874.. warning:: 875 876 It is not specified when the ``CAPTURE`` queue starts producing buffers 877 containing decoded data from the ``OUTPUT`` buffers queued after the seek, 878 as it operates independently from the ``OUTPUT`` queue. 879 880 The decoder may return a number of remaining ``CAPTURE`` buffers containing 881 decoded frames originating from the ``OUTPUT`` buffers queued before the 882 seek sequence is performed. 883 884 The ``VIDIOC_STREAMOFF`` operation discards any remaining queued 885 ``OUTPUT`` buffers, which means that not all of the ``OUTPUT`` buffers 886 queued before the seek sequence may have matching ``CAPTURE`` buffers 887 produced. For example, given the sequence of operations on the 888 ``OUTPUT`` queue: 889 890 QBUF(A), QBUF(B), STREAMOFF(), STREAMON(), QBUF(G), QBUF(H), 891 892 any of the following results on the ``CAPTURE`` queue is allowed: 893 894 {A', B', G', H'}, {A', G', H'}, {G', H'}. 895 896 To determine the CAPTURE buffer containing the first decoded frame after the 897 seek, the client may observe the timestamps to match the CAPTURE and OUTPUT 898 buffers or use V4L2_DEC_CMD_STOP and V4L2_DEC_CMD_START to drain the 899 decoder. 900 901.. note:: 902 903 To achieve instantaneous seek, the client may restart streaming on the 904 ``CAPTURE`` queue too to discard decoded, but not yet dequeued buffers. 905 906Dynamic Resolution Change 907========================= 908 909Streams that include resolution metadata in the bytestream may require switching 910to a different resolution during the decoding. 911 912.. note:: 913 914 Not all decoders can detect resolution changes. Those that do set the 915 ``V4L2_FMT_FLAG_DYN_RESOLUTION`` flag for the coded format when 916 :c:func:`VIDIOC_ENUM_FMT` is called. 917 918The sequence starts when the decoder detects a coded frame with one or more of 919the following parameters different from those previously established (and 920reflected by corresponding queries): 921 922* coded resolution (``OUTPUT`` width and height), 923 924* visible resolution (selection rectangles), 925 926* the minimum number of buffers needed for decoding, 927 928* bit-depth of the bitstream has been changed. 929 930Whenever that happens, the decoder must proceed as follows: 931 9321. After encountering a resolution change in the stream, the decoder sends a 933 ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to 934 ``V4L2_EVENT_SRC_CH_RESOLUTION``. 935 936 .. important:: 937 938 Any client query issued after the decoder queues the event will return 939 values applying to the stream after the resolution change, including 940 queue formats, selection rectangles and controls. 941 9422. The decoder will then process and decode all remaining buffers from before 943 the resolution change point. 944 945 * The last buffer from before the change must be marked with the 946 ``V4L2_BUF_FLAG_LAST`` flag, similarly to the `Drain` sequence above. 947 948 .. warning:: 949 950 The last buffer may be empty (with :c:type:`v4l2_buffer` ``bytesused`` 951 = 0) and in that case it must be ignored by the client, as it does not 952 contain a decoded frame. 953 954 .. note:: 955 956 Any attempt to dequeue more ``CAPTURE`` buffers beyond the buffer marked 957 with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from 958 :c:func:`VIDIOC_DQBUF`. 959 960The client must continue the sequence as described below to continue the 961decoding process. 962 9631. Dequeue the source change event. 964 965 .. important:: 966 967 A source change triggers an implicit decoder drain, similar to the 968 explicit `Drain` sequence. The decoder is stopped after it completes. 969 The decoding process must be resumed with either a pair of calls to 970 :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the 971 ``CAPTURE`` queue, or a call to :c:func:`VIDIOC_DECODER_CMD` with the 972 ``V4L2_DEC_CMD_START`` command. 973 9742. Continue with the `Capture Setup` sequence. 975 976.. note:: 977 978 During the resolution change sequence, the ``OUTPUT`` queue must remain 979 streaming. Calling :c:func:`VIDIOC_STREAMOFF` on the ``OUTPUT`` queue would 980 abort the sequence and initiate a seek. 981 982 In principle, the ``OUTPUT`` queue operates separately from the ``CAPTURE`` 983 queue and this remains true for the duration of the entire resolution change 984 sequence as well. 985 986 The client should, for best performance and simplicity, keep queuing/dequeuing 987 buffers to/from the ``OUTPUT`` queue even while processing this sequence. 988 989Drain 990===== 991 992To ensure that all queued ``OUTPUT`` buffers have been processed and related 993``CAPTURE`` buffers are given to the client, the client must follow the drain 994sequence described below. After the drain sequence ends, the client has 995received all decoded frames for all ``OUTPUT`` buffers queued before the 996sequence was started. 997 9981. Begin drain by issuing :c:func:`VIDIOC_DECODER_CMD`. 999 1000 * **Required fields:** 1001 1002 ``cmd`` 1003 set to ``V4L2_DEC_CMD_STOP``. 1004 1005 ``flags`` 1006 set to 0. 1007 1008 ``pts`` 1009 set to 0. 1010 1011 .. warning:: 1012 1013 The sequence can be only initiated if both ``OUTPUT`` and ``CAPTURE`` 1014 queues are streaming. For compatibility reasons, the call to 1015 :c:func:`VIDIOC_DECODER_CMD` will not fail even if any of the queues is 1016 not streaming, but at the same time it will not initiate the `Drain` 1017 sequence and so the steps described below would not be applicable. 1018 10192. Any ``OUTPUT`` buffers queued by the client before the 1020 :c:func:`VIDIOC_DECODER_CMD` was issued will be processed and decoded as 1021 normal. The client must continue to handle both queues independently, 1022 similarly to normal decode operation. This includes: 1023 1024 * handling any operations triggered as a result of processing those buffers, 1025 such as the `Dynamic Resolution Change` sequence, before continuing with 1026 the drain sequence, 1027 1028 * queuing and dequeuing ``CAPTURE`` buffers, until a buffer marked with the 1029 ``V4L2_BUF_FLAG_LAST`` flag is dequeued, 1030 1031 .. warning:: 1032 1033 The last buffer may be empty (with :c:type:`v4l2_buffer` 1034 ``bytesused`` = 0) and in that case it must be ignored by the client, 1035 as it does not contain a decoded frame. 1036 1037 .. note:: 1038 1039 Any attempt to dequeue more ``CAPTURE`` buffers beyond the buffer 1040 marked with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from 1041 :c:func:`VIDIOC_DQBUF`. 1042 1043 * dequeuing processed ``OUTPUT`` buffers, until all the buffers queued 1044 before the ``V4L2_DEC_CMD_STOP`` command are dequeued, 1045 1046 * dequeuing the ``V4L2_EVENT_EOS`` event, if the client subscribed to it. 1047 1048 .. note:: 1049 1050 For backwards compatibility, the decoder will signal a ``V4L2_EVENT_EOS`` 1051 event when the last frame has been decoded and all frames are ready to be 1052 dequeued. It is a deprecated behavior and the client must not rely on it. 1053 The ``V4L2_BUF_FLAG_LAST`` buffer flag should be used instead. 1054 10553. Once all the ``OUTPUT`` buffers queued before the ``V4L2_DEC_CMD_STOP`` call 1056 are dequeued and the last ``CAPTURE`` buffer is dequeued, the decoder is 1057 stopped and it will accept, but not process, any newly queued ``OUTPUT`` 1058 buffers until the client issues any of the following operations: 1059 1060 * ``V4L2_DEC_CMD_START`` - the decoder will not be reset and will resume 1061 operation normally, with all the state from before the drain, 1062 1063 * a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the 1064 ``CAPTURE`` queue - the decoder will resume the operation normally, 1065 however any ``CAPTURE`` buffers still in the queue will be returned to the 1066 client, 1067 1068 * a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the 1069 ``OUTPUT`` queue - any pending source buffers will be returned to the 1070 client and the `Seek` sequence will be triggered. 1071 1072.. note:: 1073 1074 Once the drain sequence is initiated, the client needs to drive it to 1075 completion, as described by the steps above, unless it aborts the process by 1076 issuing :c:func:`VIDIOC_STREAMOFF` on any of the ``OUTPUT`` or ``CAPTURE`` 1077 queues. The client is not allowed to issue ``V4L2_DEC_CMD_START`` or 1078 ``V4L2_DEC_CMD_STOP`` again while the drain sequence is in progress and they 1079 will fail with -EBUSY error code if attempted. 1080 1081 Although not mandatory, the availability of decoder commands may be queried 1082 using :c:func:`VIDIOC_TRY_DECODER_CMD`. 1083 1084End of Stream 1085============= 1086 1087If the decoder encounters an end of stream marking in the stream, the decoder 1088will initiate the `Drain` sequence, which the client must handle as described 1089above, skipping the initial :c:func:`VIDIOC_DECODER_CMD`. 1090 1091Commit Points 1092============= 1093 1094Setting formats and allocating buffers trigger changes in the behavior of the 1095decoder. 1096 10971. Setting the format on the ``OUTPUT`` queue may change the set of formats 1098 supported/advertised on the ``CAPTURE`` queue. In particular, it also means 1099 that the ``CAPTURE`` format may be reset and the client must not rely on the 1100 previously set format being preserved. 1101 11022. Enumerating formats on the ``CAPTURE`` queue always returns only formats 1103 supported for the current ``OUTPUT`` format. 1104 11053. Setting the format on the ``CAPTURE`` queue does not change the list of 1106 formats available on the ``OUTPUT`` queue. An attempt to set a ``CAPTURE`` 1107 format that is not supported for the currently selected ``OUTPUT`` format 1108 will result in the decoder adjusting the requested ``CAPTURE`` format to a 1109 supported one. 1110 11114. Enumerating formats on the ``OUTPUT`` queue always returns the full set of 1112 supported coded formats, irrespectively of the current ``CAPTURE`` format. 1113 11145. While buffers are allocated on any of the ``OUTPUT`` or ``CAPTURE`` queues, 1115 the client must not change the format on the ``OUTPUT`` queue. Drivers will 1116 return the -EBUSY error code for any such format change attempt. 1117 1118To summarize, setting formats and allocation must always start with the 1119``OUTPUT`` queue and the ``OUTPUT`` queue is the master that governs the 1120set of supported formats for the ``CAPTURE`` queue. 1121