This investigation has branched out of #1415, related to this google doc discussion.

Support

VK_KHR_sampler_ycbcr_conversion (included in Vulkan 1.1) introduced built-in functionality to bind YCbCr images and sample them as RGB in the shaders. It's gated by samplerYcbcrConversion feature, which is available on more than half of the reports (see windows vk1.1 and android ext).

The specification also defines interaction with Android hardware buffers:

Images that will be backed by an Android hardware buffer can use an external format by setting VkImageCreateInfo::format to VK_FORMAT_UNDEFINED and including a VkExternalFormatANDROID structure in the pNext chain. Images can be created with an external format even if the Android hardware buffer has a format which has an equivalent Vulkan format to enable consistent handling of images from sources that might use either category of format.

@critsec kindly pointed out that Android CTS actually requires the feature to be supported on Vulkan-1.1 devices, so we can expect the coverage to grow.

API

To create a sampler with Y′CBCR conversion enabled, add a VkSamplerYcbcrConversionInfo structure to the pNext chain of the VkSamplerCreateInfo structure. To create a sampler Y′CBCR conversion, the samplerYcbcrConversion feature must be enabled. Conversion must be fixed at pipeline creation time, through use of a combined image sampler with an immutable sampler in VkDescriptorSetLayoutBinding.

A VkSamplerYcbcrConversionInfo must be provided for samplers to be used with image views that access VK_IMAGE_ASPECT_COLOR_BIT if the format appears in Formats requiring sampler Y′CBCR conversion for VK_IMAGE_ASPECT_COLOR_BIT image views , or if the image view has an external format .

In other words, the steps are:

1. create VkSamplerYcbcrConversion object by specifying the format and the color model of the multi-planar image
2. create VkSampler with the specified conversion
3. create VkDescriptorSetLayout with a binding that is a combined image with an immutable sampler. This immutable sampler (that we created in the previous step) is provided right here.
4. write to VkDescriptorSet (of this layout) with a VkImage that has a special way of binding multi-planar memory (at any point before rendering)
5. business as usual: pipeline creation, binding, etc

Proposal

I think, accepting an external texture API that is incompatible with this functionality (available on most Vulkan devices) would be a big missed opportunity - to get driver-specific optimizations and lower the power cost of playing and processing videos.
We should try to make the external texture API compatible with it, instead, so that a Vulkan implementation can opt into using the built-in support if it can.

Interestingly, it also gives an implementation the advantage if it does not want to use Vulkan YCbCr, or even not using Vulkan entirely. Knowing everything that is to know about the external texture at GPUBindGroupLayout type allow us to generate the optimal shader code for the pipeline (since it's visible to createRenderPipeline).

In order to support this, #1415 host side would need to be modified, so that it requires a bit more data:

const layout = device.createBindGroupLayout({
    entries: [{
        binding: 0,
        visibility: GPUShaderStage.FRAGMENT,
        // creates `VkSamplerYcbcrConversion` and `VkSampler` on Vulkan
        videoTexture: {
               format: "y8cb8cr8_422",
        },
    }]
});

Alternatively, if we want to support videos that change the format mid-play, we could make it occupy the whole bind group:

// returns `GPUBindGroupLayout`
const layout = device.createExternalBindGroupLayout({
        visibility: GPUShaderStage.FRAGMENT,
        // specify all the formats that will need to be supported here.
        // a Vulkan implementation may create a separate `VkDescriptorSetLayout` for each, internally.
        formats: [
            "y8cb8cr8_422",
            "y16cb16cr16_422",
        ],
});

If this seems too complex, we could postpone the idea of external textures entirely to avoid being rushed. We can ship some version of a copyXxToTexture in MVP to cover the basic needs, while keeping working on the external texture API.

Disclaimer: this is a rough proposal, may be missing important details. I'm not an expert in video playback or color spaces. Any edits are welcome!

cc @chadversary @Kangz