Chroma Reconstruction is the step to scale the Chroma Planes into the same size as Luminance Plane. This step ensures user fetch correct texel value from downsampled Chroma plane with Luminance sample position. Poor Chroma Reconstruction step will cause artifacts (e.g. this article presents visual difference between different Chroma Reconstruction methods).

Chroma Reconstruction in Vulkan spec

Vulkan spec provides details about Chroma Reconstructions for VkSamplerYcbcrConversion . Vulkan spec defines x-y offsets on chroma plane and provide two configs: VK_CHROMA_LOCATION_COSITED_EVEN and VK_CHROMA_LOCATION_MIDPOINT .

• VK_CHROMA_LOCATION_COSITED_EVEN specifies that downsampled chroma samples are aligned with luma samples with even coordinates.
• VK_CHROMA_LOCATION_MIDPOINT specifies that downsampled chroma samples are located half way between each even luma sample and the nearest higher odd luma sample.

The following two diagrams from Vulkan spec show the relationship between unnormalized (u,v) coordinates and (i,j) integer texel positions in the luma channel (shown in black, with circles showing integer sample positions) and the texel coordinates of reduced-resolution chroma channels, shown as crosses in red.

Vulkan spec also defines two reconstruction method: Explicit Reconstruction and Implicit Reconstruction with ChromaFilter(could be LINEAR or NEAREST):

• Explicit Reconstruction is performed as an explicit step independent of filtering.
• Implicit Reconstruction is performed as an implicit part of filtering prior to color model conversion, with no separate post-conversion texel filtering step. It takes place by the samples being interpolated, as required by the filter settings of the sampler, except that chromaFilter takes precedence for the chroma samples.

According to Vulkan spec, texel values in Chroma planes after Chroma Reconstruction step are decided by x-y chroma offsets, ChromaFilter, and reconstruction methods.
There is one special case:

• x-y chroma offsets has no effect when value of ChromaFilter is NEAREST.

WebGPU GPUExternalTexture

My understanding of the 0-copy path for GPUExternalTexture with textureSampleLevel will generate codes(e.g. handling 420 formats) like this:

[[binding(x), group(y)]] var SamplerLuma: sampler;
[[binding(x+1), group(y)]] var TexLuma: texture_2d<f32>;
[[binding(x+2), group(y)]] var SamplerCbCr sampler;
[[binding(x+3), group(y)]] var TexCbCr: texture_2d<f32>;
[[stage(fragment)]] fn fs_main(
                [[location(0)]] texcoord : vec2<f32>
            ) -> [[location(0)]] vec4<f32> {
   var luma = textureSample(TexLuma, SamplerLuma, texcoord);
   var chroma = textureSample(TexCbCr, SamplerCbCr, texcoord);
   var yuv = vec3<f32>(luma.r, chroma.rg);
   if (expand_TV_yuv_to_PC_yuv) {
      yuv = rescale_yuv(yuv);
   }
   var rgb = ConvertYUVToRGB(yuv, matrix);
}

In Vulkan spec the texel values after implicit Chroma Reconstructions are calculated by :

So I think we handle Chroma Reconstruction correctly with above shader code in the case:

• x-y chromaOffsets are all VK_CHROMA_LOCATION_MIDPOINT ,
• And reconstruction method is implicit (support all ChromaFilter configs).

Raise several questions for other Chroma Reconstruction cases.

Questions

• How to deal with explicit reconstructions?

  • Option: Don't commit to support it.
    • The difference between implicit reconstruction and explicit reconstruction is similar to pre-filter and post-filter the community has been discussed in meeting-mintes-2021-05-03. Explicit reconstruction could be achieved through blits. (e.g. through VideoProcessBlt or the blit shaders). But it is not 0-copy anymore. The cost of correctness would be performance.
    • Reconstruction method is not related to video frame info. It is chosen by users. Browser could make its own strategy.(Not True, see comments GPUExternalTexture: Deal with Chroma Reconstruction #2098 (comment))

• How to deal with VK_CHROMA_LOCATION_COSITED_EVEN offset?
  I think we should support it because some frames like 4:2:0-MPEG2 in Microsoft doc use this offset. Possible to deal with it through coords calculation (for implicit reconstruction method). At least we could fallback to 1-copy path to handle it.

• Do we need to add words about chroma reconstructions in spec ?
  For example, if community decides not commit to support explicit reconstructions, WebGPU or WGSL spec could add words "using textureSamplerLevel function to read values from external_texture tend to have implicit chroma reconstructions".