Shrek spent a good amount of time working through exhaustive tests of the multisampling masking and alpha-to-coverage behaviors. This uncovered a number of questions especially surrounding the alpha component that's used as an input to alpha-to-coverage. We haven't yet tested them.

Originally posted by @kainino0x in gpuweb/cts#2201 (comment)
See that thread (and linked PRs) for more context.

I didn't even think about the fact we could have a fragment output that isn't used by an attachment.

Looks like on Win D3D12 and Linux Vulkan, the alphaToCoverage mask still uses shader output to target[0].alpha value

I had a lot of thoughts about this so have formatted it as a checklist of things to investigate/do:

• What happens on D3D12 or Vulkan if there is no fragment output at location(0)? undefined value, 1, 0, error?
  Vulkan spec sounds like it would be effectively 1 like I mentioned before.
  We should also test multiple GPUs in case there's any undefined behavior here.
  IMO most likely we should just add validation that there's a location(0) output (with an alpha component) if using alphaToCoverage.

• Similarly, what happens on Metal if there's an attachment 0 but it doesn't have an alpha channel (is an R or RG format)?

• What if it has one but writeMask doesn't include alpha, or even is 0?

• We should bring findings to the gpuweb group, ideally with a proposal on what the spec should say and how to polyfill it across backends. Options:

  • 1: If allowed I think we would have to do the Metal behavior (use 1.0 if there's no attachment, regardless of what the shader outputs). In order to polyfill this behavior we would need to inject code to overwrite the fragment location 0 output to 1 if there's no corresponding attachment in the pipeline. Seems overly complex.
    • It's possible the Metal behavior is actually unintentional, a bug caused by optimizing away writes to the unused attachment 0. Worth asking.
  • 2: Just validate you must have an attachment 0 to use alphaToCoverage. This requirement doesn't strictly make sense but would prevent users from being able to hit the inconsistent behavior. And we could lift it in the future since Vulkan/D3D12 don't require it.

• A tangential thought: The Vulkan spec also has something called alphaToOne, which, IIUC, replaces the alpha channel of output 0 with 1.0 after generating the alphaToCoverage mask. We might already test this, but we should double check that this is not enabled by default on D3D12/Metal. Actually, the D3D11 functional spec is clear about this, so that's good:

  The fact that .a from output o0 is read and converted to coverage when AlphaToCoverageEnable is true does not change the .a value going to the Blender at RenderTarget 0 (if a RenderTarget happens to be set there).

• And another interesting tidbit from the D3D11 functional spec, worth mentioning even though we can't actually test NaNs. Let's leave a "POSTV1" TODO to add this when we add NaN support.

  An alpha value of NaN results in a no coverage (zero) mask.