[wgsl] Add FAQ for direction on shader language #562
Conversation
Describes the decision for WGSL, its main traits. Avoids rehash of previous arguments.
|
(from the twitter-sphere, but don't worry not coming with pitchforks) I would have liked to have made the meeting but was unable to unfortunately. That said, I do have some questions if it's alright! Reading the FAQ as stated, is the implication that I will not be able to ship SPIR-V directly and must introduce a "decompilation" step to WGSL prior to ship in my toolchain? In the examples shown, I'm not seeing a preprocessor. Will there be one? Can WGSL code be modularized and linked? Moving the precision to a template parameter of each individual type is a somewhat big decision IMO. Is there an easy way of changing the precision for a particular platform/browser/hardware tuple? Is there a rationale for the heavily rust-flavored syntax when most existing shading languages aren't rust-flavored? Can you provide some clarity around struct/type declarations? In particular, alignment rules, and how interoperability with SPIR-V hierarchical types will be achieved? Cheers |
WebGPU will not ingest SPIR-V directly. If your toolchain is currently terminating at SPIR-V, you will have to compile that into WGSL before sending to WebGPU.
There is no preprocess for WGSL. The preprocessor has historically been a large area of incompatibilities and bugs. Not sure what you mean by modularized and linked, it's text, you can concatenate your text together before sending to WebGPU.
So, you'd want to send a different shader to Chrome on Mac then Chrome on Windows?
The syntax is easier to parse, which makes it harder to parse wrong.
That is currently an open discussion issue. As spec'd the offsets must be provided for |
|
Hey @jeremyong, Thanks for your interest! I'd like to keep this PR focused on landing the bare-bones FAQ as far as it goes. Your other questions are great but start bleeding into design considerations. I suggest filing one or more issues with the [wgsl] in the title (and we'll tag with wgsl label). Then we can address them in a more compartmentalized and complete way. (And if you don't file the issues, in due time I can fork them off myself.) |
|
Just a question from someone building an experimental WebGPU backend into a web 3D engine: given that it says this was agreed upon at a meeting and that the 4 reviewers on this PR are reps for the big 4: will this be the one to unify all vendors? (i.e. no more W(HL)SL or SPIR-V bytecode input to construct GPUShaderModuleDescriptor? (SPIR-V was answered above now, thanks) |
|
@zenmumbler yes, it is the WebGPU shading language going forward (though it'll take some time to have browsers with some initial implementations). |
It would be nice to have this be explained in the FAQ a bit more and potentially have it contrasted to what WebAssembly is doing (eg. ship binary representation but have a nice human readable textual representation for writing & debugging). |
I should mention that part of my motivation for this line of questions was because we have a great deal of code/infrastructure that needs to do all the above and more. Supporting different platforms/OSs/etc requires custom preprocessing, string pasting, macro expansion, etc. so a new language to now parse and/or manipulate to integrate with existing material/vfx editors and tools is definitely a non-trivial burden. |
|
In that case, you can generate conforming SPIR-V which we can then convert to WGSL. The WGSL spec has been written with feedback from the process of converting SPIR-V to WGSL. |
One has to consider the reality of the situation; which is that shader compiles already are costing hours (a full day for a rebuild isn't unheard of). Edit: the other reality being that I've yet to encounter a game developer that's actually happy about what's being decided on here. |
|
The other fact is that going from IR which is already in SSA form and presumably would save pipeline compilation time back to a high level language strikes me as a bit odd? I can understand providing a new frontend language if the idea is to improve shader authorship workflow (one of the main goals for SPIR-V was to enable this IIRC), but providing a new frontend language to serve as the main ingestion point (which will then subsequently get compiled again) is definitely awkward IMO. |
design/ShaderLanguageFAQ.md
Outdated
| (See https://xkcd.com/927/) | ||
|
|
||
| For developers invested in the SPIR-V ecosystem for Vulkan and OpenGL, | ||
| the key will be convertibility between WGLSL and SPIR-V. |
There was a problem hiding this comment.
Fixed in next commit.
| * It is strongly typed. | ||
| * It directly represents block-structured programming, including | ||
| sequence, selection (if/else), and repetition (looping). | ||
| * It can be faithfully and simply converted to and from SPIR-V |
There was a problem hiding this comment.
I'd also mention HLSL and MSL. Something like:
- It can be faithfully and simply converted to HLSL and Metal Shading Language.
Also might want to consider adding links to these shading languages.
There was a problem hiding this comment.
Ignoring the bikeshed over what "simply converted" means, I don't think this should be a goal of WGSL. I thought the working group agreed that WGSL should be defined in terms of SPIR-V semantics. If a conversion to MSL ends up being a challenge, that is the responsibility of the implementer.
There was a problem hiding this comment.
It absolutely can be defined in terms of SPIR-V semantics. That doesn't mean that it's okay for it to be extremely difficult to compile to HLSL or MSL.
There was a problem hiding this comment.
Please see latest commit
design/ShaderLanguageFAQ.md
Outdated
|
|
||
| You can say it letter-by-letter, like “W G S L”. | ||
|
|
||
| You can say it as a single word, like “wiggzle”, “wiggles”, or “wigsel”. |
There was a problem hiding this comment.
I would delete this line. The community will eventually coalesce around a pronunciation without us making these suggestions.
design/ShaderLanguageFAQ.md
Outdated
| } | ||
| ``` | ||
|
|
||
|
|
There was a problem hiding this comment.
I'd repeat here that this syntax is still under heavy development. Perhaps with a link for a "where can I comment constructively if I have opinions about this syntax"
design/ShaderLanguageFAQ.md
Outdated
|
|
||
| WGSL is the shader language for WebGPU. | ||
| Any browser implementing WebGPU will be required to accept shaders in WGSL. | ||
| Specifically, the WebGPU conformance test suite will use shaders written |
design/ShaderLanguageFAQ.md
Outdated
| ## What is the shader language for WebGPU? | ||
|
|
||
| WebGPU will use a new shader language “WebGPU Shader Language”, | ||
| or “WGSL” for short. |
There was a problem hiding this comment.
I think we need a sentence about how WGSL is designed specifically for WebGPU. Perhaps there will be offshoots later for other purposes, but the stated purpose is the shading language for WebGPU.
Also perhaps it should be mentioned somewhere that they are designed in tandem, together, as a unit, to be used in the same codebase.
There was a problem hiding this comment.
Can you describe or point to a write up that expresses your viewpoint on what specifically it is about the web that makes existing instruction sets/languages unsuitable to build off of.
There was a problem hiding this comment.
I can point to a write-up about non-human-writable languages: #43
Other members of the group can point you to write-ups about other languages.
|
|
||
| At this time, we expect at least three essentially independent | ||
| implementations of WGSL ingestion as part of browser support for WebGPU. | ||
| Independent implementation serves as an important check on the quality |
There was a problem hiding this comment.
"serves as a check on the quality"? Doesn't that technically indicate that independent implementations reduce quality? (as in "checks and balances")
There was a problem hiding this comment.
I don't think so? It's saying that having multiple implementations means we know the spec can be implemented and isn't just a product of a single compiler. Those implementations check that the spec is actually correct, vs what a given implementation happens to do.
There was a problem hiding this comment.
That's to contrast with WebGL which is a compiler monoculture now.
There was a problem hiding this comment.
Right, I agree with the intention, but I believe the specific words in this document do not match the intention.
There was a problem hiding this comment.
WebGL is not a compiler monoculture, and it's actually a major problem for users. Safari's WebGL GLSL parser has bugs I've worked around on past projects. And Chrome-on-Android does not go through ANGLE's shader translator (best I know?), leading to me needing to work around plenty of driver issues in my WebGL applications.
The graphics community has years of experience with the "multiple reimplementations; no reference tooling" state of affairs with GLSL. One of Vulkan's successes in this arena was developing reference tooling, supported by Khronos themselves.
There was a problem hiding this comment.
I think the words say what I intend them to say.
Independent implementation ensures that multiple people have looked at the spec very closely, i.e. close enough to implement it. Doing so shines a light on lurking ambiguity or gaps in the spec. It's not a exhaustive proof, but "an important check". The other line of defense is the test suite. Hopefully independent implementation will induce ambiguity-reducing discussion, and when that's resolved the decision is baked into both a spec update and a test update.
That's my intent here, but I did't want to write so many words for it.
There was a problem hiding this comment.
In typical English idiom, "serves as an important check on X" means placing a limitation on X or reducing X, rather than providing evidence for X. c.f. "Congress serves as an important check on the power of the Presidency." Perhaps it can have that other meaning too, but in that case why use an ambiguous phrase? You could say "Independent implementation provides evidence for the quality of the spec."
This may seem nitpicky but use of ambiguous wording like this can be legitimately confusing to some people (e.g. non-native English speakers or the neurodiverse).
design/ShaderLanguageFAQ.md
Outdated
| (See https://xkcd.com/927/) | ||
|
|
||
| For developers invested in the SPIR-V ecosystem for Vulkan and OpenGL, | ||
| the key will be convertibility between WGLSL and SPIR-V. |
There was a problem hiding this comment.
"The key?" I don't know what that means.
Rather than being aspirational, please be factual. There will almost certainly eventually be open source compilers from many languages, likely including SPIR-V, to WGSL. The WebGPU standardization group will not produce these compilers.
There was a problem hiding this comment.
I'm not even sure it's worth calling out SPIR-V explicitly. There will almost certainly be many compilers. They assuredly will intend to be faithful.
There was a problem hiding this comment.
I read the point here not as "convertibility is possible, and there will be compilers from many languages" but as "simple and clean convertibility with SPIR-V specifically is the goal".
There was a problem hiding this comment.
I don't think "simple and clean convertibility" is relevant to the audience of this document. "Simple and clean" is a means to an end; the end is portability, both in terms of behavior and performance. There are other ways of achieving that end.
There was a problem hiding this comment.
The ease of integration with existing shader technologies is very much relevant to the audience of this document. You can even see people chiming in about this topic, as a question, directly above this comment thread.
design/ShaderLanguageFAQ.md
Outdated
| For shaders that only use WebGPU features, WebGPU Community Group | ||
| members expect to develop tooling to faithfully convert between WGSL | ||
| and “Shader” SPIR-V. | ||
| By “faithful”, we mean that all salient properties of a shader should |
There was a problem hiding this comment.
Doesn't every compiler (intend to) faithfully preserve behavior for the subset of the source language it intends to support? This paragraph doesn't seem to mean anything, and I would delete it.
There was a problem hiding this comment.
This is attempting to say that a round trip of WGSL -> SPIR-V ->WGSL won't be byte for byte identical.
There was a problem hiding this comment.
The round tripping of WGSL -> MSL -> WGSL also won't be byte for byte identical. Also, the round tripping of WGSL -> HLSL -> WGLSL also won't be byte for byte identical. I see no reason for this paragraph to exist.
There was a problem hiding this comment.
I mean something less than byte-for-byte identicality, but more than a typical round trip between shader languages.
I hinted at it with the salience of factors affecting "performance". Details are TBD. But for example, the achievable parallelism should be preserved through the round trip I care about. E.g. effective occupancy of a warp on NV hardware or wavefront on AMD hardware. It's likely that a good proxy for that is private register count, and the hypothesis is that can be faithfully counted by staring at WGSL source. The goal is to avoid having a translation to/from WGSL induce falling off a performance cliff. In my view that's critical to the success of WebGPU.
That's why the paragraph exists.
| For developers invested in the SPIR-V ecosystem for Vulkan and OpenGL, | ||
| the key will be convertibility between WGLSL and SPIR-V. | ||
| For shaders that only use WebGPU features, WebGPU Community Group | ||
| members expect to develop tooling to faithfully convert between WGSL |
There was a problem hiding this comment.
Not in the purview of the standardization group, though. I'm not sure what business this sentence has being in the standardization group's repo.
There was a problem hiding this comment.
The work may not be in the purview, but the question has come up, so having a response to it is good.
There was a problem hiding this comment.
I'm not disputing having responses to questions. I'm disputing those responses being hosted by the WebGPU standardization group.
Another way of saying this: If my company decides to make a Forth to WGSL compiler, there's no reason for us to mention it in this repository.
There was a problem hiding this comment.
If the availability of a Forth-to-WGSL was a frequently asked question, I think it would totally deserve a mention in the list of Frequently Asked Questions. As it stands, the toolchain support around WGSL and compatibility with other commonly-used shading language ecosystem technologies is probably the most frequently asked question about the project, so I think it makes sense to have an answer for it.
| be preserved by a round trip conversion between the two ways of writing | ||
| the shader: e.g. it should retain its semantics and its performance | ||
| characteristics. | ||
| Furthermore, those WebGPU Community Group members plan to add an option |
There was a problem hiding this comment.
Again, not in the purview of the standardization group. If one member company wants to do something outside the bounds of the standardization group, they can go right ahead. But it isn't relevant to this repo.
There was a problem hiding this comment.
As above, the question comes up, so it is relevant to the FAQ, and what given members are thinking is useful information.
There was a problem hiding this comment.
But it isn't relevant to this repo.
True, but in the end we'll just end up with every big participant to engage in an info war by releasing blog posts, where we'd have even less collaboration and mutual agreement on what's being announced...
@litherum do you disagree with the intention here? You don't want to have the existing compilers target WGSL?
If not, we could rephrase this as something like "WebGPU CG members anticipate ..." or "expect that ... will emerge" without mentioning who exactly is going to work on that.
There was a problem hiding this comment.
Having compilers target this language would be useful and valuable, regardless of the source language. Specifically mentioning a project by one member company which uses this language as a destination language, is irrelevant, and potentially even misleading.
There was a problem hiding this comment.
I didn't mention any specific projects, or particular member companies. I believe I already provided caveats and qualified language as @kvark suggested.
I mentioned an ecosystem that is highly relevant going forward, and was the proposed technology prior to the decision (which makes the question highly relevant).
Hey @litherum, would you prefer if I prefaced this section with a prominent notice along the lines of: "This section is outside the purview of the community group" ?
|
One of the big advantages of SpirV was being able to compile existing GLSL shader code asynchronously. Will there at least be an effort to make shaders compilable asynchronously? That includes any translation layers vendors might make. |
|
Not addressed in this FAQ is why the group decided on WGSL. I think it’d be beneficial to add a section about why this language was chosen as opposed to direct SPIRV or other alternatives. |
|
I made proposed changes to this PR here. I didn't want my proposed changes to be in @dneto0's repo directly, since members of the CG would probably not think to go there to review potential changes to this PR. I also didn't want to just push directly to @dneto0's branch, because that feels kind of rude somehow. |
- Highlight that WGSL is for WebGPU; being designed together - Offshoots of WGSL may appear over time - Add note about translating to HLSL and MSL - Remove examples of single-word pronunciations - Move the "Where can I find the spec" section up, because... - Add a "how to provide feedback" question - Add note at tend of sample syntax section that the language is still under heavy development, and reference the prior where-to-give-feedback note. - Use @Kangz's note about WGSL support supplanting other shader language support in WebGPU implementations. Some @Litherium suggestions not taken.
Ah, I didn't see this in time. I pushed an update which incorporated much of your feedback, but not all. Let me take a look. Also, consider taking a look at mine. |
|
I will rebase mine on top of yours. |
There was a problem hiding this comment.
The thing I'm worried about is someone interested in WGSL, coming to this repository, reading this document, and coming away with the belief that WGSL only works well with SPIR-V.
In the interest of progress, we're willing to compromise about the status of SPIR-V. We're even willing to elevate the status of SPIR-V above the status of HLSL and MSL. We recognize that there are developers who use SPIR-V in their shader pipelines today.
However, this document, especially the last section, sways too far in the direction of privileging SPIR-V. We have three target languages, and WebGPU (and therefore WGSL) need to work great with all of them. The concerns of developers using SPIR-V today are valid, and the concerns of developers using MSL and HLSL today are just as valid.
This community group should not bless certain workflows or development pipelines above others. Instead, we should respect the desires and use cases of authors, support them, add features for them, fix bugs for them, all without using our position to push web authors toward specific native technologies or libraries.
design/ShaderLanguageFAQ.md
Outdated
|
|
||
| ## Where do I provide feedback on WGSL? | ||
|
|
||
| To provide feedabck on WGSL, file an issue on the spec repo site: |
There was a problem hiding this comment.
| To provide feedabck on WGSL, file an issue on the spec repo site: | |
| To provide feedback on WGSL, file an issue on the spec repo site: |
There was a problem hiding this comment.
fixed in next commit.
Until the Goals section of the WGSL spec is updated, reference the "design criteria" issue gpuweb#586
I agree. This is a case of me not wanting to speak for other constituencies. I invite you or others to add the colour commentary you think is fitting and informative to your intended audience. My last paragraph is deliberately scoped: it was predicated with "For developers invested in the SPIR-V ecosystem ...". Stylistically I disliked jamming all that content into one paragraph, but wanted to keep the clear scoping. |
|
Just curious, why was a textual format chosen, if I'm understanding correctly? Especially for web the whole asm.js > WebAssembly has shown the power of binary (smaller size, less parsing time), also won't each browser vendor now have to implement compilers for each platform: WGSL > (SPIR-V, MSL, HLSL)? I may have misunderstood but wasn't the whole idea of the newer low level API's (Vulkan, Metal, DX12) to minimize having multiple drivers implement complex things like shader compilers? Just my 2 cents |
|
There are pros and cons for both text and binary. In the end, the community group chose text. The asm vs webassembly isn't really an equal comparison as you'll still have HTML, JS and CSS pages which are all written in text. |
|
Hmm not sure I follow? If I write my 3D app in say C++ or Rust and compile to WebAssembly, then there will only be a very small amount of HTML/JS that the browser need to download/parse. My main app code will be in WebAssambly. However I may have quite a large amount of shader code - that I can imagine would take the browser more time to download/parse as text. Not trying to pick a fight, just to understand the reasoning behind this. |
|
I don't know if we have any numbers on the time it will take to download and parse WGSL code at this point. |
|
Guess it depends a lot on amount and complexity of shaders :) I just remember back in the day when working with the linux port of unreal how it took like 30 minutes for it to cross compile shaders from HLSL to GLSL ... would be wonderful to avoid anything like that. |
|
Even if the input was binary, we have to compile back to Metal Shading Language or HLSL. Only on the case of running on Vulkan could you avoid the compile, but then you still have to run optimization passes to make sure the SPIR-V is for the WebGPU SPIR-V environment. |
|
Right :) I guess we'll see minification of WGSL for large shader sources .. together with gzip transfer probably makes download size pretty close to binary |
|
@pjoe To quote aras-p on github, "SPIR-V is a very verbose format, several times larger than same programs expressed in other shader formats (e.g. ... GLSL). The SSA-form with ever increasing IDs is not very appreciated by regular data compressors either. " [ref: github.com/aras-p/smol-v] So don't assume that because SPIR-V is binary, that it is more compact than WGSL. The text format of WGSL could turn out to be a net performance win due to smaller file sizes and due to program text being more compressible. |
|
Hi @doug-moen The issues you point out have been discussed by the WebGPU community group before. This PR intentionally stayed away from the why aspect. (To very briefly answer the SPIR-V size question: I recommend Aras' SMOL-V to compress your shaders. It's great. See more info at https://aras-p.info/blog/2018/10/31/SPIR-V-Compression-SMOL-vs-MARK/ ) |
|
https://webkit.org/blog/9528/webgpu-and-wsl-in-safari/ demonstrates SPIR-V being both slower to compile and having larger gzipped wire size than WSL (a textual shading language). |
|
Discussed at 2020-03-10 Teleconference |
|
Closing in favour of #687. |
Describes the decision for WGSL, its main traits.
Avoids rehash of previous arguments.