[PT2E][Quant] Refactor quantizer and qnnpack qantizer code to support dqlinear config #99399

kimishpatel · 2023-04-18T03:47:44Z

Stack from ghstack (oldest at bottom):

This diff introduces a few refactors:

Move observer creation to utils.py.
Use quantization spec to supply args to observers.
Use annotation function registration corresponding QuantizationConfig. This
will be later used in dynamic quantized linear.

Differential Revision: D45073790

… dqlinear config This diff introduces a few refactors: - Move observer creation to utils.py. - Use quantization spec to supply args to observers. - Use annotation function registration corresponding QuantizationConfig. This will be later used in dynamic quantized linear. Differential Revision: [D45073790](https://our.internmc.facebook.com/intern/diff/D45073790/) [ghstack-poisoned]

pytorch-bot · 2023-04-18T03:47:46Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/99399

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

download.pytorch.org flaky

✅ No Failures

As of commit f38d2ba:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

… dqlinear config This diff introduces a few refactors: - Move observer creation to utils.py. - Use quantization spec to supply args to observers. - Use annotation function registration corresponding QuantizationConfig. This will be later used in dynamic quantized linear. Differential Revision: [D45073790](https://our.internmc.facebook.com/intern/diff/D45073790/) ghstack-source-id: 186387917 Pull Request resolved: #99399

jerryzh168 · 2023-04-18T20:53:04Z

torch/ao/quantization/_pt2e/quantizer/qnnpack_quantizer.py

+_QUANT_CONFIG_TO_ANNOTATOR = {}
+
+
+def register_annotator(quantization_configs: List[QuantizationConfig]):


why is this indexed by quantization_config? are we still planning to support module fqn/type based config or other existing features in QConfigMapping or do we just want to support global qconfig?

So the way I imagined this to be used is like this:
qnnpack, i will rename this to xnnpack, quantizer has two configs:
symmetric_static_config
symmetric_dynamic_config.

THis decorator registers annotation function for each.

Now if I want to quantize a model with dynamic q linear for linear and static quant for the rest, I would do it like this with composable quantizer.

dq_linear_config = QNNPACKQuantizer.get_dynamic_linear_config() dq_linear_quantizer = QNNPACKQuantizer().set_global(dq_linear_config) static_qconfig = QNNPACKQuantizer.get_symmetric_config() static_quantizer = QNNPACKQuantizer().set_global(static_config) composed_quantizer = ComposableQuantizer([dq_linear_quantizer, static_quantizer])

We can alternatively do this via:

qnnpack_quantizer = QNNPACKQuantizer().set_global(static_config) # following api doesnt yet exist qnnpack_quantizer.set_module_type_config(torch.nn.Linear, QNNPACKQuantizer.get_dynamic_linear_config()

I think both are fine. RIght now I am doing this mainly because it felt somewhat easier to use, but I am open to just doing via set_module_config or similar API.

Let me think a bit more. If it makes sense to just set_module_type_config api, then I will remove the config-to-annotator mapping part from the PR and follow up separately.

Ok I thought about this and I dont think we should do

dq_linear_config = QNNPACKQuantizer.get_dynamic_linear_config() dq_linear_quantizer = QNNPACKQuantizer().set_global(dq_linear_config) static_qconfig = QNNPACKQuantizer.get_symmetric_config() static_quantizer = QNNPACKQuantizer().set_global(static_config) composed_quantizer = ComposableQuantizer([dq_linear_quantizer, static_quantizer])

But for now I will use annotator api until we do this better

… to support dqlinear config" This diff introduces a few refactors: - Move observer creation to utils.py. - Use quantization spec to supply args to observers. - Use annotation function registration corresponding QuantizationConfig. This will be later used in dynamic quantized linear. Differential Revision: [D45073790](https://our.internmc.facebook.com/intern/diff/D45073790/) [ghstack-poisoned]

jerryzh168 · 2023-04-21T16:28:22Z

just to confirm the plan is to follow up with set_module_type_config api right?

jerryzh168

accepting to unblock for now, we can revisit the API a bit later

… to support dqlinear config" This diff introduces a few refactors: - Move observer creation to utils.py. - Use quantization spec to supply args to observers. - Use annotation function registration corresponding QuantizationConfig. This will be later used in dynamic quantized linear. Differential Revision: [D45073790](https://our.internmc.facebook.com/intern/diff/D45073790/) [ghstack-poisoned]

… dqlinear config Pull Request resolved: #99399 This diff introduces a few refactors: - Move observer creation to utils.py. - Use quantization spec to supply args to observers. - Use annotation function registration corresponding QuantizationConfig. This will be later used in dynamic quantized linear. ghstack-source-id: 187514867 Differential Revision: [D45073790](https://our.internmc.facebook.com/intern/diff/D45073790/)

… to support dqlinear config" This diff introduces a few refactors: - Move observer creation to utils.py. - Use quantization spec to supply args to observers. - Use annotation function registration corresponding QuantizationConfig. This will be later used in dynamic quantized linear. Differential Revision: [D45073790](https://our.internmc.facebook.com/intern/diff/D45073790/) [ghstack-poisoned]

kimishpatel · 2023-05-03T00:39:46Z

@pytorchbot merge

pytorchmergebot · 2023-05-03T00:41:42Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorch-bot bot added the release notes: AO frontend label Apr 18, 2023

This was referenced Apr 18, 2023

[PT2E][Quant] Support for embedding op quantization via ExecuTorchNativeQuantizer #99106

Closed

[PT2E][Quant] Fix but in quant spec of symmetric static quant #99398

Closed

github-actions bot added the release notes: quantization release notes category label Apr 18, 2023

jerryzh168 reviewed Apr 18, 2023

View reviewed changes

kimishpatel mentioned this pull request Apr 20, 2023

[PT2E][Quant] Use subgraph matcher annotate linear pattern #99580

Closed

jerryzh168 approved these changes Apr 22, 2023

View reviewed changes

kimishpatel added 6 commits April 21, 2023 22:04

kimishpatel added 2 commits May 1, 2023 09:16

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 3, 2023

pytorchmergebot added the merging label May 3, 2023

pytorchmergebot added Merged and removed merging labels May 3, 2023

pytorchmergebot closed this in 771a9de May 3, 2023

facebook-github-bot deleted the gh/kimishpatel/132/head branch June 8, 2023 17:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[PT2E][Quant] Refactor quantizer and qnnpack qantizer code to support dqlinear config #99399

[PT2E][Quant] Refactor quantizer and qnnpack qantizer code to support dqlinear config #99399

Uh oh!

kimishpatel commented Apr 18, 2023 •

edited

Loading

Uh oh!

pytorch-bot bot commented Apr 18, 2023 •

edited

Loading

Uh oh!

jerryzh168 Apr 18, 2023

Uh oh!

kimishpatel Apr 20, 2023

Uh oh!

kimishpatel Apr 20, 2023

Uh oh!

kimishpatel Apr 22, 2023

Uh oh!

kimishpatel Apr 22, 2023

Uh oh!

jerryzh168 commented Apr 21, 2023

Uh oh!

jerryzh168 left a comment

Uh oh!

kimishpatel commented May 3, 2023

Uh oh!

pytorchmergebot commented May 3, 2023

Uh oh!

Uh oh!

		_QUANT_CONFIG_TO_ANNOTATOR = {}


		def register_annotator(quantization_configs: List[QuantizationConfig]):

[PT2E][Quant] Refactor quantizer and qnnpack qantizer code to support dqlinear config #99399

[PT2E][Quant] Refactor quantizer and qnnpack qantizer code to support dqlinear config #99399

Uh oh!

Conversation

kimishpatel commented Apr 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Apr 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/99399

❗ 1 Active SEVs

✅ No Failures

Uh oh!

jerryzh168 Apr 18, 2023

Choose a reason for hiding this comment

Uh oh!

kimishpatel Apr 20, 2023

Choose a reason for hiding this comment

Uh oh!

kimishpatel Apr 20, 2023

Choose a reason for hiding this comment

Uh oh!

kimishpatel Apr 22, 2023

Choose a reason for hiding this comment

Uh oh!

kimishpatel Apr 22, 2023

Choose a reason for hiding this comment

Uh oh!

jerryzh168 commented Apr 21, 2023

Uh oh!

jerryzh168 left a comment

Choose a reason for hiding this comment

Uh oh!

kimishpatel commented May 3, 2023

Uh oh!

pytorchmergebot commented May 3, 2023

Merge started

Uh oh!

Uh oh!

kimishpatel commented Apr 18, 2023 •

edited

Loading

pytorch-bot bot commented Apr 18, 2023 •

edited

Loading