refactor(rome_cli): improve the stability of the max_diagnostics tests
#3729
Conversation
✅ Deploy Preview for docs-rometools ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
|
@leops any chance to get this merged and see if we're able to lower the random failures? |
Yes we could try to merge it, I'll rebase it and move it out of draft |
7ff4fda to
f80506d
Compare
|
Comparing refactor(rome_cli): improve the stability of the
1 page tested
|
| Chrome Desktop | iPhone, 4G LTE | Motorola Moto G Power, 3G connection |
|---|---|---|
 |
 |
 |
Most significant changes
 |
Value | Budget |
|---|---|---|
| Total JavaScript Size in Bytes Chrome Desktop |
1.33 MB  from 86.8 KB |
 |
| Total JavaScript Size in Bytes iPhone, 4G LTE |
1.33 MB from 86.8 KB |
|
| Total JavaScript Size in Bytes Motorola Moto G Power, 3G connection |
1.33 MB from 86.8 KB |
|
| Cumulative Layout Shift iPhone, 4G LTE |
0.082  from 0.009 |
|
| Cumulative Layout Shift Motorola Moto G Power, 3G connection |
0.049  from 0.009 |
|
17 other significant changes: Number of Requests on Chrome Desktop, Number of Requests on iPhone, 4G LTE, Number of Requests on Motorola Moto G Power, 3G connection, Total Blocking Time on Chrome Desktop, Total Page Size in Bytes on Chrome Desktop, Total Page Size in Bytes on iPhone, 4G LTE, Total Page Size in Bytes on Motorola Moto G Power, 3G connection, JS Parse & Compile on Chrome Desktop, Total Image Size in Bytes on Chrome Desktop, Total Image Size in Bytes on iPhone, 4G LTE, Total Image Size in Bytes on Motorola Moto G Power, 3G connection, Total CSS Size in Bytes on Chrome Desktop, Total CSS Size in Bytes on iPhone, 4G LTE, Total CSS Size in Bytes on Motorola Moto G Power, 3G connection, Total HTML Size in Bytes on Chrome Desktop, Total HTML Size in Bytes on iPhone, 4G LTE, Total HTML Size in Bytes on Motorola Moto G Power, 3G connection
Calibre: Site dashboard | View this PR | Edit settings | View documentation
Summary
The tests for the
--max-diagnosticsCLI arguments are failing semi-randomly (it seems to happen mostly with macOS on CI but I managed to reproduce the failure at least once on Windows), this PR tries to improve these tests by counting the console messages emitted for diagnostics but still checking the rest of the console output against a snapshot to try and narrow down the issueTest Plan
Run the new tests on CI