Background

The frame rate jumps around constantly and the app experience is quite jittery. Annecdotally, this was filled as a regression but we don't actually have a way to confirm that this is due to a behavioral change in iOS.

Why are we blocking on waitForNextDrawable.

Assuming "CPU to Display Latency" is "Time from present drawable to system being done with drawable", then right now we have times bouncing between ~21 ms and 26ms. Some math: if we finish frame 0 at ~8.3 ms, then the best case scenario that drawable isn't ready until ~29ms, but we started waiting for it at 25ms. That is 4ms of waitForNextDrawable. In the worst case, we actually just miss the frame completely.

Turning on presentsWithTransaction from a background thead doesn't seem to change the presentation times much, though the appear much more stable and seemingly cause our rendering code to not block on waitForNextDrawable at all. Its unclear how this works, but from running metal system trace there were no additional clues. presentsWithTransaction from the main thread had no such effect.

Why is the display latency so high?

Interestingly if I test on a non high frame rate iOS device, the display latency is comparable. but with 16ms frames we have a lot more leeway. Searching apple forums for info on this turns up nothing. Interestingly, if I follow the advice in the profile above and set the CAMetalLayer.opaque property to YES, then presentation delay drops to 13ms. This may be an option for "regular" flutter apps but not add2app, but I haven't measured this.

How can we work around blocking on waitForNextDrawable?

Generally the advice I've seen on Apple docs/forums is that CAMetalLayer.nextDrawable should be one of the last things we do in frame. That is, rather than use the metal drawable texture as the resolve for our onscreen rendering, we should actually create our own swapchain and then blit into the drawable at the end of the frame. In the example, above where we waited for the next drawable for ~4ms, means that on average we have half as much time to complete the frame.

Note on our vsync

Something odd I've noticed is that regardless of whether metal system traces says that it presented a drawable for 8, 16 ms, the CADisplayLink is always firing at 8ms. Even the previous timestamp doesn't seem to represent when we actually presented our drawable, just the last system vsync.

I'm not sure if this is working as expected, but we don't have any backpressure on this system beyond the pipeline count until we hit the raster thread, so its very possible for the UI thread to run ahead. In fact, this is very much a goal of the UI thread - except that I suspect we're making things worse but not dropping frames earlier in the pipeline. I've tried to make changes to this but it is difficult. I think ideally the vsync implementations would have some awareness of the last time that flutter actually presented so that it could smooth things out. i.e. if the last frame duration was 16ms then assume the next one will take 16ms. Its possible we can make applications that are already janky better by adjusting this logic.

Dead ends

I had preveiously investigated unconditional thread merging, as this allowed presentsWithTransaction uncontionally. On further investigation, I found that this didn't actually help the frame stability, instead it triggered code in the vsync waiter to throttle our frame rate to 80.

Short-term Options

• Cap device frame rate at 80. This seems to gauarantee that we dont have terribly unstable frames but is defiitely a "feels bad man" solution.

• Apply opaque=YES to CAMetalLayer. I'm not sure if its possible to create a "transparent" flutter view on iOS. By default we clear the screen to solid black, but I think you applications could punch through this. We could always give an opt out.

• PresentWithTransaction = YES and give apps a Plist Opt out. Despite the fact that we don't know how/why this works, and the fact that it breaks add2app and certain plugins, it definitely does fix the rendering jankiness. Last resort.

Longer-term options

• Rewrite iOS rendering to acquire metal drawable later, including partial repaint. This will require a substantial reworking of partial repaint. Porting this change to Skia may be difficult without removing partial repaint entirely.