[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs in the code in this repository.
If you have a general question, need help debugging, or fall into some
other category use one of these other channels:

• For general technical questions, post a question on StackOverflow
  with the firebase tag.
• For general Firebase discussion, use the firebase-talk
  google group.
• For help troubleshooting your application that does not fall under one
  of the above categories, reach out to the personalized
  Firebase support channel.

[REQUIRED] Step 2: Describe your environment

• Android Studio version: Android Studio Giraffe | 2022.3.1 Patch 2
• Firebase Component: Firestore
• Component version: com.google.firebase:firebase-bom:32.3.1

[REQUIRED] Step 3: Describe the problem

During our development (our app is already in production with Firestore for nearly a year so we're very familiar with the Firestore Android SDK), we spotted one rare issue that we've never experienced before, which is Firestore getting stuck permanently after a large volume of mutations on the client side. Although this is our first time experiencing the issue, after days of debugging we couldn't find any reason why it was getting stuck, here's what we happens:

1. We're using the same device that we've be constantly using for app development throughout the last year.
2. We're certain that there was no internet connection issue. As our app is designed to work both offline and online, so we're very familiar with Firestore and knows that its Task won't complete if there was no internet connection, because it can only complete when a write is committed on server
3. After a large volume of mutations, in a roughly sequence of inserting thousands of documents -> deleting thousands of documents -> inserting thousands of documents -> etc (about ~10 times), Firestore got stuck because we waited for more than 1 day (24 HOURS), but still Firestore couldn't complete its operations. We were verifying that it's getting stuck by issuing a dummy call to Firestore, which to delete a non-existent document, however, this Task never completed, failed or cancelled.
4. While firestore got stuck, we used Profiler inside Android Studio and can see that FirestoreWorker has been running 100% of the CPU time (of its own thread).
5. We kept the device connected, charged and on idle for 3 days, but the dummy call (deletion of non existence document) still couldn't complete.
6. We tried restarting the app many times, but it never recovered from the "stuck" state

As of right now, we've already re-installed our app yesterday to recover from this issue. However, while we were debugging the issue, we did dump the Firestore database at that time, because I thought it might be helpful for Firestore engineers to investigate. Here's the link to the DB. The db itself is gigantic, at size of > 1GB, our app does not override any Firestore's persistence settings, so everything about offline persistence should be the default.

We've filed a support case to Firebase Support earlier (last week) with case number 10250473, however, they said they were to spot any issue on the server side and suggested me to contact the client SDK team directly.

Steps to reproduce:

Ever since then, we never encountered the issue again. Similarly during our ~1 year development with Firestore Android SDK, that was the only time we encountered the issue. We believe that it could be a rare case, however, when the issue does happen, it is SERIOUS because the only way to get out of it is to uninstall and reinstall the app. Therefore, we're kindly asking Firestore team to take a look at this issue.

Relevant Code:

Explained above