We are experiencing a "Post request fails. Cannot decode the explanation response..." error while trying to generate explanations for batch Vertex AI predictions with a custom container.

We did not follow any specific notebooks in this repo, but we did use a few notebooks as a reference, such as this one:
https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage4/get_started_with_vertex_xai.ipynb

Here is what we are trying to accomplish.
We have a custom training container and a custom prediction container.
We want to generate a feature importance explanations for a custom prediction container. Our predictions are batch predictions. Our model is a custom sklearn model.

Here is how the predict() function in our custom prediction container.

# Define a function for prediction route
@app.post(os.environ["AIP_PREDICT_ROUTE"])
async def predict(request: Request):
    body = await request.json()
    # parse the request instances
    instances = body["instances"]
    logging.info(body.get("parameters", {}).get("columns", {}))
    # pass it to the model/pipeline for prediction scores
    predictions = _model.predict_proba(instances).tolist()
    # return the batch prediction scores
    return {"predictions": predictions}

Our custom prediction container uses FastAPI.

When we set up our explanations, we specified our explanation metadata like this:

my_explanation_metadata = ExplanationMetadata(
            inputs={
                "features": {
                    "index_feature_mapping": columns,
                    "encoding": "BAG_OF_FEATURES",
                }
            },
            outputs={
                "prediction": {}
            },  # it can be anything: "label", "prediction", "purchase"
        )

Our explanation parameters are as follows:

    explanation:
        top_k: -1
        sampled_shapley_attribution:
            path_count: 10

We are experiencing an intermittent error while generating prediction explanations:

Post request fails. Cannot decode the explanation response
……..
Error: Expecting ',' delimiter: line 1 column 1214963 (char 1214962)

When I examine the errors in BQ, the text is super long, many thousands of lines, the word “sessions” appears 1438 times in 1 row. So in 1 row in BQ, it appears that there are many explanations for many different rows. I think the explanations weren’t parsed into rows properly for some reason.

People did report a similar error here, but no solutions.
https://stackoverflow.com/questions/72193993/why-do-i-occasionally-get-post-request-fails-cannot-decode-the-prediction-resp

This issue is intermittent and the result is non-deterministic. I re-ran the exact same job with the exact same inputs and data, and it did succeed, 100%. So basically it works but not in a very reliable way. In case of a partial failure we need to re-run the prediction job.

Is this a known issue? Anyone else experienced the same problem?