Environment details

I am currently using model gemini-1.5-flash-002 with a vertex ai data store for grounding. Everything is working great until trying to upgrade the model to anything greater than 1.5.

My grounding data seems to get ignored when I pass both responseMimeType: 'application/json' and tools: [{ retrieval: { vertexAiSearch: { datastore: 'path-to-data-store' } } }] in the generate content config

• Programming language: JavaScript
• OS: Linux
• Language runtime version: Node.js 22.14.0
• Package version: 1.10.0

Steps to reproduce

1. Create a Vertex AI data store
2. Make a generateContent request with JSON mime type and data store retrieval tools

  const response = await googleGenAi.models.generateContent({
    model: 'gemini-2.0-flash',
    contents: 'search query',
    config: {
      responseMimeType: 'application/json',
      temperature: 0,
      maxOutputTokens: 8192,
      tools: [{ retrieval: { vertexAiSearch: { datastore: 'path-to-data-store' } } }],
    },
  })

3. Inspect response.candidates[0].groundingMetadata

gemini-2.0-flash with responseMimeType: 'application/json'

gemini-2.0-flash without responseMimeType: 'application/json'

gemini-1.5-flash-002 with responseMimeType: 'application/json'

gemini-1.5-flash-002 without responseMimeType: 'application/json'

Another thing I noticed is the 2.0 model changing my retrieval query internally causing my results to be worse than 1.5 most of the time