这是indexloc提供的服务,不要输入任何密码
Skip to content

v1.8.0

Compare
Choose a tag to compare
@github-actions github-actions released this 28 Apr 16:57
· 15 commits to main since this release
cdf780a

Added

  • Add option to skip rows on response type mismatch (#162)

Option to skip rows on response type mismatch

When sending response data for a query if we encounter a value that does not match the type declared in the connector
schema the default behavior is to respond with an error. That prevents the user from getting any data. This change adds
an option to silently skip rows that contain type mismatches so that the user can get a partial set of result data.

This can come up if, for example, you have database documents with a field that nearly always contains an int value,
but in a handful of cases that field contains a string. Introspection may determine that the type of the field is
int if the random document sampling does not happen to check one of the documents with a string. Then when you run
a query that does read one of those documents the query fails because the connector refuses to return a value of an
unexpected type.

The new option, onResponseTypeMismatch, has two possible values: fail (the existing, default behavior), or skipRow
(the new, opt-in behavior). If you set the option to skipRow in the example case above the connector will silently
exclude documents with unexpected string values in the response. This allows you to get access to the "good" data.
This is opt-in because we don't want to exclude data if users are not aware that might be happening.

The option is set in connector configuration in configuration.json. Here is an example configuration:

{
  "introspectionOptions": {
    "sampleSize": 1000,
    "noValidatorSchema": false,
    "allSchemaNullable": false
  },
  "serializationOptions": {
    "extendedJsonMode": "relaxed",
    "onResponseTypeMismatch": "skipRow"
  }
}

The skipRow behavior does not affect aggregations, or queries that do not request the field with the unexpected type.