这是indexloc提供的服务,不要输入任何密码
Skip to content

MSSQLToBigQuery incorrect type conversion bit-->integer #11874

@QuinRiva

Description

@QuinRiva

Apache Airflow version: 1.10.12

Environment: Centos 7 Host

  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release): Ubuntu 20.04 Docker Container
  • Kernel (e.g. uname -a): Linux c6c6e8230c17 3.10.0-1127.18.2.el7.x86_64 Improving the search functionality in the graph view #1 SMP Sun Jul 26 15:27:06 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools: Docker Compose
  • Others ?

What happened:

The schema file generated by MSSQLToGCSOperator specifies that a field defined as bit in MSSQL, is defined as INTEGER. However, the JSON file defines the value as true/false. When attempting to import into BigQuery using the GCSOperatorToBigQuery operator, the job fails:
Error while reading data, error message: JSON parsing error in row starting at position 0: Could not convert value 'boolean_value: false' to integer. Field: manager; Value: 0

What you expected to happen:

Value should be defined as BOOLEAN in the schema.

How to reproduce it:
Ensure dbo.Customers table has a bit column:

export_customers = MsSqlToGoogleCloudStorageOperator(
                task_id='export_customers',
                sql='SELECT * FROM dbo.Customers;',
                bucket='mssql-export',
                filename='data/customers/export.json',
                schema_filename='schemas/export.json',
                mssql_conn_id='mssql_default',
                google_cloud_storage_conn_id='google_cloud_default',
                dag=dag
            )

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions