fivetran · fivetran-joemarkiewicz · Feb 3, 2021 · Feb 1, 2021 · Feb 1, 2021 · Feb 1, 2021
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -0,0 +1,60 @@
+version: 2
+
+jobs:
+  build:
+    docker:
+      - image: circleci/python:3.7.9-stretch
+
+    steps:
+      - checkout
+
+      - run:
+          run: setup_creds
+          command: |
+            echo $GCLOUD_SERVICE_KEY | base64 --decode --ignore-garbage > ${HOME}/gcloud-service-key.json
+      - run:
+          name: "Setup dbt"
+          command: |
+            python3 -m venv venv
+            . venv/bin/activate
+            pip install --upgrade pip setuptools
+            pip install -r integration_tests/requirements.txt
+            mkdir -p ~/.dbt
+            cp integration_tests/ci/sample.profiles.yml ~/.dbt/profiles.yml
+      - run:
+          name: "Run Tests - Redshift"
+          command: |
+            . venv/bin/activate
+            echo `pwd`
+            cd integration_tests
+            dbt deps
+            dbt seed --target redshift --full-refresh
+            dbt run --target redshift --full-refresh
+            dbt test --target redshift
+      - run:
+          name: "Run Tests - Snowflake"
+          command: |
+            . venv/bin/activate
+            echo `pwd`
+            cd integration_tests
+            dbt deps
+            dbt seed --target snowflake --full-refresh
+            dbt run --target snowflake --full-refresh
+            dbt test --target snowflake
+      - run:
+          name: "Run Tests - BigQuery"
+          environment:
+              GCLOUD_SERVICE_KEY_PATH: "/home/circleci/gcloud-service-key.json"
+
+          command: |
+            . venv/bin/activate
+            echo `pwd`
+            cd integration_tests
+            dbt deps
+            dbt seed --target bigquery
+            dbt run --target bigquery --full-refresh
+            dbt test --target bigquery
+      - save_cache:
+          key: deps2-{{ .Branch }}
+          paths:
+            - "venv"
diff --git a/README.md b/README.md
@@ -1,4 +1,4 @@
-# GitHub 
+# GitHub ([Docs](https://dbt-github.netlify.app/))
 
 This package models GitHub data from [Fivetran's connector](https://fivetran.com/docs/applications/github). It uses data in the format described by [this ERD](https://docs.google.com/presentation/d/1lx6ez7-x-s-n2JCnCi3SjG4XMmx9ysNUvaNCaWc3I_I/edit).
 
@@ -10,12 +10,12 @@ This package contains transformation models, designed to work simultaneously wit
 
 | **model**                  | **description**                                                                                                                                               |
 | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| github\_issues             | Each record represents a GitHub issue, enriched with data about its assignees, milestones, and time comparisons.                                             |
-| github\_pull\_requests     | Each record represents a GitHub pull request, enriched with data about its repository, reviewers, and durations between review requests, merges and reviews. |
-| github\_daily\_metrics     | Each record represents a single day, enriched with metrics about PRs and issues that were created and closed during that period.                              |
-| github\_weekly\_metrics    | Each record represents a single week, enriched with metrics about PRs and issues that were created and closed during that period.                             |
-| github\_monthly\_metrics   | Each record represents a single month, enriched with metrics about PRs and issues that were created and closed during that period.                            |
-| github\_quarterly\_metrics | Each record represents a single quarter, enriched with metrics about PRs and issues that were created and closed during that period.                          |
+| [github__issues](https://github.com/fivetran/dbt_github/blob/master/models/github__issues.sql)             | Each record represents a GitHub issue, enriched with data about its assignees, milestones, and time comparisons.                                             |
+| [github__pull_requests](https://github.com/fivetran/dbt_github/blob/master/models/github__pull_requests.sql)     | Each record represents a GitHub pull request, enriched with data about its repository, reviewers, and durations between review requests, merges and reviews. |
+| [github__daily_metrics](https://github.com/fivetran/dbt_github/blob/master/models/github__daily_metrics.sql)     | Each record represents a single day, enriched with metrics about PRs and issues that were created and closed during that period.                              |
+| [github__weekly_metrics](https://github.com/fivetran/dbt_github/blob/master/models/github__weekly_metrics.sql)    | Each record represents a single week, enriched with metrics about PRs and issues that were created and closed during that period.                             |
+| [github__monthly_metrics](https://github.com/fivetran/dbt_github/blob/master/models/github__monthly_metrics.sql)   | Each record represents a single month, enriched with metrics about PRs and issues that were created and closed during that period.                            |
+| [github__quarterly_metrics](https://github.com/fivetran/dbt_github/blob/master/models/github__quarterly_metrics.sql) | Each record represents a single quarter, enriched with metrics about PRs and issues that were created and closed during that period.                          |
 
 
 ## Installation Instructions
@@ -36,6 +36,9 @@ vars:
     github_schema: your_schema_name 
 ```
 
+## Database support
+This package has been tested on BigQuery, Snowflake and Redshift.
+
 ## Contributions
 
 Additional contributions to this package are very welcome! Please create issues

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,16 +1,26 @@
 config-version: 2
 
 name: 'github'
-version: '0.0.1'
+version: '0.2.0'
 
-require-dbt-version: [">=0.18.0", "<0.19.0"]
+require-dbt-version: [">=0.18.0", "<0.20.0"] 
 
 models:
     github:
         materialized: table
         intermediate:
             materialized: ephemeral
-
+ 
 vars:
-    dbt_utils_dispatch_list:
-        - fivetran_utils
+    github:
+        issue_assignee:                 "{{ ref('stg_github__issue_assignee') }}"
+        issue_closed_history:           "{{ ref('stg_github__issue_closed_history') }}"
+        issue_comment:                  "{{ ref('stg_github__issue_comment') }}"
+        issue_label:                    "{{ ref('stg_github__issue_label') }}"
+        issue_merged:                   "{{ ref('stg_github__issue_merged') }}"
+        issue:                          "{{ ref('stg_github__issue') }}"
+        pull_request_review:            "{{ ref('stg_github__pull_request_review') }}"
+        pull_request:                   "{{ ref('stg_github__pull_request') }}"
+        repository:                     "{{ ref('stg_github__repository') }}"
+        requested_reviewer_history:     "{{ ref('stg_github__requested_reviewer_history') }}"
+        user:                           "{{ ref('stg_github__user') }}"
diff --git a/integration_tests/.gitignore b/integration_tests/.gitignore
@@ -0,0 +1,3 @@
+target/
+dbt_modules/
+logs/
diff --git a/integration_tests/ci/sample.profiles.yml b/integration_tests/ci/sample.profiles.yml
@@ -0,0 +1,37 @@
+
+# HEY! This file is used in the Github integrations tests with CircleCI.
+# You should __NEVER__ check credentials into version control. Thanks for reading :)
+
+config:
+    send_anonymous_usage_stats: False
+    use_colors: True
+
+integration_tests:
+  target: snowflake
+  outputs:
+    redshift:
+      type: redshift
+      host: "{{ env_var('CI_REDSHIFT_DBT_HOST') }}"
+      user: "{{ env_var('CI_REDSHIFT_DBT_USER') }}"
+      pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
+      dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
+      port: 5439
+      schema: github_integration_tests
+      threads: 8
+    bigquery:
+      type: bigquery
+      method: service-account
+      keyfile: "{{ env_var('GCLOUD_SERVICE_KEY_PATH') }}"
+      project: 'dbt-package-testing'
+      schema: github_integration_tests
+      threads: 8
+    snowflake:
+      type: snowflake
+      account: "{{ env_var('CI_SNOWFLAKE_DBT_ACCOUNT') }}"
+      user: "{{ env_var('CI_SNOWFLAKE_DBT_USER') }}"
+      password: "{{ env_var('CI_SNOWFLAKE_DBT_PASS') }}"
+      role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
+      database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
+      warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
+      schema: github_integration_tests
+      threads: 8
diff --git a/integration_tests/data/github_issue_assignee_data.csv b/integration_tests/data/github_issue_assignee_data.csv
@@ -0,0 +1,201 @@
+issue_id,user_id,_fivetran_synced
+10000,38281398,2019-03-11 20:49:15
+10001,43736569,2019-04-25 18:00:51
+10002,92460780,2020-08-17 21:03:16
+10003,19409605,2020-10-01 21:38:31
+10004,73074108,2019-07-22 15:33:26
+10005,59353520,2020-03-19 09:59:20
+10006,14935991,2019-05-19 09:51:04
+10007,53845604,2020-01-19 03:29:36
+10008,28317735,2018-12-27 16:11:01
+10009,42978371,2019-06-18 11:56:32
+10010,52898325,2019-05-11 22:45:55
+10011,18589335,2019-04-01 16:14:01
+10012,80191436,2019-10-13 23:24:05
+10013,15033114,2020-05-04 21:29:25
+10014,30199043,2019-03-07 23:02:49
+10015,31378258,2020-06-04 02:32:37
+10016,84297715,2020-02-16 17:04:13
+10017,44084707,2019-01-04 14:20:24
+10018,30108149,2019-02-17 04:12:45
+10019,31346311,2020-07-05 07:45:17
+10020,98342009,2019-02-18 13:16:10
+10021,86112917,2019-05-14 18:18:04
+10022,41460805,2020-03-30 11:10:31
+10023,98279508,2020-04-01 22:26:53
+10024,16168378,2019-12-02 04:03:46
+10025,89806879,2019-03-28 09:30:09
+10026,87138714,2020-06-14 22:43:57
+10027,96871370,2019-09-28 08:46:48
+10028,46491513,2019-08-12 06:31:20
+10029,22309236,2019-12-09 08:15:21
+10030,37337268,2020-07-04 00:24:59
+10031,99923764,2020-04-05 10:18:22
+10032,90574197,2020-01-16 19:20:18
+10033,59205031,2019-09-22 09:53:23
+10034,36975661,2020-07-01 08:06:40
+10035,56158893,2019-06-23 08:34:46
+10036,99983382,2020-01-01 10:42:34
+10037,30750060,2018-12-29 00:00:34
+10038,88913977,2020-04-20 15:23:47
+10039,63081420,2020-07-26 16:52:05
+10040,86377861,2020-02-17 16:17:44
+10041,91725691,2019-10-16 11:06:29
+10042,50398513,2019-12-28 20:55:35
+10043,72607497,2018-12-23 00:55:12
+10044,35902283,2019-05-14 12:12:47
+10045,49743081,2020-01-26 14:09:01
+10046,68001590,2020-06-14 10:43:00
+10047,66278027,2019-11-07 22:41:18
+10048,91151740,2019-05-10 09:30:02
+10049,47826241,2020-01-21 07:05:28
+10050,19981172,2020-07-15 06:31:59
+10051,51390142,2020-08-15 05:34:44
+10052,64170120,2019-06-12 00:22:53
+10053,21393364,2020-09-15 13:52:02
+10054,37094730,2019-06-11 00:27:57
+10055,31624231,2020-08-05 14:57:43
+10056,81209478,2018-12-24 08:11:13
+10057,74788915,2020-02-04 05:27:37
+10058,90294292,2019-01-12 01:46:19
+10059,30474858,2019-02-17 10:37:07
+10060,86478820,2019-12-24 20:58:28
+10061,92430810,2020-03-07 22:36:20
+10062,40855544,2019-08-20 16:07:28
+10063,42715690,2019-03-15 15:45:20
+10064,52929818,2018-12-01 22:44:26
+10065,26403175,2019-11-13 05:48:12
+10066,46583166,2019-04-28 00:53:54
+10067,25210116,2019-09-03 15:00:32
+10068,87563932,2019-06-12 02:38:40
+10069,61465222,2019-09-19 05:41:20
+10070,91865157,2019-08-24 17:26:23
+10071,56376022,2019-01-09 08:12:53
+10072,72688759,2020-06-16 06:37:35
+10073,27788681,2020-06-18 01:10:41
+10074,12256951,2020-11-11 07:10:03
+10075,14581598,2020-02-18 03:00:35
+10076,28028941,2020-01-13 01:49:14
+10077,77681300,2019-05-15 23:16:38
+10078,36416688,2020-10-29 01:32:32
+10079,55303012,2020-05-22 16:03:48
+10080,29598563,2019-11-24 21:26:16
+10081,44133504,2019-04-05 12:57:11
+10082,20791469,2020-05-30 01:54:00
+10083,81574158,2020-05-31 05:04:30
+10084,19787842,2020-05-03 20:24:17
+10085,90643572,2018-12-07 16:55:19
+10086,12348823,2019-10-13 07:37:34
+10087,28139846,2019-02-02 20:40:03
+10088,14635176,2020-02-04 14:58:14
+10089,76037581,2018-11-25 15:06:32
+10090,65682092,2019-10-15 01:54:29
+10091,97081735,2020-10-19 09:48:54
+10092,23599969,2019-08-15 07:37:35
+10093,34316812,2019-09-19 11:50:44
+10094,95286905,2020-04-07 13:30:03
+10095,29073513,2020-01-12 07:45:19
+10096,12612870,2020-03-11 08:37:46
+10097,20631610,2019-02-19 13:40:17
+10098,78376220,2019-06-18 04:06:34
+10099,82344243,2020-09-12 19:52:33
+10100,57614451,2020-09-21 12:16:39
+10101,30430945,2019-01-11 09:05:09
+10102,68972052,2019-01-22 00:26:32
+10103,36460660,2019-08-12 06:30:40
+10104,61512555,2020-08-03 14:12:08
+10105,57923678,2020-10-08 14:47:11
+10106,84115939,2019-03-13 03:22:16
+10107,80587892,2019-03-08 09:23:29
+10108,52949406,2020-07-25 19:06:54
+10109,53226414,2020-07-19 22:57:58
+10110,49082771,2019-08-11 17:46:28
+10111,84424380,2019-08-21 10:50:37
+10112,86458959,2020-09-21 03:33:18
+10113,35325270,2019-09-21 23:15:21
+10114,80800311,2019-10-04 23:31:46
+10115,65446804,2019-12-07 07:22:27
+10116,87483148,2019-04-12 07:54:28
+10117,78813121,2019-09-26 07:56:50
+10118,21602840,2020-03-27 02:50:38
+10119,69466835,2018-11-28 23:30:13
+10120,68814258,2019-08-07 19:39:42
+10121,87100840,2019-06-30 12:36:17
+10122,94405288,2020-05-22 07:58:11
+10123,54872027,2019-06-17 02:33:45
+10124,39781348,2019-02-24 12:17:05
+10125,41609189,2019-08-23 13:56:17
+10126,21692130,2020-07-08 13:06:28
+10127,24177381,2020-07-23 13:07:28
+10128,57164067,2019-08-11 20:50:55
+10129,13830677,2020-01-19 00:24:59
+10130,35081542,2020-05-17 17:36:18
+10131,37153137,2020-09-10 00:11:22
+10132,89626307,2020-11-07 04:09:40
+10133,89258638,2020-02-24 10:59:08
+10134,44512425,2019-09-22 01:29:50
+10135,70787175,2020-04-13 14:01:43
+10136,57352034,2020-02-25 17:13:50
+10137,22987038,2019-09-23 18:29:50
+10138,47404962,2020-10-21 20:07:18
+10139,32245841,2020-02-10 06:45:43
+10140,65993587,2019-09-25 11:30:39
+10141,52169960,2020-06-15 23:57:59
+10142,49712883,2019-04-22 11:41:42
+10143,55763315,2019-10-26 01:36:41
+10144,91824922,2019-06-03 02:07:42
+10145,10290077,2019-01-05 19:06:44
+10146,44645698,2020-06-01 06:36:01
+10147,64659805,2019-11-24 03:45:57
+10148,79114850,2019-12-26 08:46:59
+10149,91146828,2020-01-27 08:26:32
+10150,45063586,2020-05-21 03:32:19
+10151,27258297,2019-10-27 09:59:55
+10152,68579066,2019-05-22 01:30:39
+10153,43010060,2019-06-03 22:21:31
+10154,28022791,2019-06-21 08:04:34
+10155,49206879,2019-01-29 06:58:52
+10156,22083716,2019-10-25 15:16:26
+10157,73079906,2020-07-28 19:12:46
+10158,20571892,2018-12-23 13:40:39
+10159,56080575,2019-12-11 22:54:52
+10160,24064038,2019-08-08 01:16:46
+10161,73304449,2019-01-08 05:59:02
+10162,70714183,2019-05-21 11:01:35
+10163,39214961,2019-12-05 14:52:38
+10164,71039263,2019-01-01 20:17:35
+10165,11868641,2019-07-07 18:35:57
+10166,65860950,2019-07-22 18:44:47
+10167,50578018,2019-02-21 20:13:38
+10168,11780928,2019-02-12 10:59:21
+10169,68527839,2020-08-06 19:48:10
+10170,79460996,2020-02-15 16:43:23
+10171,61641266,2019-03-15 05:36:07
+10172,95195996,2020-11-13 19:53:14
+10173,19444625,2020-06-12 03:17:41
+10174,71815695,2018-12-18 22:04:02
+10175,90356424,2019-08-03 23:03:08
+10176,38386598,2020-11-07 12:36:31
+10177,84014117,2019-09-30 11:18:33
+10178,51293749,2019-01-30 00:24:12
+10179,97161007,2019-02-01 17:40:32
+10180,83321903,2018-11-20 22:26:00
+10181,22467825,2020-02-04 14:52:12
+10182,16831786,2020-10-04 03:26:12
+10183,63549782,2019-08-22 13:06:23
+10184,97591301,2020-02-25 22:50:07
+10185,59127368,2019-03-25 06:28:28
+10186,97091130,2019-06-16 16:49:47
+10187,53965836,2020-04-10 03:13:38
+10188,27887327,2020-02-05 12:13:09
+10189,56558553,2018-12-28 14:39:41
+10190,31650796,2019-04-15 17:48:54
+10191,47444758,2020-08-10 20:02:24
+10192,94347811,2019-08-13 11:44:06
+10193,67365791,2020-03-02 12:25:29
+10194,65502315,2020-02-27 03:57:17
+10195,90685793,2019-08-31 14:31:48
+10196,96029214,2020-01-07 06:11:33
+10197,53818969,2020-06-03 01:08:54
+10198,61362807,2019-02-17 18:24:39
+10199,42911811,2020-09-08 09:03:20