这是indexloc提供的服务,不要输入任何密码
Skip to content

Update data_loss_prevention_discovery_config to include field support for OtherCloudDiscoveryTarget #12114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

patrickmoy
Copy link
Member

Note that OtherCloudDiscoveryTarget currently only supports AWS S3 buckets in this update.

Release Note Template for Downstream PRs (will be copied)

See Write release notes for guidance.

dlp: added `other_cloud_target` and `other_cloud_starting_location` to `google_data_loss_prevention_discovery_config`

@github-actions github-actions bot requested a review from SirGitsalot October 24, 2024 08:50
Copy link

Hello! I am a robot. Tests will require approval from a repository maintainer to run.

@SirGitsalot, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 3 files changed, 2202 insertions(+), 608 deletions(-))
google-beta provider: Diff ( 3 files changed, 2202 insertions(+), 608 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 512 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_data_loss_prevention_discovery_config (26 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_data_loss_prevention_discovery_config" "primary" {
  other_cloud_starting_location {
    aws_location {
      all_asset_inventory_assets = # value needed
    }
  }
  targets {
    other_cloud_target {
      conditions {
        amazon_s3_bucket_conditions {
          bucket_types           = # value needed
          object_storage_classes = # value needed
        }
      }
      filter {
        single_resource {
          amazon_s3_bucket {
            aws_account {
              account_id = # value needed
            }
            bucket_name = # value needed
          }
        }
      }
    }
  }
}

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 3 files changed, 2202 insertions(+), 608 deletions(-))
google-beta provider: Diff ( 3 files changed, 2202 insertions(+), 608 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 512 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_data_loss_prevention_discovery_config (26 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_data_loss_prevention_discovery_config" "primary" {
  other_cloud_starting_location {
    aws_location {
      all_asset_inventory_assets = # value needed
    }
  }
  targets {
    other_cloud_target {
      conditions {
        amazon_s3_bucket_conditions {
          bucket_types           = # value needed
          object_storage_classes = # value needed
        }
      }
      filter {
        single_resource {
          amazon_s3_bucket {
            aws_account {
              account_id = # value needed
            }
            bucket_name = # value needed
          }
        }
      }
    }
  }
}

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 63
Passed tests: 62
Skipped tests: 0
Affected tests: 1

Click here to see the affected service packages
  • datalossprevention

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
  • TestAccDataLossPreventionDiscoveryConfig_Update

Get to know how VCR tests work

1 similar comment
@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 63
Passed tests: 62
Skipped tests: 0
Affected tests: 1

Click here to see the affected service packages
  • datalossprevention

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
  • TestAccDataLossPreventionDiscoveryConfig_Update

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

🔴 Tests failed during RECORDING mode:
TestAccDataLossPreventionDiscoveryConfig_Update [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

@modular-magician
Copy link
Collaborator

🔴 Tests failed during RECORDING mode:
TestAccDataLossPreventionDiscoveryConfig_Update [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 3 files changed, 2202 insertions(+), 608 deletions(-))
google-beta provider: Diff ( 3 files changed, 2202 insertions(+), 608 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 512 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_data_loss_prevention_discovery_config (26 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_data_loss_prevention_discovery_config" "primary" {
  other_cloud_starting_location {
    aws_location {
      all_asset_inventory_assets = # value needed
    }
  }
  targets {
    other_cloud_target {
      conditions {
        amazon_s3_bucket_conditions {
          bucket_types           = # value needed
          object_storage_classes = # value needed
        }
      }
      filter {
        single_resource {
          amazon_s3_bucket {
            aws_account {
              account_id = # value needed
            }
            bucket_name = # value needed
          }
        }
      }
    }
  }
}

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 63
Passed tests: 62
Skipped tests: 0
Affected tests: 1

Click here to see the affected service packages
  • datalossprevention

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
  • TestAccDataLossPreventionDiscoveryConfig_Update

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

🔴 Tests failed during RECORDING mode:
TestAccDataLossPreventionDiscoveryConfig_Update [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

@patrickmoy patrickmoy marked this pull request as draft October 24, 2024 18:59
@patrickmoy
Copy link
Member Author

/gcbrun

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 3 files changed, 2202 insertions(+), 608 deletions(-))
google-beta provider: Diff ( 3 files changed, 2202 insertions(+), 608 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 512 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_data_loss_prevention_discovery_config (26 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_data_loss_prevention_discovery_config" "primary" {
  other_cloud_starting_location {
    aws_location {
      all_asset_inventory_assets = # value needed
    }
  }
  targets {
    other_cloud_target {
      conditions {
        amazon_s3_bucket_conditions {
          bucket_types           = # value needed
          object_storage_classes = # value needed
        }
      }
      filter {
        single_resource {
          amazon_s3_bucket {
            aws_account {
              account_id = # value needed
            }
            bucket_name = # value needed
          }
        }
      }
    }
  }
}

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 63
Passed tests: 62
Skipped tests: 0
Affected tests: 1

Click here to see the affected service packages
  • datalossprevention

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
  • TestAccDataLossPreventionDiscoveryConfig_Update

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

🔴 Tests failed during RECORDING mode:
TestAccDataLossPreventionDiscoveryConfig_Update [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

@patrickmoy
Copy link
Member Author

/gcbrun

@patrickmoy
Copy link
Member Author

/gcbrun

1 similar comment
@patrickmoy
Copy link
Member Author

/gcbrun

@patrickmoy patrickmoy marked this pull request as ready for review November 19, 2024 08:02
@patrickmoy
Copy link
Member Author

/gcbrun

@patrickmoy patrickmoy marked this pull request as draft November 19, 2024 18:59
@patrickmoy patrickmoy marked this pull request as ready for review November 19, 2024 20:25
Copy link
Member

@SirGitsalot SirGitsalot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nit and a suggestion for a config addition, but otherwise LGTM!

- name: 'awsLocation'
type: NestedObject
properties:
- name: 'accountId'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the docs it's possible to specify accountId or allAssetInventoryAssets, but not both. I'm not sure if leaving both unset is valid, but in either case you can improve the user experience by adding either:

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conflict added.

}
resource "google_organization_iam_member" "dlp_role" {
org_id = "%{organization}"
role = "roles/dlp.orgdriver"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: there's some weird indentation in the test configs (GitHub doesn't show it, but I'll bet it's due to a combination of tabs and spaces)

Copy link

github-actions bot commented May 2, 2025

@GoogleCloudPlatform/terraform-team @SirGitsalot This PR has been waiting for review for 2 weeks. Please take a look! Use the label disable-review-reminders to disable these notifications.

@patrickmoy
Copy link
Member Author

@SirGitsalot sorry for the delay in putting in the requested changes, but can you take another look?

Copy link

github-actions bot commented May 9, 2025

@GoogleCloudPlatform/terraform-team @SirGitsalot This PR has been waiting for review for 3 weeks. Please take a look! Use the label disable-review-reminders to disable these notifications.

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 2208 insertions(+), 615 deletions(-))
google-beta provider: Diff ( 4 files changed, 2208 insertions(+), 615 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 501 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_data_loss_prevention_discovery_config (26 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_data_loss_prevention_discovery_config" "primary" {
  other_cloud_starting_location {
    aws_location {
      all_asset_inventory_assets = # value needed
    }
  }
  targets {
    other_cloud_target {
      filter {
        single_resource {
          amazon_s3_bucket {
            aws_account {
              account_id = # value needed
            }
            bucket_name = # value needed
          }
        }
      }
    }
  }
}

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 63
Passed tests: 62
Skipped tests: 0
Affected tests: 1

Click here to see the affected service packages
  • datalossprevention

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
  • TestAccDataLossPreventionDiscoveryConfig_Update

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

🔴 Tests failed during RECORDING mode:
TestAccDataLossPreventionDiscoveryConfig_Update [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

Copy link
Member

@SirGitsalot SirGitsalot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay! I think I got so used to seeing this PR in my list that I became blind to it ☹️

The debug link for the failing test log isn't working for me, but I checked it directly and it's failing with an invalid argument:

---[ REQUEST ]---------------------------------------
POST /v2/organizations/529579013760/locations/us-central1/discoveryConfigs?alt=json HTTP/1.1
Host: dlp.googleapis.com
User-Agent: Terraform/1.11.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.36.0 terraform-provider-google-beta/acc
Content-Length: 588
Content-Type: application/json
Accept-Encoding: gzip

{
 "discoveryConfig": {
  "inspectTemplates": [
   "projects/ci-test-project-188019/inspectTemplates/7702440615858682251"
  ],
  "orgConfig": {
   "location": {
    "organizationId": "529579013760"
   },
   "projectId": "ci-test-project-188019"
  },
  "otherCloudStartingLocation": {
   "awsLocation": {
    "accountId": "012345678910"
   }
  },
  "status": "RUNNING",
  "targets": [
   {
    "otherCloudTarget": {
     "dataSourceType": {
      "dataSource": "aws/s3/bucket"
     },
     "disabled": null,
     "filter": {
      "others": {}
     },
     "generationCadence": {
      "inspectTemplateModifiedCadence": {
       "frequency": "UPDATE_FREQUENCY_MONTHLY"
      },
      "refreshFrequency": "UPDATE_FREQUENCY_MONTHLY"
     }
    },
    "secretsTarget": null
   }
  ]
 }
}

-----------------------------------------------------
2025/05/16 00:06:33 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 400 Bad Request
Content-Type: application/json; charset=UTF-8
Date: Fri, 16 May 2025 00:06:33 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "error": {
    "code": 400,
    "message": "Duplicated discovery config of type AWS S3 buckets is not allowed. There is already another config with the same data profile location (Organization ID: 529579013760, AWS Account ID: 012345678910).",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "3",
        "domain": "dlp.googleapis.com"
      }
    ]
  }
}

Is 012345678910 a special testing constant, or can we randomize that?

@patrickmoy
Copy link
Member Author

Sorry for the delay! I think I got so used to seeing this PR in my list that I became blind to it ☹️

The debug link for the failing test log isn't working for me, but I checked it directly and it's failing with an invalid argument:

---[ REQUEST ]---------------------------------------
POST /v2/organizations/529579013760/locations/us-central1/discoveryConfigs?alt=json HTTP/1.1
Host: dlp.googleapis.com
User-Agent: Terraform/1.11.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.36.0 terraform-provider-google-beta/acc
Content-Length: 588
Content-Type: application/json
Accept-Encoding: gzip

{
 "discoveryConfig": {
  "inspectTemplates": [
   "projects/ci-test-project-188019/inspectTemplates/7702440615858682251"
  ],
  "orgConfig": {
   "location": {
    "organizationId": "529579013760"
   },
   "projectId": "ci-test-project-188019"
  },
  "otherCloudStartingLocation": {
   "awsLocation": {
    "accountId": "012345678910"
   }
  },
  "status": "RUNNING",
  "targets": [
   {
    "otherCloudTarget": {
     "dataSourceType": {
      "dataSource": "aws/s3/bucket"
     },
     "disabled": null,
     "filter": {
      "others": {}
     },
     "generationCadence": {
      "inspectTemplateModifiedCadence": {
       "frequency": "UPDATE_FREQUENCY_MONTHLY"
      },
      "refreshFrequency": "UPDATE_FREQUENCY_MONTHLY"
     }
    },
    "secretsTarget": null
   }
  ]
 }
}

-----------------------------------------------------
2025/05/16 00:06:33 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 400 Bad Request
Content-Type: application/json; charset=UTF-8
Date: Fri, 16 May 2025 00:06:33 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "error": {
    "code": 400,
    "message": "Duplicated discovery config of type AWS S3 buckets is not allowed. There is already another config with the same data profile location (Organization ID: 529579013760, AWS Account ID: 012345678910).",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "3",
        "domain": "dlp.googleapis.com"
      }
    ]
  }
}

Is 012345678910 a special testing constant, or can we randomize that?

Hmmm, we could randomize it, but the bigger issue here is that the custom sweeper I wrote some time ago doesn't seem to be picking this up and deleting the config... we have an API limit on number of configs per profiling type, and the sweeper is supposed to delete these in case tests fail so we don't block subsequent runs.

@github-actions github-actions bot requested a review from SirGitsalot May 19, 2025 22:05
@patrickmoy
Copy link
Member Author

patrickmoy commented May 19, 2025

What I can do is run a backfill to delete the offending config on our end to unblock this PR (we just added support for this kind of troubleshooting action), but I'll have to take another look at the sweeper to see why it's not cleaning this up.

@patrickmoy
Copy link
Member Author

I'll let you know when the backfiller is complete so we can rerun this test.

@patrickmoy
Copy link
Member Author

@SirGitsalot The config has been removed; can re-run and see if it passes.

@patrickmoy
Copy link
Member Author

/gcbrun

Copy link

@SirGitsalot This PR has been waiting for review for 3 weekdays. Please take a look! Use the label disable-review-reminders to disable these notifications.

Copy link

@GoogleCloudPlatform/terraform-team @SirGitsalot This PR has been waiting for review for 1 week. Please take a look! Use the label disable-review-reminders to disable these notifications.

Copy link

github-actions bot commented Jun 2, 2025

@GoogleCloudPlatform/terraform-team @SirGitsalot This PR has been waiting for review for 2 weeks. Please take a look! Use the label disable-review-reminders to disable these notifications.

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 2208 insertions(+), 615 deletions(-))
google-beta provider: Diff ( 4 files changed, 2208 insertions(+), 615 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 501 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_data_loss_prevention_discovery_config (26 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_data_loss_prevention_discovery_config" "primary" {
  other_cloud_starting_location {
    aws_location {
      all_asset_inventory_assets = # value needed
    }
  }
  targets {
    other_cloud_target {
      filter {
        single_resource {
          amazon_s3_bucket {
            aws_account {
              account_id = # value needed
            }
            bucket_name = # value needed
          }
        }
      }
    }
  }
}

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 63
Passed tests: 62
Skipped tests: 0
Affected tests: 1

Click here to see the affected service packages
  • datalossprevention
#### Action taken
Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
  • TestAccDataLossPreventionDiscoveryConfig_Update

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

🔴 Tests failed during RECORDING mode:
TestAccDataLossPreventionDiscoveryConfig_Update [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

return acctest.Nprintf(`
data "google_project" "project" {
}
resource "google_organization_iam_member" "org_admin" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-ran the test it's failing on these two google_organization_iam_member resources. The service account that the tests run as doesn't have the org admin role to avoid an errant test nuking access to the whole org. Instead, when an org-level IAM role is needed we add it to main.tf as you did above (although that's really just for documentation purposes and the actual change is made manually, which I'll do once I confirm what's needed).

If the service account running the test has the roles/dlp.admin role and the DLP project-level service account has the roles/dlp.orgDriver on the test organization, will that be sufficient? If so, I'll make the change for the former, and I believe that the latter will use a bootstrapped IAM resource.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That should be sufficient, yes. roles/dlp.orgDriver was intended to cover the permissions needed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added roles/dlp.admin to the test runner service account, you'll need to add the bootstrap role a la:

bootstrap_iam:
      - member: "serviceAccount:service-{project_number}@dlp-api.iam.gserviceaccount.com"
        role: "roles/dlp.orgDriver"

@patrickmoy patrickmoy requested a review from SirGitsalot June 13, 2025 20:56
Copy link

@SirGitsalot This PR has been waiting for review for 3 weekdays. Please take a look! Use the label disable-review-reminders to disable these notifications.

return acctest.Nprintf(`
data "google_project" "project" {
}
resource "google_organization_iam_member" "org_admin" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added roles/dlp.admin to the test runner service account, you'll need to add the bootstrap role a la:

bootstrap_iam:
      - member: "serviceAccount:service-{project_number}@dlp-api.iam.gserviceaccount.com"
        role: "roles/dlp.orgDriver"

Copy link

github-actions bot commented Jul 2, 2025

@patrickmoy, this PR is waiting for action from you. If no action is taken, this PR will be closed in 28 days.

Please address any comments or change requests, or re-request review from a core reviewer if no action is required.

Image showing the re-request review button

This notification can be disabled with the disable-automatic-closure label.

…veryTarget, which currently only supports AWS S3 buckets.
Copy link

@SirGitsalot This PR has been waiting for review for 3 weekdays. Please take a look! Use the label disable-review-reminders to disable these notifications.

Copy link

@GoogleCloudPlatform/terraform-team @SirGitsalot This PR has been waiting for review for 1 week. Please take a look! Use the label disable-review-reminders to disable these notifications.

Copy link

@GoogleCloudPlatform/terraform-team @SirGitsalot This PR has been waiting for review for 2 weeks. Please take a look! Use the label disable-review-reminders to disable these notifications.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants