-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Update data_loss_prevention_discovery_config to include field support for OtherCloudDiscoveryTarget #12114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Hello! I am a robot. Tests will require approval from a repository maintainer to run. @SirGitsalot, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look. You can help make sure that review is quick by doing a self-review and by running impacted tests locally. |
Hi there, I'm the Modular magician. I've detected the following information about your changes: Diff reportYour PR generated some diffs in downstreams - here they are.
Missing test reportYour PR includes resource fields which are not covered by any test. Resource: resource "google_data_loss_prevention_discovery_config" "primary" {
other_cloud_starting_location {
aws_location {
all_asset_inventory_assets = # value needed
}
}
targets {
other_cloud_target {
conditions {
amazon_s3_bucket_conditions {
bucket_types = # value needed
object_storage_classes = # value needed
}
}
filter {
single_resource {
amazon_s3_bucket {
aws_account {
account_id = # value needed
}
bucket_name = # value needed
}
}
}
}
}
}
|
Hi there, I'm the Modular magician. I've detected the following information about your changes: Diff reportYour PR generated some diffs in downstreams - here they are.
Missing test reportYour PR includes resource fields which are not covered by any test. Resource: resource "google_data_loss_prevention_discovery_config" "primary" {
other_cloud_starting_location {
aws_location {
all_asset_inventory_assets = # value needed
}
}
targets {
other_cloud_target {
conditions {
amazon_s3_bucket_conditions {
bucket_types = # value needed
object_storage_classes = # value needed
}
}
filter {
single_resource {
amazon_s3_bucket {
aws_account {
account_id = # value needed
}
bucket_name = # value needed
}
}
}
}
}
}
|
Tests analyticsTotal tests: 63 Click here to see the affected service packages
Action takenFound 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
|
1 similar comment
Tests analyticsTotal tests: 63 Click here to see the affected service packages
Action takenFound 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
|
🔴 Tests failed during RECORDING mode: 🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR. |
🔴 Tests failed during RECORDING mode: 🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR. |
Hi there, I'm the Modular magician. I've detected the following information about your changes: Diff reportYour PR generated some diffs in downstreams - here they are.
Missing test reportYour PR includes resource fields which are not covered by any test. Resource: resource "google_data_loss_prevention_discovery_config" "primary" {
other_cloud_starting_location {
aws_location {
all_asset_inventory_assets = # value needed
}
}
targets {
other_cloud_target {
conditions {
amazon_s3_bucket_conditions {
bucket_types = # value needed
object_storage_classes = # value needed
}
}
filter {
single_resource {
amazon_s3_bucket {
aws_account {
account_id = # value needed
}
bucket_name = # value needed
}
}
}
}
}
}
|
Tests analyticsTotal tests: 63 Click here to see the affected service packages
Action takenFound 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
|
🔴 Tests failed during RECORDING mode: 🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR. |
/gcbrun |
Hi there, I'm the Modular magician. I've detected the following information about your changes: Diff reportYour PR generated some diffs in downstreams - here they are.
Missing test reportYour PR includes resource fields which are not covered by any test. Resource: resource "google_data_loss_prevention_discovery_config" "primary" {
other_cloud_starting_location {
aws_location {
all_asset_inventory_assets = # value needed
}
}
targets {
other_cloud_target {
conditions {
amazon_s3_bucket_conditions {
bucket_types = # value needed
object_storage_classes = # value needed
}
}
filter {
single_resource {
amazon_s3_bucket {
aws_account {
account_id = # value needed
}
bucket_name = # value needed
}
}
}
}
}
}
|
Tests analyticsTotal tests: 63 Click here to see the affected service packages
Action takenFound 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
|
🔴 Tests failed during RECORDING mode: 🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR. |
/gcbrun |
/gcbrun |
1 similar comment
/gcbrun |
/gcbrun |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One nit and a suggestion for a config addition, but otherwise LGTM!
- name: 'awsLocation' | ||
type: NestedObject | ||
properties: | ||
- name: 'accountId' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
According to the docs it's possible to specify accountId
or allAssetInventoryAssets
, but not both. I'm not sure if leaving both unset is valid, but in either case you can improve the user experience by adding either:
- conflicts to both fields, if they're allowed to be unset
- exactly_one_of if one of them is required to be set
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Conflict added.
} | ||
resource "google_organization_iam_member" "dlp_role" { | ||
org_id = "%{organization}" | ||
role = "roles/dlp.orgdriver" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: there's some weird indentation in the test configs (GitHub doesn't show it, but I'll bet it's due to a combination of tabs and spaces)
@GoogleCloudPlatform/terraform-team @SirGitsalot This PR has been waiting for review for 2 weeks. Please take a look! Use the label |
@SirGitsalot sorry for the delay in putting in the requested changes, but can you take another look? |
@GoogleCloudPlatform/terraform-team @SirGitsalot This PR has been waiting for review for 3 weeks. Please take a look! Use the label |
Hi there, I'm the Modular magician. I've detected the following information about your changes: Diff reportYour PR generated some diffs in downstreams - here they are.
Missing test reportYour PR includes resource fields which are not covered by any test. Resource: resource "google_data_loss_prevention_discovery_config" "primary" {
other_cloud_starting_location {
aws_location {
all_asset_inventory_assets = # value needed
}
}
targets {
other_cloud_target {
filter {
single_resource {
amazon_s3_bucket {
aws_account {
account_id = # value needed
}
bucket_name = # value needed
}
}
}
}
}
}
|
Tests analyticsTotal tests: 63 Click here to see the affected service packages
Action takenFound 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
|
🔴 Tests failed during RECORDING mode: 🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the delay! I think I got so used to seeing this PR in my list that I became blind to it
The debug link for the failing test log isn't working for me, but I checked it directly and it's failing with an invalid argument:
---[ REQUEST ]---------------------------------------
POST /v2/organizations/529579013760/locations/us-central1/discoveryConfigs?alt=json HTTP/1.1
Host: dlp.googleapis.com
User-Agent: Terraform/1.11.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.36.0 terraform-provider-google-beta/acc
Content-Length: 588
Content-Type: application/json
Accept-Encoding: gzip
{
"discoveryConfig": {
"inspectTemplates": [
"projects/ci-test-project-188019/inspectTemplates/7702440615858682251"
],
"orgConfig": {
"location": {
"organizationId": "529579013760"
},
"projectId": "ci-test-project-188019"
},
"otherCloudStartingLocation": {
"awsLocation": {
"accountId": "012345678910"
}
},
"status": "RUNNING",
"targets": [
{
"otherCloudTarget": {
"dataSourceType": {
"dataSource": "aws/s3/bucket"
},
"disabled": null,
"filter": {
"others": {}
},
"generationCadence": {
"inspectTemplateModifiedCadence": {
"frequency": "UPDATE_FREQUENCY_MONTHLY"
},
"refreshFrequency": "UPDATE_FREQUENCY_MONTHLY"
}
},
"secretsTarget": null
}
]
}
}
-----------------------------------------------------
2025/05/16 00:06:33 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 400 Bad Request
Content-Type: application/json; charset=UTF-8
Date: Fri, 16 May 2025 00:06:33 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0
{
"error": {
"code": 400,
"message": "Duplicated discovery config of type AWS S3 buckets is not allowed. There is already another config with the same data profile location (Organization ID: 529579013760, AWS Account ID: 012345678910).",
"status": "INVALID_ARGUMENT",
"details": [
{
"@type": "type.googleapis.com/google.rpc.ErrorInfo",
"reason": "3",
"domain": "dlp.googleapis.com"
}
]
}
}
Is 012345678910
a special testing constant, or can we randomize that?
Hmmm, we could randomize it, but the bigger issue here is that the custom sweeper I wrote some time ago doesn't seem to be picking this up and deleting the config... we have an API limit on number of configs per profiling type, and the sweeper is supposed to delete these in case tests fail so we don't block subsequent runs. |
What I can do is run a backfill to delete the offending config on our end to unblock this PR (we just added support for this kind of troubleshooting action), but I'll have to take another look at the sweeper to see why it's not cleaning this up. |
I'll let you know when the backfiller is complete so we can rerun this test. |
@SirGitsalot The config has been removed; can re-run and see if it passes. |
/gcbrun |
@SirGitsalot This PR has been waiting for review for 3 weekdays. Please take a look! Use the label |
@GoogleCloudPlatform/terraform-team @SirGitsalot This PR has been waiting for review for 1 week. Please take a look! Use the label |
@GoogleCloudPlatform/terraform-team @SirGitsalot This PR has been waiting for review for 2 weeks. Please take a look! Use the label |
Hi there, I'm the Modular magician. I've detected the following information about your changes: Diff reportYour PR generated some diffs in downstreams - here they are.
Missing test reportYour PR includes resource fields which are not covered by any test. Resource: resource "google_data_loss_prevention_discovery_config" "primary" {
other_cloud_starting_location {
aws_location {
all_asset_inventory_assets = # value needed
}
}
targets {
other_cloud_target {
filter {
single_resource {
amazon_s3_bucket {
aws_account {
account_id = # value needed
}
bucket_name = # value needed
}
}
}
}
}
}
|
Tests analyticsTotal tests: 63 Click here to see the affected service packages
Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
|
🔴 Tests failed during RECORDING mode: 🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR. |
return acctest.Nprintf(` | ||
data "google_project" "project" { | ||
} | ||
resource "google_organization_iam_member" "org_admin" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I re-ran the test it's failing on these two google_organization_iam_member
resources. The service account that the tests run as doesn't have the org admin role to avoid an errant test nuking access to the whole org. Instead, when an org-level IAM role is needed we add it to main.tf
as you did above (although that's really just for documentation purposes and the actual change is made manually, which I'll do once I confirm what's needed).
If the service account running the test has the roles/dlp.admin
role and the DLP project-level service account has the roles/dlp.orgDriver
on the test organization, will that be sufficient? If so, I'll make the change for the former, and I believe that the latter will use a bootstrapped IAM resource.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That should be sufficient, yes. roles/dlp.orgDriver
was intended to cover the permissions needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added roles/dlp.admin
to the test runner service account, you'll need to add the bootstrap role a la:
bootstrap_iam:
- member: "serviceAccount:service-{project_number}@dlp-api.iam.gserviceaccount.com"
role: "roles/dlp.orgDriver"
@SirGitsalot This PR has been waiting for review for 3 weekdays. Please take a look! Use the label |
return acctest.Nprintf(` | ||
data "google_project" "project" { | ||
} | ||
resource "google_organization_iam_member" "org_admin" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added roles/dlp.admin
to the test runner service account, you'll need to add the bootstrap role a la:
bootstrap_iam:
- member: "serviceAccount:service-{project_number}@dlp-api.iam.gserviceaccount.com"
role: "roles/dlp.orgDriver"
@patrickmoy, this PR is waiting for action from you. If no action is taken, this PR will be closed in 28 days. Please address any comments or change requests, or re-request review from a core reviewer if no action is required. This notification can be disabled with the |
…veryTarget, which currently only supports AWS S3 buckets.
@SirGitsalot This PR has been waiting for review for 3 weekdays. Please take a look! Use the label |
@GoogleCloudPlatform/terraform-team @SirGitsalot This PR has been waiting for review for 1 week. Please take a look! Use the label |
@GoogleCloudPlatform/terraform-team @SirGitsalot This PR has been waiting for review for 2 weeks. Please take a look! Use the label |
Note that OtherCloudDiscoveryTarget currently only supports AWS S3 buckets in this update.
Release Note Template for Downstream PRs (will be copied)
See Write release notes for guidance.