Skip to content

Commit 44758c2

Browse files
felipecrescencio-citdaniel-citbharathkkb
authored
fix: support logbucket sink in same project (#118)
* Fix Copyright year * New variable to deal with sink and logbucket in same project * When sink and logbucket in same project no new service account is created * New examples for logbucket at project-level * Changed name of new variable to a meaningful name * Created example of logbucket with project usage * Changed name of bucket due to colisions during integrated tests * Setup a new project for logbucket project testing in same and other projects * Integrated tests to Logbucket Project example * Revert mischanged copyright year * Shorten variable names * Fix logbucket/project example README.md Co-authored-by: Daniel Andrade <dandrade@ciandt.com> * Fix logbucket/project example README.md Co-authored-by: Daniel Andrade <dandrade@ciandt.com> * Revert mischanged copyright year * Inline constants * Changed variables creation place to right before usage * Remove retention_days to keep it default user better understanding example * -Created constant defaultRetentionDays and kept both examples with this default for user better understanding code and example -Changed assert bucket name to remove substring and compare to a built string fully qualified name -Combined similar asserts to tabular * Removed because empty is string default value Co-authored-by: Daniel Andrade <dandrade@ciandt.com> * Removed sink prefix because there is only one writerIdentity Co-authored-by: Daniel Andrade <dandrade@ciandt.com> * Fix assert writerIdentity message for examples Co-authored-by: Daniel Andrade <dandrade@ciandt.com> Co-authored-by: Bharath KKB <bharathkrishnakb@gmail.com>
1 parent ef43513 commit 44758c2

File tree

13 files changed

+395
-3
lines changed

13 files changed

+395
-3
lines changed

‎build/int.cloudbuild.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,21 @@ steps:
6969
- go-verify-logbucket-org
7070
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
7171
args: ['/bin/bash', '-c', 'cft test run TestLogBucketOrgModule --stage teardown --verbose']
72+
- id: go-apply-logbucket-project
73+
waitFor:
74+
- init-all
75+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
76+
args: ['/bin/bash', '-c', 'cft test run TestLogBucketProjectModule --stage apply --verbose']
77+
- id: go-verify-logbucket-project
78+
waitFor:
79+
- go-apply-logbucket-org
80+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
81+
args: ['/bin/bash', '-c', 'cft test run TestLogBucketProjectModule --stage verify --verbose']
82+
- id: go-teardown-logbucket-project
83+
waitFor:
84+
- go-verify-logbucket-project
85+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
86+
args: ['/bin/bash', '-c', 'cft test run TestLogBucketProjectModule --stage teardown --verbose']
7287
tags:
7388
- 'ci'
7489
- 'integration'

‎examples/logbucket/project/README.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Log Export: Log Bucket destination at Project level
2+
3+
These examples configures a project-level log sink that feeds a logging log bucket destination with log bucket and log sink in the same project or in separated projects.
4+
5+
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
6+
## Inputs
7+
8+
| Name | Description | Type | Default | Required |
9+
|------|-------------|------|---------|:--------:|
10+
| parent\_resource\_project | The ID of the project in which the log export will be created. | `string` | n/a | yes |
11+
| project\_destination\_logbkt\_id | The ID of the project in which log bucket destination will be created. | `string` | n/a | yes |
12+
13+
## Outputs
14+
15+
| Name | Description |
16+
|------|-------------|
17+
| log\_bkt\_name\_same\_proj | The name for the log bucket for sink and logbucket in same project example. |
18+
| log\_bkt\_same\_proj | The project where the log bucket is created for sink and logbucket in same project example. |
19+
| log\_bucket\_name | The name for the log bucket. |
20+
| log\_bucket\_project | The project where the log bucket is created. |
21+
| log\_sink\_dest\_uri\_same\_proj | A fully qualified URI for the log sink for sink and logbucket in same project example. |
22+
| log\_sink\_destination\_uri | A fully qualified URI for the log sink. |
23+
| log\_sink\_id\_same\_proj | The project id where the log sink is created for sink and logbucket in same project example. |
24+
| log\_sink\_project\_id | The project id where the log sink is created. |
25+
| log\_sink\_resource\_name | The resource name of the log sink that was created. |
26+
| log\_sink\_resource\_name\_same\_proj | The resource name of the log sink that was created in same project example. |
27+
| log\_sink\_writer\_identity | The service account that logging uses to write log entries to the destination. |
28+
| log\_sink\_writer\_identity\_same\_proj | The service account in same project example that logging uses to write log entries to the destination. |
29+
30+
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

‎examples/logbucket/project/main.tf‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
/**
2+
* Copyright 2022 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
resource "random_string" "suffix" {
18+
length = 4
19+
upper = false
20+
special = false
21+
}
22+
23+
module "log_export" {
24+
source = "../../../"
25+
destination_uri = module.destination.destination_uri
26+
filter = "resource.type = gce_instance"
27+
log_sink_name = "logbucket_other_project"
28+
parent_resource_id = var.parent_resource_project
29+
parent_resource_type = "project"
30+
unique_writer_identity = true
31+
}
32+
33+
module "destination" {
34+
source = "../../..//modules/logbucket"
35+
project_id = var.project_destination_logbkt_id
36+
name = "logbucket_from_other_project_${random_string.suffix.result}"
37+
location = "global"
38+
log_sink_writer_identity = module.log_export.writer_identity
39+
}
40+
41+
#-------------------------------------#
42+
# Log Bucket and Sink in same project #
43+
#-------------------------------------#
44+
module "log_export_same_proj" {
45+
source = "../../../"
46+
destination_uri = module.dest_same_proj.destination_uri
47+
filter = "resource.type = gce_instance"
48+
log_sink_name = "logbucket_same_project"
49+
parent_resource_id = var.project_destination_logbkt_id
50+
parent_resource_type = "project"
51+
unique_writer_identity = true
52+
}
53+
54+
module "dest_same_proj" {
55+
source = "../../..//modules/logbucket"
56+
project_id = var.project_destination_logbkt_id
57+
name = "logbucket_from_same_project_${random_string.suffix.result}"
58+
location = "global"
59+
log_sink_writer_identity = module.log_export_same_proj.writer_identity
60+
grant_write_permission_on_bkt = false
61+
}

‎examples/logbucket/project/outputs.tf‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
/**
2+
* Copyright 2022 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
output "log_bucket_project" {
18+
description = "The project where the log bucket is created."
19+
value = module.destination.project
20+
}
21+
22+
output "log_bucket_name" {
23+
description = "The name for the log bucket."
24+
value = module.destination.resource_name
25+
}
26+
27+
output "log_sink_project_id" {
28+
description = "The project id where the log sink is created."
29+
value = module.log_export.parent_resource_id
30+
}
31+
32+
output "log_sink_destination_uri" {
33+
description = "A fully qualified URI for the log sink."
34+
value = module.destination.destination_uri
35+
}
36+
37+
output "log_sink_resource_name" {
38+
description = "The resource name of the log sink that was created."
39+
value = module.log_export.log_sink_resource_name
40+
}
41+
42+
output "log_sink_writer_identity" {
43+
description = "The service account that logging uses to write log entries to the destination."
44+
value = module.log_export.writer_identity
45+
}
46+
47+
48+
#-------------------------------------#
49+
# Log Bucket and Sink in same project #
50+
#-------------------------------------#
51+
output "log_bkt_same_proj" {
52+
description = "The project where the log bucket is created for sink and logbucket in same project example."
53+
value = module.dest_same_proj.project
54+
}
55+
56+
output "log_bkt_name_same_proj" {
57+
description = "The name for the log bucket for sink and logbucket in same project example."
58+
value = module.dest_same_proj.resource_name
59+
}
60+
61+
output "log_sink_id_same_proj" {
62+
description = "The project id where the log sink is created for sink and logbucket in same project example."
63+
value = module.log_export_same_proj.parent_resource_id
64+
}
65+
66+
output "log_sink_dest_uri_same_proj" {
67+
description = "A fully qualified URI for the log sink for sink and logbucket in same project example."
68+
value = module.dest_same_proj.destination_uri
69+
}
70+
71+
output "log_sink_resource_name_same_proj" {
72+
description = "The resource name of the log sink that was created in same project example."
73+
value = module.log_export_same_proj.log_sink_resource_name
74+
}
75+
76+
output "log_sink_writer_identity_same_proj" {
77+
description = "The service account in same project example that logging uses to write log entries to the destination."
78+
value = module.log_export_same_proj.writer_identity
79+
}

‎examples/logbucket/project/variables.tf‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
/**
2+
* Copyright 2022 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
variable "project_destination_logbkt_id" {
18+
description = "The ID of the project in which log bucket destination will be created."
19+
type = string
20+
}
21+
22+
variable "parent_resource_project" {
23+
description = "The ID of the project in which the log export will be created."
24+
type = string
25+
}

‎examples/logbucket/project/versions.tf‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
/**
2+
* Copyright 2022 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
18+
terraform {
19+
required_version = ">= 0.13"
20+
required_providers {
21+
google = {
22+
source = "hashicorp/google"
23+
version = "~> 4.0"
24+
}
25+
random = {
26+
source = "hashicorp/random"
27+
}
28+
}
29+
}

‎modules/logbucket/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ module "destination" {
3737

3838
| Name | Description | Type | Default | Required |
3939
|------|-------------|------|---------|:--------:|
40+
| grant\_write\_permission\_on\_bkt | (Optional) Indicates whether the module is responsible for granting write permission on the logbucket. This permission will be given by default, but if the user wants, this module can skip this step. This is the case when the sink route logs to a log bucket in the same Cloud project, no new service account will be created and this module will need to bypass granting permissions. | `bool` | `true` | no |
4041
| location | The location of the log bucket. | `string` | `"global"` | no |
4142
| log\_sink\_writer\_identity | The service account that logging uses to write log entries to the destination. (This is available as an output coming from the root module). | `string` | n/a | yes |
4243
| name | The name of the log bucket to be created and used for log entries matching the filter. | `string` | n/a | yes |

‎modules/logbucket/main.tf‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,8 @@ resource "google_logging_project_bucket_config" "bucket" {
4343
# Service account IAM membership #
4444
#--------------------------------#
4545
resource "google_project_iam_member" "logbucket_sink_member" {
46+
count = var.grant_write_permission_on_bkt ? 1 : 0
47+
4648
project = google_logging_project_bucket_config.bucket.project
4749
role = "roles/logging.bucketWriter"
4850
member = var.log_sink_writer_identity

‎modules/logbucket/variables.tf‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,3 +40,9 @@ variable "retention_days" {
4040
type = number
4141
default = 30
4242
}
43+
44+
variable "grant_write_permission_on_bkt" {
45+
description = "(Optional) Indicates whether the module is responsible for granting write permission on the logbucket. This permission will be given by default, but if the user wants, this module can skip this step. This is the case when the sink route logs to a log bucket in the same Cloud project, no new service account will be created and this module will need to bypass granting permissions."
46+
type = bool
47+
default = true
48+
}

‎test/integration/logbucket-project/logbucket_project_test.go‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
// Copyright 2022 Google LLC
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package logbucket_project
16+
17+
import (
18+
"fmt"
19+
"testing"
20+
21+
"github.com/GoogleCloudPlatform/cloud-foundation-toolkit/infra/blueprint-test/pkg/gcloud"
22+
"github.com/GoogleCloudPlatform/cloud-foundation-toolkit/infra/blueprint-test/pkg/tft"
23+
"github.com/GoogleCloudPlatform/cloud-foundation-toolkit/infra/blueprint-test/pkg/utils"
24+
"github.com/stretchr/testify/assert"
25+
)
26+
27+
const (
28+
defaultRetentionDays int64 = 30
29+
)
30+
31+
func TestLogBucketProjectModule(t *testing.T) {
32+
33+
bpt := tft.NewTFBlueprintTest(t,
34+
tft.WithTFDir("../../../examples/logbucket/project"),
35+
)
36+
bpt.DefineVerify(func(assert *assert.Assertions) {
37+
bpt.DefaultVerify(assert)
38+
39+
for _, tc := range []struct {
40+
projId string
41+
bktName string
42+
sinkDest string
43+
sinkProjId string
44+
sinkName string
45+
writerIdentity string
46+
}{
47+
{
48+
projId: bpt.GetStringOutput("log_bucket_project"),
49+
bktName: bpt.GetStringOutput("log_bucket_name"),
50+
sinkDest: bpt.GetStringOutput("log_sink_destination_uri"),
51+
sinkProjId: bpt.GetStringOutput("log_sink_project_id"),
52+
sinkName: bpt.GetStringOutput("log_sink_resource_name"),
53+
writerIdentity: bpt.GetStringOutput("log_sink_writer_identity"),
54+
},
55+
{
56+
projId: bpt.GetStringOutput("log_bkt_same_proj"),
57+
bktName: bpt.GetStringOutput("log_bkt_name_same_proj"),
58+
sinkDest: bpt.GetStringOutput("log_sink_dest_uri_same_proj"),
59+
sinkProjId: bpt.GetStringOutput("log_sink_id_same_proj"),
60+
sinkName: bpt.GetStringOutput("log_sink_resource_name_same_proj"),
61+
// writerIdentity: As sink and bucket are in same project no service account is needed and writerIdentity is empty
62+
},
63+
} {
64+
//************************
65+
// Assert bucket details *
66+
//************************
67+
bktFullName := fmt.Sprintf("projects/%s/locations/%s/buckets/%s", tc.projId, "global", tc.bktName)
68+
logBucketDetails := gcloud.Runf(t, fmt.Sprintf("logging buckets describe %s --location=%s --project=%s", tc.bktName, "global", tc.projId))
69+
70+
// assert log bucket name, retention days & location
71+
assert.Equal(bktFullName, logBucketDetails.Get("name").String(), "log bucket name should match")
72+
assert.Equal(defaultRetentionDays, logBucketDetails.Get("retentionDays").Int(), "retention days should match")
73+
74+
logSinkDetails := gcloud.Runf(t, fmt.Sprintf("logging sinks describe %s --project=%s", tc.sinkName, tc.sinkProjId))
75+
76+
// assert log sink name, destination & filter
77+
assert.Equal(tc.sinkDest, logSinkDetails.Get("destination").String(), "log sink destination should match")
78+
assert.Equal("resource.type = gce_instance", logSinkDetails.Get("filter").String(), "log sink filter should match")
79+
assert.Equal(tc.writerIdentity, logSinkDetails.Get("writerIdentity").String(), "log sink writerIdentity should match")
80+
}
81+
82+
//*****************************
83+
// Assert SAs and Permissions *
84+
//*****************************
85+
bktDestProjId := bpt.GetStringOutput("log_bkt_same_proj")
86+
sinkWriterIdentity := bpt.GetStringOutput("log_sink_writer_identity")
87+
88+
projPermissionsDetails := gcloud.Runf(t, fmt.Sprintf("projects get-iam-policy %s", bktDestProjId))
89+
listMembers := utils.GetResultStrSlice(projPermissionsDetails.Get("bindings.#(role==\"roles/logging.bucketWriter\").members").Array())
90+
91+
// assert sink writer identity service account permission
92+
assert.Contains(listMembers, sinkWriterIdentity, "log sink writer identity permission should match")
93+
assert.Len(listMembers, 1, "only one writer identity should have logbucket write permission")
94+
})
95+
bpt.Test()
96+
}

0 commit comments

Comments
 (0)