getting "googleapi: got HTTP response code 404 with body: Not Found" error

[deepakdimri-ce]

i have CloseBucket Cloud Function deployed as per the instructions from the repo and its getting triggered from SCC notification as well but when its comes to action its throwing "googleapi: got HTTP response code 404 with body: Not Found" error. what could be the reason for this error?

[onetwopunch]

Could you please paste the error message showing which API call responded with a 404 and what resource was not found?

[deepakdimri-ce]

Below is series of events from Cloud Logging. This is all what i see in the logs every-time i create a test bucket with public access enabled.

  "textPayload": "Function error: googleapi: got HTTP response code 404 with body: Not Found\n",
  "insertId": "000000-5a80908c-51bb-4230-bdff-5e0b1076c5be",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "project_id": "contextawaredemo",
      "function_name": "CloseBucket"
    }
  },
  "timestamp": "2021-04-11T03:39:01.165Z",
  "severity": "ERROR",
  "labels": {
    "execution_id": "s7x8amfejjsc"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/d86d0e6ba3788f216df5231577f12864",
  "receiveTimestamp": "2021-04-11T03:39:11.444262292Z"
}
{
  "textPayload": "Function execution took 21 ms, finished with status: 'error'",
  "insertId": "000000-48665883-aaca-4132-9f6c-05a133854b86",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "project_id": "contextawaredemo",
      "function_name": "CloseBucket",
      "region": "us-central1"
    }
  },
  "timestamp": "2021-04-11T03:39:01.165885367Z",
  "severity": "DEBUG",
  "labels": {
    "execution_id": "s7x8amfejjsc"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/d86d0e6ba3788f216df5231577f12864",
  "receiveTimestamp": "2021-04-11T03:39:11.444262292Z"
}
https://console.cloud.google.com/logs/query;cursorTimestamp=2021-04-11T03:39:01.165Z;query=resource.type%20%3D%20%22cloud_function%22%0Aresource.labels.function_name%20%3D%20%22CloseBucket%22%0Aresource.labels.region%20%3D%20%22us-central1%22%0A%20severity%3E%3DDEFAULT%0Atimestamp%3D%222021-04-11T03:39:01.165885367Z%22%0AinsertId%3D%22000000-48665883-aaca-4132-9f6c-05a133854b86%22;timeRange=2021-04-11T03:27:11.682Z%2F2021-04-11T05:27:11.682214Z?project=contextawaredemo```

[daniel-cit]

@onetwopunch @deepakdimri-ce Just a detail that may help the investigation.

The minimal deployment for the security response automation is the Router cloud function + any other remediation cloud function.

Only the Router function knows how to read SCC findings.

The router function is the one that reads the finding and generates a struct with only the project name + the bucket name that is used by the CloseBucket function.

The SCC notification must send messages to the Router Cloud Function.

The error may be related to trying to get the bucket policy from an bucket name that is the empty string "" because the ClouseBucket cloud function was not able to get the bucket name from the finding, because only the Router knows how to read the finding.

[deepakdimri-ce]

thanks @daniel-cit - i have got Router cloud function deployed, with scc notifications going to Router function. but now i see Router function reporting "finding already remediated" message and not doing any thing. i am testing it with having a test GCS bucket wide open for public access. there are no logs showing up in CloseBucket function. any idea? i am still trying to investigate

  "textPayload": "Function execution took 1386 ms, finished with status: 'ok'",
  "insertId": "000000-1ae451f4-8cfa-4130-a84d-8bf01809a83c",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "project_id": "contextawaredemo",
      "function_name": "Router"
    }
  },
  "timestamp": "2021-04-13T04:41:21.119026453Z",
  "severity": "DEBUG",
  "labels": {
    "execution_id": "al9kpgx1qubq"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/7a629f3a9e0db83ad0700c3febdb85cf",
  "receiveTimestamp": "2021-04-13T04:41:30.435945600Z"
}
{
  "textPayload": "Function execution started",
  "insertId": "000000-25733616-6443-4dfc-972d-67a02dd830f8",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "function_name": "Router",
      "project_id": "contextawaredemo"
    }
  },
  "timestamp": "2021-04-13T04:41:22.584244984Z",
  "severity": "DEBUG",
  "labels": {
    "execution_id": "al9kivpbarui"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/08672e5aff134e0c97a40773928f61c8",
  "receiveTimestamp": "2021-04-13T04:41:30.435945600Z"
}
{
  "textPayload": "2021/04/13 04:41:22 /workspace/serverless_function_source_code/cloudfunctions/router/router.go:326: finding already remediated",
  "insertId": "000000-a39d5a68-b346-41ed-bdde-3efde1e8a8f4",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "project_id": "contextawaredemo",
      "region": "us-central1",
      "function_name": "Router"
    }
  },
  "timestamp": "2021-04-13T04:41:22.589Z",
  "labels": {
    "execution_id": "al9kivpbarui"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/08672e5aff134e0c97a40773928f61c8",
  "receiveTimestamp": "2021-04-13T04:41:30.435945600Z"
}```

[deepakdimri-ce]

this is my sra.yaml file looks like replaced with dummy org and folder ids:

kind: Remediation
metadata:
  name: router
spec:
  parameters:
    sha:
      public_bucket_acl:
        - action: close_bucket
          target:
            - organizations/999999999/folders/888888888/projects/contextawaredemo
          excludes:
            - organizations/999999999/folders/88888888/projects/forresterdemo
          properties:
            dry_run: false```

[daniel-cit]

Hi @deepakdimri-ce , The "finding already remediated" is a known issue #197

You can try to force the execution by going to the Console and deleting the security mark "sra-remediated-event-time" in the original Security Command Center finding.

Regarding the yaml file, you should use this file as an initial version https://github.com/GoogleCloudPlatform/security-response-automation/wiki/Sample-configuration.

A trick point is that It is an yaml file and preserving the indentation is importante.

[deepakdimri-ce]

thanks @daniel-cit for helping me with this.

I have tried deleting the mark against the finding in SCC but then it just comes back again on the console.
I also tried with the sample yaml file but no luck either. It ends up with same error textPayload: "2021/04/14 07:50:47 /workspace/serverless_function_source_code/cloudfunctions/router/router.go:326: finding already remediated"

is there any other pointers you could offer? i am kinda stuck with this.. and i need it working for a demo to my customers.

[daniel-cit]

Hi @deepakdimri-ce until #197 is fixed what you could do as a temporary workaround would be to comment the code that is marking the finding as remediated, something like this:

Go to the function that mark the finding as remediated

security-response-automation/cloudfunctions/router/router.go

Line 176 in d115061

func markAsRemediated(ctx context.Context, name, eventTime string, services *Services) error {

And comment its body from:

func markAsRemediated(ctx context.Context, name, eventTime string, services *Services) error {
	m := map[string]string{"sra-remediated-event-time": eventTime}
	if _, err := services.SecurityCommandCenter.AddSecurityMarks(ctx, name, m); err != nil {
		return err
	}
	return nil
}

to:

func markAsRemediated(ctx context.Context, name, eventTime string, services *Services) error {
	// m := map[string]string{"sra-remediated-event-time": eventTime}
	// if _, err := services.SecurityCommandCenter.AddSecurityMarks(ctx, name, m); err != nil {
	// 	return err
	// }
	return nil
}

This would prevent then security mark to return after you delete it in the console
BUT
this will make the finding be reprocessed by any change on it not related to the original cause if the finding satisfies you notification filter, like adding a security mark to the finding.

[deepakdimri-ce]

hi @daniel-cit - thanks again. i tried commenting the code as you suggested and now i see its not marking the findings. But it still does not do anything. router logs ends up with below messages and there is nothing interesting showing up in closebucket function logs.

{
  "textPayload": "2021/04/14 17:19:36 /workspace/serverless_function_source_code/cloudfunctions/router/router.go:329: got rule \"public_bucket_acl\" with 1 automations",
  "insertId": "000000-6ef59f11-7aeb-4df1-8bbc-e6077c712d32",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "function_name": "Router",
      "project_id": "contextawaredemo"
    }
  },
  "timestamp": "2021-04-14T17:19:36.849Z",
  "labels": {
    "execution_id": "7s915gvgh7ax"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/4c0ac9650bb8c2575022f9f383c8ed88",
  "receiveTimestamp": "2021-04-14T17:19:47.339841511Z"
}

{
  "textPayload": "2021/04/14 17:19:37 /workspace/serverless_function_source_code/clients/logger.go:60: failed to publish: \"failed to check if project \\\"\\\" is within the target or is excluded: failed to get project ancestry path: googleapi: got HTTP response code 404 with body: <!DOCTYPE html>\\n<html lang=en>\\n  <meta charset=utf-8>\\n  <meta name=viewport content=\\\"initial-scale=1, minimum-scale=1, width=device-width\\\">\\n  <title>Error 404 (Not Found)!!1</title>\\n  <style>\\n    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\\n  </style>\\n  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\\n  <p><b>404.</b> <ins>That’s an error.</ins>\\n  <p>The requested URL <code>/v1/projects/:getAncestry?alt=json&amp;prettyPrint=false</code> was not found on this server.  <ins>That’s all we know.</ins>\\n\"",
  "insertId": "000000-424459ec-08f9-4eca-99c3-3cdd92b89b11",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "function_name": "Router",
      "project_id": "contextawaredemo"
    }
  },
  "timestamp": "2021-04-14T17:19:37.153Z",
  "labels": {
    "execution_id": "7s915gvgh7ax"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/4c0ac9650bb8c2575022f9f383c8ed88",
  "receiveTimestamp": "2021-04-14T17:19:47.339841511Z"
}```

[daniel-cit]

Searching form part of the error message in your log failed to check if project we find this code:

func publish(ctx context.Context, services *Services, action, topic, projectID string, target, exclude []string, values interface{}) error {
	ok, err := services.Resource.CheckMatches(ctx, projectID, target, exclude)
	if err != nil {
		return errors.Wrapf(err, "failed to check if project %q is within the target or is excluded", projectID)
	}

The projectID subistituition in the message is \\\"\\\" which indicates that the value for the projectID is the empty string.

Since the code was able to hit the publish method it means it was able to find the correct automation and the issue looks like to be related to the reading the project from the finding.

Looking at real findings in the Security Command Center we can see
that there used to be a field in the finding for the ProjectId:
SourceProperties.ProjectId = "PROJECT_ID" but current findings have removed the field.
the field existed at least up to january 2021.

Now the only place to find the project id is in the ResourcePath property

SourceProperties.ResourcePath = [ "projects/PROJECT_ID/", "folders/0000000000000/", "organizations/0000000000000/" ]

It looks like to be a general change over all the SHA findings.

Because of this the SHA findings remediations will not work until the parsing of the findings is fixed.

[deepakdimri-ce]

Yeah i see ResourcePath has PROJECT_ID but then we should get this fixed in the code to reflect what is being sent by the SCC. it's weird that we have this code base which does not auto remediate the findings as expected from it?

[tomscript]

thanks @deepakdimri-ce for reporting this and @daniel-cit for identifying the cause. really appreciate it

• yep the SHA finding was changed a while back
• @deepakdimri-ce will send me a sample of the new finding, i'll also dig a bit and send a poc PR what i think will work.

[tomscript]

@deepakdimri-ce can u give #206 a test? it should work, but i have not actually tested it myself other than just locally

[deepakdimri-ce]

Thanks @tomscript for your help - i will test it later today and will let you know. In the meanwhile i have got this from SHA engineering team when asked for example finding event:
Please use resource.project_display_name instead. SCC Platform adds the project information based on the resource name provided by the finding provider and is expected to be there even for non SHA findings. Here's an example: https://bit.googleplex.com/#/raghuagarwal/5790160849469440

Example:

{
  "notificationConfigName": "organizations/123456789011/notificationConfigs/test-notifications-ga-2",
  "finding": {
    "name": "organizations/123456789011/sources/3873364276576983545/findings/48f10d56d956051a1ebfa6788a24e375",
    "parent": "organizations/123456789011/sources/3873364276576983545",
    "resourceName": "//compute.googleapis.com/projects/rd-project-725348--936/regions/asia-southeast2/subnetworks/7659896186824968098",
    "state": "ACTIVE",
    "category": "PRIVATE_GOOGLE_ACCESS_DISABLED",
    "externalUri": "https://console.cloud.google.com/networking/subnetworks/details/asia-southeast2/default?project=rd-project-725348--936",
    "sourceProperties": {
      "Recommendation": "Go to https://console.cloud.google.com/networking/subnetworks/details/asia-southeast2/default?project=rd-project-725348--936, click \"Edit\", and enable Private Google Access. Then, remove public (external) IPs from any VM instances whose only external traffic is to Google APIs by following the steps at: https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#unassign_ip",
      "ReactivationCount": 0,
      "ExceptionInstructions": "Add the security mark \"allow_private_google_access_disabled\" to the asset with a value of \"true\" to prevent this finding from being activated again.",
      "Explanation": "Private Google Access enables VM instances with only internal (private) IP addresses to reach the public IP addresses of Google APIs and services. Learn more at: https://cloud.google.com/vpc/docs/configure-private-google-access",
      "ScannerName": "SUBNETWORK_SCANNER",
      "ResourcePath": [
        "projects/rd-project-725348--936/",
        "organizations/123456789011/"
      ],
      "compliance_standards": {
        "cis": [
          {
            "ids": [
              "3.8"
            ]
          }
        ]
      }
    },
    "securityMarks": {
      "name": "organizations/123456789011/sources/3873364276576983545/findings/48f10d56d956051a1ebfa6788a24e375/securityMarks"
    },
    "eventTime": "2021-04-15T14:18:18.029106Z",
    "createTime": "2020-10-22T09:39:08.327Z",
    "severity": "LOW",
    "canonicalName": "projects/555555555/sources/3873364276576983545/findings/48f10d56d956051a1ebfa6788a24e375"
  },
  "resource": {
    "name": "//compute.googleapis.com/projects/rd-project-725348--936/regions/asia-southeast2/subnetworks/7659896186824968098",
    "project": "//cloudresourcemanager.googleapis.com/projects/555555555",
    "projectDisplayName": "rd-project-725348--936",
    "parent": "//cloudresourcemanager.googleapis.com/projects/555555555",
    "parentDisplayName": "rd-project-725348--936"
  }
}