[Bug] Provider produced inconsistent result after apply in elasticstack_fleet_integration_policy resource

[bplies-ATX]

Describe the bug
Cannot successfully use the elasticstack_fleet_integration_policy resource with a serious input containing anything substantial.

To Reproduce
Steps to reproduce the behavior:

1. TF configuration used '...'

data "elasticstack_fleet_integration" "custom_logs_stream" {
  name = "filestream"
}

resource "elasticstack_fleet_integration_policy" "sentinel_app_logs" {
  name                = "Sentinel application logs"
  namespace           = "sentinel"
  description         = ""
  agent_policy_id     = elasticstack_fleet_agent_policy.qa_environment_agent_policy.id
  integration_name    = data.elasticstack_fleet_integration.custom_logs_stream.name
  integration_version = "1.2.0"
  enabled             = true

  input {
    enabled  = true
    input_id = "filestream-filestream"
    streams_json = jsonencode({
      "filestream.generic" : {
        "enabled" : true,
        "vars" : {
          "paths" : ["/logs/sentinel*.log"],
          "data_stream.dataset" : "custom.app",
          "processors" : file("${path.module}/../../common/integration_processors/sentinel.yml"),
          "exclude_files" : ["\\.gz$"],
          # These were observed from Kibana UI request and provided here to avoid drift
          "include_files" : [],
          "tags" : [],
          "recursive_glob" : true,
          "clean_inactive" : -1,
          "harvester_limit" : 0,
          "fingerprint" : true,
          "fingerprint_offset" : 0,
          "fingerprint_length" : 1024,
          "file_identity_native" : false,
          "exclude_lines" : [],
          "include_lines" : []
        }
      }
    })
    vars_json           = "{}"
  }

  lifecycle { # Attempt to smooth over any provider problems for now
    ignore_changes = [
      input,
      vars_json,
    ]
  }
}

2. TF operations to execute to get the error '...' [e.g terraform plan,terraform apply, terraform destroy]

terraform plan

  # elasticstack_fleet_integration_policy.sentinel_app_logs is tainted, so must be replaced
-/+ resource "elasticstack_fleet_integration_policy" "sentinel_app_logs" {
      ~ id                  = "bda3f7dc-9e5c-4945-a0b2-5ca99ed7d56b" -> (known after apply)
        name                = "Sentinel application logs"
      ~ policy_id           = "bda3f7dc-9e5c-4945-a0b2-5ca99ed7d56b" -> (known after apply)
      ~ vars_json           = (sensitive value)
        # (6 unchanged attributes hidden)

      ~ input {
          ~ streams_json = (sensitive value)
          + vars_json    = (sensitive value)
            # (2 unchanged attributes hidden)
        }
    }

A previous failed apply did create but with the same error as below. Was totally fine having it tear down and re-create from scratch...

terraform apply
...
elasticstack_fleet_integration_policy.sentinel_app_logs: Destroying... [id=bda3f7dc-9e5c-4945-a0b2-5ca99ed7d56b]
elasticstack_fleet_integration_policy.sentinel_app_logs: Destruction complete after 4s
elasticstack_fleet_integration_policy.sentinel_app_logs: Creating...

Error: Provider produced inconsistent result after apply

When applying changes to
elasticstack_fleet_integration_policy.sentinel_app_logs, provider
"provider[\"registry.terraform.io/elastic/elasticstack\"]" produced an
unexpected new value: .input[0].streams_json: inconsistent values for
sensitive attribute.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
elasticstack_fleet_integration_policy.sentinel_app_logs, provider
"provider[\"registry.terraform.io/elastic/elasticstack\"]" produced an
unexpected new value: .input[0].vars_json: inconsistent values for sensitive
attribute.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Expected behavior
The provider should be able to create a elasticstack_fleet_integration_policy resource

Debug output
Skip

Screenshots
Skip

Versions (please complete the following information):

• OS: Linux
• Terraform Version 1.8.1
• Provider version v0.12.1 (very recent)
• Elasticsearch Version 8.17.1

Additional context

Possibly somewhat related to #1093 (policy_id) and #889 (sensitive values)

[tobio]

Related to #617

@bplies-ATX thanks for the report. We're working on some broader improvements to this resource which will include fixing the bugs around vars_json.

Currently, input vars an streams must be completely defined within the resource declaration. That is:

• All vars with a default value in the integration must be defined, even if that's just re-using the default
• All streams must be defined, even if they're disabled
• All stream vars with a default value must be defined similarly to input vars.

Comparing your resource to the default integration in Kibana, it looks like you're missing the parsers var. It should be something like:

"parsers": "#- ndjson:\\n#    target: \\"\\"\\n#    message_key: msg\\n#- multiline:\\n#    type: count\\n#    count_lines: 3\\n",

Though I expect parsers: "# blank" should work as well.