Document remote cluster setup with non-eck clusters

[strawgate]

With #8167 we added the ability to easily configure CCS between two eck clusters

What's less clear is if and how a user who wants to use an eck cluster as a remote cluster would configure it.

The documentation we provide leads the user to setting up certificate based CCS "Connect from an Elasticsearch cluster running outside the Kubernetes cluster" but it doesn't look like we tell the user they need to setup client certificates https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-remote-clusters.html

We should document the setup for connecting a cluster from outside of eck using API keys and rename the current section to be for certificate based ccs

[barkbay]

Although we still need to write proper documentation, here are the steps to connect to a hosted deployment in https://cloud.elastic.co/deployments from an ECK managed Elasticsearch cluster:

Warning

The instructions below assume that the ECK cluster is not configured to connect to others Elasticsearch clusters, within the same Kubernetes clusters, using spec.remoteClusters. It is a known limitation because there is currently no way to add trusted certificates to the remote cluster client when spec.remoteClusters is set.

In the hosted deployment

Create an API key

• Create a remote cluster API key in Kibana (https://<deployment>.kb.<region>.<cloud_provider>.elastic-cloud.com/app/management/security/api_keys/create)
• Copy the encoded key to a safe location. You will need it in the next steps.

Get the proxy URL and server name

• In your deployment security settings (https://cloud.elastic.co/deployments/<deployment id>/security), get the proxy address and the server name in the Remote cluster parameters section.
• Change the port from 9400 to 9443 in the proxy address.

In the ECK cluster

Store the API key encoded value in a Secret

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: hosted-remote-api-keys
stringData:
  cluster.remote.elastic-cloud.credentials: <encoded value>
type: Opaque
EOF

Configure the Elasticsearch resource

Enable the remote cluster SSL client and load the API key as a secure setting:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: cluster-in-eck
spec:
  version: 8.17.4
  secureSettings:
    - secretName: hosted-remote-api-keys
  nodeSets:
    - name: default
      config:
        node.store.allow_mmap: false
        xpack:
          security:
            remote_cluster_client:
              ssl:
                enabled: true ## repeat this for each nodeSet
      count: 3

Create the remote cluster using the Elasticsearch API

PUT /_cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "elastic-cloud": {
          "mode":"proxy",
          "proxy_address": "<proxy address, ending with 9443, from step above>",
          "server_name": "<server name from step above>"
        }
      }
    }
  }
}

Check that the ECK cluster is connected to your hosted deployment

GET /_remote/info

{
  "elastic-cloud": {
    "connected": true,
    "mode": "proxy",
    "proxy_address": "[...]elastic-cloud.com:9443",
    "server_name": "[....].elastic-cloud.com",
    "num_proxy_sockets_connected": 18,
    "max_proxy_socket_connections": 18,
    "initial_connect_timeout": "30s",
    "skip_unavailable": true,
    "cluster_credentials": "::es_redacted::"
  }
}

[eedugon]

@strawgate , thanks for raising this. For the API Key concern @barkbay has already replied. I'd like to share something and ask for more details about the other issue you mention with the current existing document.

it doesn't look like we tell the user they need to setup client certificates

What do you mean? As far as I understand, there's no need to setup any special client certificate, as both clusters (if they exist and have security enabled) should have transport certificates already created (self-managed clusters create SSL certificates automatically for transport and HTTP layers by default, as showed here).

The transport certificates of the nodes themselves are used by the nodes as client certificates when talking to the other cluster, that's why we only need to ensure the cluster trust each other's transport CA.

Maybe the fact that we assume both clusters have transport SSL already configured hasn't been properly described in the doc. Let us know your thoughts.

[strawgate]

@eedugon the docs now say:

The Elasticsearch security features need to be enabled on both clusters, on every node. Security is enabled by default. If it’s disabled, set xpack.security.enabled to true in elasticsearch.yml. Refer to General security settings.

I think the bigger issue was that users were trying to setup CCS with ECK by following the instructions in the doc, and because the doc didn't mention that it was setting up CCS using Client Certificate based authentication, users were then trying to setup API key authentication.

At the time I posted this, the docs did not say anything other than that the clusters need to trust each other's transport CAs (though you can trust transport CAs and have client authentication disabled in one of the clusters which will prevent CCS from working.)

I dont think this is that important now that the docs explicitly mention that enabling security on both clusters is required and that there are now API docs.

but it's been a couple months so i dont exactly remember the full details.

[eedugon]

thanks @strawgate , that makes all sense, We will focus on documenting the API key method as soon as we can, and ensure we differentiate both methods clearly in the docs.

[eedugon]

This PR should address everything discussed here: elastic/docs-content#4170

@strawgate , it would be great if you could take a look at the new docs and structure.