document setting nofile limit on different systems

Author: endorama

solutions/observability/apm/apm-server/binary.md

@@ -61,6 +61,8 @@ curl -L -O https://artifacts.elastic.co/downloads/apm-server/apm-server-{{versio
 tar xzvf apm-server-{{version.stack}}-linux-x86_64.tar.gz
 ```
 
+See [modifying the `nofile` ulimit](#modifying-nofile-ulimit).
+
 $$$apm-mac$$$
 **Mac:**
 
@@ -926,4 +928,56 @@ It’s possible to embed your APM Server configuration in a custom image. Here i
 ```dockerfile
 FROM docker.elastic.co/apm/apm-server:9.0.0
 COPY --chmod=0644 --chown=1000:1000 apm-server.yml /usr/share/apm-server/apm-server.yml
-```
+```
+
+#### Modifying `nofile` ulimit [ulimit-on-docker]
+
+Limits can be set from the command line using `--ulimit=soft:hard`, see [Set ulimits in container (--ulimit)](https://docs.docker.com/reference/cli/docker/container/run/#ulimit).
+
+```sh
+docker run -d \
+  -p 8200:8200 \
+  --name=apm-server \
+  --user=apm-server \
+  --volume="$(pwd)/apm-server.docker.yml:/usr/share/apm-server/apm-server.yml:ro" \
+  docker.elastic.co/apm/apm-server:9.0.0 \
+  --strict.perms=false -e \
+  --ulimit=524287:524287 \
+  -E output.elasticsearch.hosts=["elasticsearch:9200"] <1> <2>
+```
+
+1. Substitute your {{es}} hosts and ports.
+2. If you are using {{ech}}, replace the `-E output.elasticsearch.hosts` line with the Cloud ID and elastic password using the syntax shown earlier.
+
+
+## Modify the `nofile` ulimit [modify-nofile-ulimit]
+
+When run as a standalone binary APM Server will inherit the `nofile` limit from the user running the process. On most system this is configured to `1024`. This limit is too low for higher throughput scenarios or when using Tail Based Sampling.
+
+To chose the new limit, consider these guidelines:
+- there is no system performance impact of using a limit of ``;
+- a limit of `1024` would suffice for low throughput use cases;
+- the major contributor to open files will be the number of incoming connections;
+- Tail Based Sampling is file based, when enabling it the number of open files will be higher in proportion to the throughput and sampling policies.
+
+To configure the limit for your user, you need to know the username you will run APM Server process with.
+
+```sh
+whoami
+```
+
+Edit `/etc/security/limits.conf` with root privileges:
+
+```sh
+sudo nano /etc/security/limits.conf
+```
+
+Add the following lines to set soft and hard limits for your user:
+
+```text
+apm-server soft nofile 524287 <1>
+apm-server hard nofile 524287 <1>
+```
+
+1. Replace `apm-server` with the username you will run APM Server process with.
+

solutions/observability/apm/apm-server/systemd.md

@@ -88,6 +88,16 @@ systemctl restart apm-server
 It is recommended that you use a configuration management tool to include drop-in unit files. If you need to add a drop-in manually, use `systemctl edit apm-server.service`.
 ::::
 
+#### Configuring the NOFILE limit [configuring-nofile-limit]
+
+::::{note}
+There should be no need to manually configure this limit when running APM Server.
+::::
+
+In systemd the `LimitNOFILE` defaults are set to `1024` (soft) and `524288` (hard) and most Linux systems with systemd will not change these values or reduce them drastically. Golang since 1.19 (see [golang/go#46279](https://github.com/golang/go/issues/46279)) automatically bump the process limit up to the available hard limit. This means that by default APM Server runs with the limit set to the hard limit value by the Operating System is being run on, generally `524287` on a recent system. There should be no reason to change this limit, as back-pressure from too many open files will happen from memory usage.
+
+For guidelines on the value to set this value to see [Modifying the `nofile` ulimit](/solutions/observability/apm/apm-server/binary.md#modify-nofile-ulimit).
+
 #### Configuration file ownership [apm-config-file-ownership]
 
 On systems with POSIX file permissions, the APM Server configuration file is subject to ownership and file permission checks. These checks prevent unauthorized users from providing or modifying configurations that are run by APM Server.

solutions/observability/apm/apm-server/tail-based-sampling.md

@@ -20,6 +20,14 @@ Most options on this page are supported by all APM Server deployment methods whe
 Enhanced privileges are required to use tail-based sampling. For more information, refer to [Create a tail-based sampling role](/solutions/observability/apm/create-assign-feature-roles-to-apm-server-users.md#apm-privileges-tail-based-sampling).
 ::::
 
+::::{note}
+If you are manually configuring systemd `LimitNOFILE` or `LimitNOFILESoft` when using Tail Based Sampling and it affects the APM Server process, this may result in a `too many open files` error. Please see [configuring the NOFILE limit](/solutions/observability/apm/systemd.md#configuring-nofile-limit).
+::::
+
+::::{note}
+If you are running the binary standalone (not using the provided dev or rpm packages or the docker images) you need to adjust the `nofile` limit based on your throughput requirements. See [modifying the `nofile` ulimit](/solutions/observability/apm/apm-server/binary.md#modify-nofile-ulimit).
+::::
+
 Tail-based sampling configuration options.
 
 :::::::{tab-set}