Skip to content

Commit 7942dc2

Browse files
kangastakangasta
authored
Add gitlab-ci/docker-hub-ratelimit example (#23)
1 parent 120fe30 commit 7942dc2

File tree

6 files changed

+192
-0
lines changed

6 files changed

+192
-0
lines changed

‎docs/examples/gitlab-ci/docker-hub-ratelimit/.gitlab-ci.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
default:
2+
image: docker:cli
3+
services:
4+
- name: "docker:dind"
5+
command: ["--tls=false", "--host=tcp://0.0.0.0:2375"]
6+
7+
variables:
8+
DOCKER_TLS_CERTDIR: ""
9+
DOCKER_HOST: tcp://docker:2375
10+
11+
stages:
12+
- test
13+
14+
test-caching:
15+
stage: test
16+
before_script:
17+
- apk add curl jq
18+
script:
19+
- docker pull ubuntu:24.04
20+
- ./scripts/check-docker-hub-ratelimit.sh 1st.txt
21+
- docker rmi ubuntu:24.04
22+
- docker pull ubuntu:24.04
23+
- ./scripts/check-docker-hub-ratelimit.sh 2nd.txt
24+
- diff 1st.txt 2nd.txt

‎docs/examples/gitlab-ci/docker-hub-ratelimit/README.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
---
2+
description: Tutorial on how to tests if GitLab CI accesses Docker Hub directly and how to configure pull-through cache to an Kubernetes cluster working as a GitLab runner.
3+
---
4+
5+
# Docker Hub ratelimit
6+
7+
The recommended way of running Docker commands in a GitLab CI Kubernets runner is to use a service to run Docker-in-Docker container (`docker:dind`). By default the `docker:dind` container pulls container images from Docker Hub without caching. This can cause problems as pulls from Docker Hub are rate-limited.
8+
9+
This example tests how `docker pull ubuntu:24.04` commands consume Docker Hub ratelimit and provides example on how to configure pull-through cache to an Kubernetes cluster working as a GitLab runner.
10+
11+
## How to check if `docker pull` commands are cached
12+
13+
Create a new GitLab project with `.gitlab-ci.yml` and `scripts/check-docker-hub-ratelimit.sh` files.
14+
15+
The `scripts/check-docker-hub-ratelimit.sh` script prints current Docker Hub ratelimit and also writes the result to a file if filename is given as first parameter. See [Docker Hub usage and rate limits](https://docs.docker.com/docker-hub/download-rate-limit/#how-can-i-check-my-current-rate) article in Docker documentation for more details.
16+
17+
```yaml title="scripts/check-docker-hub-ratelimit.sh"
18+
---8<--- "docs/examples/gitlab-ci/docker-hub-ratelimit/scripts/check-docker-hub-ratelimit.sh"
19+
```
20+
21+
The pipeline defined by `.gitlab-ci.yml` tries to pull the same Docker image twice and checks if the rate limit headers are different after first and second pull.
22+
23+
```yaml title=".gitlab-ci.yml"
24+
---8<--- "docs/examples/gitlab-ci/docker-hub-ratelimit/.gitlab-ci.yml"
25+
```
26+
27+
The pipeline should fail, if images are pulled directly from Docker Hub.
28+
29+
## How to setup pull-through cache to Kubernetes runner
30+
31+
Manifests for configuring pull-through cache and configmap are available in the repository that provides this website.
32+
33+
```sh
34+
git clone https://github.com/kangasta/cicd-examples.git
35+
cd docs/examples/gitlab-ci/docker-hub-ratelimit
36+
```
37+
38+
Configure pull-through cache and configmap for GitLab runner by running `kubectl apply -f manifests/`.
39+
40+
```sh
41+
kubectl apply -f manifests/
42+
```
43+
44+
These manifests assume that GitLab runner is using gitlab-runner namespace. Edit the namespace value in [docker-config.yaml](./manifests/docker-config.yaml) if this is not the case.
45+
46+
Modify the GitLab runner configuration so that the configmap defined in [docker-config.yaml](./manifests/docker-config.yaml) is mounted to all containers launched by the GitLab runner. Example of a full `values.yaml` help input file below.
47+
48+
```yaml
49+
gitlabUrl: # Your instance URL
50+
rbac: { create: true }
51+
runnerToken: # Your runner token
52+
runners:
53+
config: |
54+
[[runners]]
55+
executor = "kubernetes"
56+
[runners.kubernetes]
57+
image = "alpine:3.12"
58+
privileged = true
59+
[[runners.kubernetes.volumes.config_map]]
60+
name = "dockerd-config"
61+
mount_path = "/etc/docker/daemon.json"
62+
sub_path = "daemon.json"
63+
```

‎docs/examples/gitlab-ci/docker-hub-ratelimit/manifests/docker-config.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
name: gitlab-runner
5+
labels:
6+
name: gitlab-runner
7+
---
8+
apiVersion: v1
9+
kind: ConfigMap
10+
metadata:
11+
name: dockerd-config
12+
namespace: gitlab-runner
13+
data:
14+
daemon.json: |
15+
{
16+
"registry-mirrors": [
17+
"http://registry.pull-through-cache.svc.cluster.local"
18+
],
19+
"insecure-registries" : ["registry.pull-through-cache.svc.cluster.local"]
20+
}

‎docs/examples/gitlab-ci/docker-hub-ratelimit/manifests/registry-cache.yaml‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
name: pull-through-cache
5+
labels:
6+
name: pull-through-cache
7+
---
8+
apiVersion: v1
9+
kind: ConfigMap
10+
metadata:
11+
name: registry-config
12+
namespace: pull-through-cache
13+
data:
14+
config.yml: |
15+
version: 0.1
16+
http:
17+
addr: 0.0.0.0:80
18+
proxy:
19+
remoteurl: https://registry-1.docker.io
20+
storage:
21+
filesystem: {}
22+
---
23+
apiVersion: apps/v1
24+
kind: Deployment
25+
metadata:
26+
name: registry
27+
namespace: pull-through-cache
28+
labels:
29+
app: registry
30+
spec:
31+
replicas: 1
32+
selector:
33+
matchLabels:
34+
app: registry
35+
template:
36+
metadata:
37+
labels:
38+
app: registry
39+
spec:
40+
containers:
41+
- name: registry
42+
image: registry:2
43+
volumeMounts:
44+
- name: registry-config
45+
mountPath: "/etc/docker/registry/"
46+
readOnly: true
47+
volumes:
48+
- name: registry-config
49+
configMap:
50+
name: registry-config
51+
items:
52+
- key: "config.yml"
53+
path: "config.yml"
54+
---
55+
apiVersion: v1
56+
kind: Service
57+
metadata:
58+
name: registry
59+
namespace: pull-through-cache
60+
labels:
61+
app: registry
62+
spec:
63+
type: ClusterIP
64+
ports:
65+
- port: 80
66+
targetPort: 80
67+
protocol: TCP
68+
selector:
69+
app: registry

‎docs/examples/gitlab-ci/docker-hub-ratelimit/scripts/check-docker-hub-ratelimit.sh‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#!/bin/sh -e
2+
3+
fetch_ratelimit() {
4+
curl -I -H "Authorization: Bearer $1" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1 | grep -i ratelimit
5+
}
6+
7+
target=$1
8+
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
9+
10+
if [ -n "$target" ]; then
11+
fetch_ratelimit $token | tee $target;
12+
else
13+
fetch_ratelimit $token;
14+
fi

‎mkdocs.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ nav:
1515
- Home: README.md
1616
- Docker:
1717
- ./examples/docker/docker-on-wsl/README.md
18+
- GitLab CI:
19+
- ./examples/gitlab-ci/docker-hub-ratelimit/README.md
1820
- Jenkins:
1921
- ./examples/jenkins/jenkins-host-docker/README.md
2022
- ./examples/jenkins/ansible-kubernetes/README.md

0 commit comments

Comments
 (0)