feat: improved subdomain-url handling (#211)

* fix: subdomain-url handling

- removed implicit tests based on SubdomainLocalhostGatewayURL
- subdomain tests are now run only against origins defined via
  `--subdomain-url`
- DNSLink gateway tests are no longer tied to origin passed via
  `--subdomain-url`

* fix(wip): avoid mangling Host header handling

- splitting into subdomain and dnslink gateway variants
- towards avoiding running everything three times
- keep custom Host header when defined

* refactor: remove subdomains from dnslink fixtures

these concepts should not mix, and dnslinks should be testable
independent of --subdomain-url

i also added more useful export format for them, compatible with
IPNS_NS_MAP used by Boxo/Kubo/Rainbow

* refactor: handling Host header vs proxying

towards fixing underlying issue

* feat: make start-kubo-docker

all-in-one command for running local kubo with all fixtures in

* fix: correctly pass subdomain-url to go test

* docs: skip and run examples

* refactor: UnwrapSubdomainTests → proxy-gateway

This replaces magic helpers.UnwrapSubdomainTests with explicit
spec preset named proxy-gateway, which can be enabled/disabled
the same way as other presets.

While it was nice to run every test in both proxy modes,
it was extremely noisy and hard to debug once things went wrong.

By having explicit tests in *_proxy_test.go it is easy to skip them
where proxy functionality is not relevant.

Other:

- Send meaningful User-Agent
- Sent detached path in HTTP Connect proxy tunnel tests (instead of full
  URL – in tunnel mode we pretend we talk to remote server on the other
  end of tunnel, and not send full URL like it is done in plain proxy
  mode)

* docs: fix/improve docker examples

* refactor: require explicit gateway URLs

This is important UX change. We no longer ship with default URLs.

User has to provide explicit one, or the test suite will refuse to run.
This ensures misconfigurations and testing different gateway endpoint
than desired do not happen.

Explicit is better than implicit.

Author: lidel

.github/workflows/test-kubo-e2e.yml

@@ -46,11 +46,7 @@ jobs:
       - name: Provision Kubo Gateway
         run: |
           # Import car files
-          cars=$(find ./fixtures -name '*.car')
-          for car in $cars
-          do
-              ipfs dag import --pin-roots=false --stats "$car"
-          done
+          find ./fixtures -name '*.car' -exec ipfs dag import --pin-roots=false {} \;
 
           # Import ipns records
           records=$(find ./fixtures -name '*.ipns-record')
@@ -63,7 +59,7 @@ jobs:
         uses: ./gateway-conformance/.github/actions/test
         with:
           gateway-url: http://127.0.0.1:8080
-          subdomain-url: http://example.com
+          subdomain-url: http://example.com:8080
           json: output.json
           xml: output.xml
           html: output.html

.gitignore

@@ -8,6 +8,7 @@
 /gateway-conformance
 /dnslinks.json
 /*.ipns-record
+extracted-fixtures
 
 # Logs
 logs
@@ -142,3 +143,4 @@ dist
 
 # reports
 reports
+report.json

CHANGELOG.md

@@ -4,6 +4,36 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.6.0] - 2024-05-27 ### Changed
+- Gateway URL
+  - `--gateway-url` is no longer defaulting to predefined URL. User has to
+    provide it via CLI or `GATEWAY_URL` environment variable or the test suite
+    will refuse to start.
+  - This aims to ensure no confusion about which gateway endpoint is being
+    tested.
+  - Docs and examples use `--gateway-url http://127.0.0.1:8080` to ensure no
+    confusion with `localhost:8080` subdomain gateway feature in IPFS
+    implementations like Kubo.
+- Subdomain URL and UX related to subdomain tests
+  - The `--subdomain-url` is no longer set by default.
+  - User has to provide the origin of the subdomain gateway via CLI or
+    `SUBDOMAIN_GATEWAY_URL` to be used during subdomain tests. This aims to
+    ensure no confusion about which domain name is being tested.
+  - Simplified the way `--subdomain-url` works. We no longer run implicit tests
+    against `http://localhost` in addition to the URL passed via
+    `--subdomain-url`. To test more than one domain, run test multiple times.
+-  DNSLink test fixtures changed
+  - DNSLink fixtures no longer depend on `--subdomain-url` and use unrelated
+    `*.example.org` domains instead.
+  - `gateway-conformance extract-fixtures` creates `dnslinks.IPFS_NS_MAP` with
+    content that can be directly set as `IPNS_NS_MAP` environment variable
+    supported by various implementations, incl.
+    [Kubo](https://github.com/ipfs/kubo/blob/master/docs/environment-variables.md#ipfs_ns_map)
+    and
+    [Rainbow](https://github.com/ipfs/rainbow/blob/main/docs/environment-variables.md#ipfs_ns_map).
+- Docker
+  - The image can now be run under non-root user
+
 ## [0.5.2] - 2024-05-20
 ### Changed
 - Fixed: relaxed dag-cbor error check ([#205](https://github.com/ipfs/gateway-conformance/pull/205))

Dockerfile

@@ -1,6 +1,7 @@
 FROM golang:1.20-alpine
 WORKDIR /app
-ENV GATEWAY_CONFORMANCE_HOME=/app
+ENV GATEWAY_CONFORMANCE_HOME=/app \
+    GOCACHE=/go/cache
 
 COPY ./go.mod ./go.sum ./
 RUN go mod download
@@ -9,4 +10,7 @@ COPY . .
 ARG VERSION=dev
 RUN go build -ldflags="-X github.com/ipfs/gateway-conformance/tooling.Version=${VERSION}" -o ./gateway-conformance ./cmd/gateway-conformance
 
+# Relaxed perms for cache dir to allow running under regular user
+RUN mkdir -p $GOCACHE && chmod -R 777 $GOCACHE
+
 ENTRYPOINT ["/app/gateway-conformance"]

Makefile

@@ -1,15 +1,18 @@
 GIT_COMMIT := $(shell git rev-parse --short HEAD)
 DIRTY_SUFFIX := $(shell test -n "`git status --porcelain`" && echo "-dirty" || true)
 CLI_VERSION := dev-$(GIT_COMMIT)$(DIRTY_SUFFIX)
+KUBO_VERSION ?= latest
+KUBO_DOCKER_NAME ?= kubo-$(KUBO_VERSION)-gateway-conformance
 
 all: gateway-conformance
 
 clean: clean-docker
 	rm -f ./gateway-conformance
 	rm -f *.ipns-record
 	rm -f fixtures.car
-	rm -f dnslinks.json
-	rm -f ./reports/*
+	rm -f dnslinks.*
+	rm -f dnslink*.yml
+	rm -rf ./reports/*
 
 test-cargateway: provision-cargateway fixtures.car gateway-conformance
 	./gateway-conformance test --json reports/output.json --gateway-url http://127.0.0.1:8040 --specs -subdomain-gateway
@@ -22,13 +25,27 @@ test-kubo: provision-kubo gateway-conformance
 	./gateway-conformance test --json reports/output.json --gateway-url http://127.0.0.1:8080 --specs -subdomain-gateway
 
 provision-cargateway: ./fixtures.car
-	# cd go-libipfs/examples/car && go install
 	car -c ./fixtures.car &
 
 provision-kubo:
 	find ./fixtures -name '*.car' -exec ipfs dag import --stats --pin-roots=false {} \;
 	find ./fixtures -name '*.ipns-record' -exec sh -c 'ipfs routing put --allow-offline /ipns/$$(basename -s .ipns-record "{}" | cut -d'_' -f1) "{}"' \;
 
+#start-kubo-docker: stop-kubo-docker gateway-conformance
+#	./gateway-conformance extract-fixtures --dir=.temp/fixtures
+#	docker pull ipfs/kubo:$(KUBO_VERSION)
+#	docker run -d --rm --net=host --name $(KUBO_DOCKER_NAME) -v "$(shell realpath .temp/fixtures)":/fixtures -v kubo-config.example.sh:/container-init.d/001-config.sh ipfs/kubo:$(KUBO_VERSION) daemon --init --offline
+#	@until docker exec $(KUBO_DOCKER_NAME) ipfs --api=/ip4/127.0.0.1/tcp/5001 dag stat /ipfs/QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn >/dev/null 2>&1; do sleep 0.1; done
+#	docker exec $(KUBO_DOCKER_NAME) find /fixtures -name '*.car' -exec ipfs dag import --stats --pin-roots=false {} \;
+#	docker exec $(KUBO_DOCKER_NAME) find /fixtures -name '*.ipns-record' -exec sh -c 'ipfs routing put --allow-offline /ipns/$$(basename -s .ipns-record "{}" | cut -d'_' -f1) "{}"' \;
+#	TODO: provision Kubo config at Gateway.PublicGateways  to have subdomain gateway on example.com and also enable inlining on localhost
+#	      See: https://github.com/ipfs/kubo/blob/a07852a3f0294974b802923fb136885ad077384e/.github/workflows/gateway-conformance.yml#L22-L34
+#	      (this is not as trivial as it sounds because Kubo does not apply config inrealtime, and a restart is required.)
+
+stop-kubo-docker: clean
+	docker stop $(KUBO_DOCKER_NAME) || true
+	docker rm -f $(KUBO_DOCKER_NAME) || true
+
 # tools
 fixtures.car: gateway-conformance
 	./gateway-conformance extract-fixtures --merged=true --dir=.

README.md

@@ -24,12 +24,12 @@
   - [CLI](#cli)
   - [Docker](#docker)
   - [Github Action](#github-action)
-  - [Web Dashboard](#web-dashboard)
 - [Commands](#commands)
   - [Examples](#examples)
 - [Releases](#releases)
 - [Development](#development)
   - [Test DSL Syntax](#test-dsl-syntax)
+  - [Web Dashboard](#web-dashboard)
 - [License](#license)
 
 <!-- /TOC -->
@@ -67,25 +67,32 @@ Two high level [commands](/docs/commands.md) exist:
 ```console
 $ # Install the gateway-conformance binary
 $ go install github.com/ipfs/gateway-conformance/cmd/gateway-conformance@latest
-$ # run subdomain-gateway tests against endpoint at http://localhost:8080 output as JSON
-$ gateway-conformance test --gateway-url http://localhost:8080 --json report.json --specs +subdomain-gateway,-path-gateway -- -timeout 30m
+
+$ # skip path gateway tests, and run subdomain-gateway tests against endpoint at http://127.0.0.1:8080 and use *.ipfs.example.com subdomains, output as JSON
+$ gateway-conformance test --gateway-url http://127.0.0.1:8080 --subdomain-url http://example.com:8080 --json report.json --specs +subdomain-gateway,-path-gateway -- -timeout 5m
 ```
 
-If you are looking for TLDR, see [examples](/docs/examples.md).
+> [!TIP]
+> If want skip individual tests, or only run specific ones based on a regex, see [`/docs/examples`](/docs/examples.md).
 
 ### Docker
 
-Prebuilt image at `ghcr.io/ipfs/gateway-conformance` can be used for both `test` and `extract-fixtures` commands:
+The `gateway-conformace` requires golang runtime to be present to facilitate `go test`.
+If you want to run it on a box without having to instal golang runtime, prebuilt image at `ghcr.io/ipfs/gateway-conformance` is provided.
+
+It can be used for both `test` and `extract-fixtures` commands:
 
 ```console
-$ # extract fixtures to ./fixtures directory
-$ docker run -v "${PWD}:/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance:vA.B.C extract-fixtures --directory fixtures --merged false
+$ # extract fixtures to ./extracted-fixtures directory under the current user's permissions
+$ docker run -u "$(id -u):$(id -g)" -v "$(pwd):/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance:latest extract-fixtures --directory extracted-fixtures
 
-$ # run subdomain-gateway tests against endpoint at http://localhost:8080
-$ docker run --network host -v "${PWD}:/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance:vA.B.C test --gateway-url http://localhost:8080 --json report.json --specs +subdomain-gateway,-path-gateway -- -timeout 30m
+$ # skip path gateway tests, and run subdomain-gateway tests against endpoint at http://127.0.0.1:8080 and use *.ipfs.example.com subdomains, output as JSON
+$ docker run --net=host -u "$(id -u):$(id -g)" -v "$(pwd):/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance:latest test --gateway-url http://127.0.0.1:8080 --subdomain-url http://example.com:8080 --json report.json --specs +subdomain-gateway,-path-gateway -- -timeout 5m
 ```
 
-**NOTE:** replace `vA.B.C` with a [semantic version](https://github.com/ipfs/gateway-conformance/releases) version you want to test against
+> [!IMPORTANT]
+> - for stable CI/CD, replace `latest` with a [semantic version](https://github.com/ipfs/gateway-conformance/releases) version you want to test against
+> - `-u` ensures extracted fixtures and created report files can be read by your local user, make sure to adjust it to suit your use case
 
 ### Github Action
 
@@ -94,22 +101,9 @@ Common operations are possible via reusable GitHub actions:
 - [`ipfs/gateway-conformance/.github/actions/extract-fixtures`](https://github.com/ipfs/gateway-conformance/blob/main/.github/actions/extract-fixtures/action.yml)
 
 To learn how to integrate them in the CI of your project, see real world examples in:
-- [`kubo/../gateway-conformance.yml`](https://github.com/ipfs/kubo/blob/master/.github/workflows/gateway-conformance.yml) (fixtures imported into tested node)
-- [`boxo/../gateway-conformance.yml`](https://github.com/ipfs/boxo/blob/main/.github/workflows/gateway-conformance.yml) (fixtures imported into a sidecar kubo node that is peered with tested library)
-- [`rainbow/../gateway-conformance.yml`](https://github.com/ipfs/rainbow/blob/main/.github/workflows/gateway-conformance.yml) (fixtures imported into a kubo node that acts as a delegated block backend)
-
-### Web Dashboard
-
-Conformance test suite output can be plain text or JSON, which in turn can be
-represented as a web dashboard which aggregates results from many test runs and
-renders them on a static website.
-
-The Implementation Dashboard instance at
-[conformance.ipfs.tech](https://conformance.ipfs.tech/) is a view that
-showcases some of well known and complete implementations of IPFS Gateways
-in the ecosystem.
-
-Learn more at [`/docs/web-dashboard.md`](/docs/web-dashboard.md)
+- [`kubo/../gateway-conformance.yml`](https://github.com/ipfs/kubo/blob/master/.github/workflows/gateway-conformance.yml) (fixtures imported into tested kubo node that exposes HTTP gateway feature)
+- [`boxo/../gateway-conformance.yml`](https://github.com/ipfs/boxo/blob/main/.github/workflows/gateway-conformance.yml) (fixtures imported into a sidecar kubo node that is peered with small HTTP server used for testing `boxo/gateway`  library)
+- [`rainbow/../gateway-conformance.yml`](https://github.com/ipfs/rainbow/blob/main/.github/workflows/gateway-conformance.yml) (fixtures imported into a kubo node that acts as a remote block provider, than tested against different `boxo/gateway` backends)
 
 ## Commands
 
@@ -142,6 +136,19 @@ Interested in write a new test case?
 Test cases are written in Domain Specific Language (DLS) based on Golang. 
 More details at [`/docs/test-dsl-syntax.md`](/docs/test-dsl-syntax.md)
 
+### Web Dashboard
+
+Conformance test suite output can be plain text or JSON, which in turn can be
+represented as a web dashboard which aggregates results from many test runs and
+renders them on a static website.
+
+Experimental Implementation Dashboard instance at
+[conformance.ipfs.tech](https://conformance.ipfs.tech/) is a view that
+showcases some of well known and complete implementations of IPFS Gateways
+in the ecosystem.
+
+Learn more at [`/docs/web-dashboard.md`](/docs/web-dashboard.md)
+
 ## License
 
 This project is dual-licensed under Apache 2.0 and MIT terms: