Namespace removal is blocked by cozystack-api server

[WanzenBug]

Describe the bug
I created a three namespaces, creating a volume and STS in each. Then, I deleted all the resources, including the namespace. The namespaces are now stuck in Terminating state.

Environment

• Cozystack version v0.38.1
• Provider: on-prem (virtual)

To Reproduce
Steps to reproduce the behavior:

1. kubectl create ns stuck-ns
2. kubectl delete ns stuck-ns
3. kubectl get ns stuck-ns -oyaml will show the error

Expected behaviour
The namespace should be removed.

Actual behaviour
The namespace remains stuck in Terminating state. It seems that the cozystack-api is cause for the issue (see logs). On the status we can see:

  - lastTransitionTime: "2025-12-01T12:51:16Z"
    message: 'Failed to delete all resource types, 25 remaining: content is not a
      list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured, content is not a list: []unstructured.Unstructured,
      content is not a list: []unstructured.Unstructured'
    reason: ContentDeletionFailed
    status: "True"
    type: NamespaceDeletionContentFailure

Those 25 resource types would correspond to the 25 apps.cozystack.io/v1alpha1 api-resources.

Logs
The cozystack-api continously generates these logs messages, which seems relevant.

2025-12-01T12:53:15Z	ERROR	klog	Unhandled Error	{"error": "apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"content is not a list: []unstructured.Unstructured\"}: content is not a list: []unstructured.Unstructured"}
k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorToAPIStatus
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/handlers/responsewriters/status.go:71
k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/handlers/responsewriters/writers.go:316
k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/handlers/rest.go:113
k8s.io/apiserver/pkg/endpoints/handlers.transformResponseObject
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/handlers/response.go:339
k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/handlers/get.go:307
k8s.io/apiserver/pkg/endpoints.(*APIInstaller).registerResourceHandlers.restfulListResource.func6
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/installer.go:1270
k8s.io/apiserver/pkg/endpoints.(*APIInstaller).registerResourceHandlers.InstrumentRouteFunc.func7
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/metrics/metrics.go:600
github.com/emicklei/go-restful/v3.(*Container).dispatch
	/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.11.0/container.go:299
github.com/emicklei/go-restful/v3.(*Container).Dispatch
	/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.11.0/container.go:204
k8s.io/apiserver/pkg/server.director.ServeHTTP
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/server/handler.go:146
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func22
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filterlatency/filterlatency.go:110
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filters.withAuthorization.func1
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filters/authorization.go:83
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filterlatency/filterlatency.go:84
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func23
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filterlatency/filterlatency.go:110
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server/filters.(*priorityAndFairnessHandler).Handle.func9
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/server/filters/priority-and-fairness.go:292
k8s.io/apiserver/pkg/util/flowcontrol.(*configController).Handle.func2
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/util/flowcontrol/apf_filter.go:192
k8s.io/apiserver/pkg/util/flowcontrol/fairqueuing/queueset.(*request).Finish.func1
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/util/flowcontrol/fairqueuing/queueset/queueset.go:391
k8s.io/apiserver/pkg/util/flowcontrol/fairqueuing/queueset.(*request).Finish
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/util/flowcontrol/fairqueuing/queueset/queueset.go:392
k8s.io/apiserver/pkg/util/flowcontrol.(*configController).Handle
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/util/flowcontrol/apf_filter.go:179
k8s.io/apiserver/pkg/server/filters.(*priorityAndFairnessHandler).Handle.func10
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/server/filters/priority-and-fairness.go:298
k8s.io/apiserver/pkg/server/filters.(*priorityAndFairnessHandler).Handle
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/server/filters/priority-and-fairness.go:299
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filterlatency/filterlatency.go:84
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func24
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filterlatency/filterlatency.go:110
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.WithImpersonation.func4
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filters/impersonation.go:50
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filterlatency/filterlatency.go:84
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func25
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filterlatency/filterlatency.go:110
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filterlatency/filterlatency.go:84
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func27
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filterlatency/filterlatency.go:110
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filters/authentication.go:120
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filterlatency/filterlatency.go:94
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.WithWarningRecorder.func11
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/endpoints/filters/warning.go:35
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2294
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1
	/go/pkg/mod/k8s.io/apiserver@v0.31.2/pkg/server/filters/timeout.go:115

Additional context
.

Checklist

• I have checked the documentation
• I have searched for similar issues
• I have included all required information
• I have provided clear steps to reproduce
• I have included relevant logs

[dosubot]

Hi @WanzenBug! I'm Dosu and I’m helping the cozystack team.

This is a known bug in Cozystack v0.38.1 (and v0.38.0) where the API server returns typed objects instead of unstructured objects for its custom resources, causing a type mismatch during resource cleanup. This leads to the error content is not a list: []unstructured.Unstructured and blocks namespace deletion for any namespace that ever contained one of the 25 apps.cozystack.io/v1alpha1 resources. The root cause is how the API server registers and returns list responses for these resources, which breaks Kubernetes' expectations during namespace finalization and cleanup. Fixing this requires a code-level change in Cozystack so that list responses always use the correct unstructured format—there is no known workaround at this time. For more details, see GitHub issue #1673.

If you need to urgently remove stuck namespaces, you may attempt Kubernetes-generic force deletion methods (like manually editing the namespace object to remove finalizers), but these are not guaranteed to work if the API server cannot list and clean up the resources. The permanent fix will require an upgrade once a patch is released.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}