Skip to content

Commit 16ac342

Browse files
reginaelysejaideepr97
reginaelyse
and
jaideepr97
authored
add documentation for applicationSets (#639)
Co-authored-by: Jaideep Rao <jaideep.r97@gmail.com>
1 parent fca32f8 commit 16ac342

File tree

1 file changed

+339
-0
lines changed

1 file changed

+339
-0
lines changed

‎docs/OpenShift GitOps Usage Guide.md‎

Lines changed: 339 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
17. [Upgrade GitOps Operator from v1.1.2 to v1.2.0 (GA)](#upgrade-gitops-operator-from-v112-to-v120-ga)
2121
18. [GitOps Monitoring Dashboards](#gitops-monitoring-dashboards)
2222
19. [Integrate GitOps with Secrets Management](Integrate%20GitOps%20with%20Secrets%20Management.md)
23+
20. [Using ApplicationSets](#using-applicationsets)
2324

2425
## Installing OpenShift GitOps
2526

@@ -1003,3 +1004,341 @@ As of GitOps Operator v1.10.0, the operator will deploy monitoring dashboards in
10031004
![Dashboard Select Dropdown](assets/39.gitops_monitoring_dashboards_dropdown.png)
10041005
10051006
**Note: At this time disabling or changing the content of the dashboards is not supported.**
1007+
1008+
## Using ApplicationSets
1009+
1010+
#### Background:
1011+
1012+
Operators may be responsible for hundreds of clusters (including cluster additions and deletions), and thus installing and managing these add-ons must be fully automated. Monorepos present problems with manual effort, and giving users the correct level of access to applications can be a challenge overall.
1013+
1014+
A lot of these problems can be solved with the use of the App-of-Apps pattern; however, the App-of-Apps pattern still presents some issues:
1015+
- Users must still maintain a collection of individual Application resources in git
1016+
- Because of the repetitive nature of writing these Application specs, users often end up scripting the generation of Application specs
1017+
- Because the Application CR exposes sensitive fields such as project, cluster, namespace a high degree of trust is required of developers
1018+
- sync and deletion behaviour of a 'parent app' can sometimes produce counterintuitive results on child apps; etc..
1019+
1020+
A better way to solve these problems than using the App-of-Apps pattern is through the use of ApplicationSets.
1021+
1022+
### Introducing ApplicationSets
1023+
1024+
ApplicationSets are a separate CRD that allows cluster administrators to define a single ApplicationSet Kubernetes YAML resource that will generate (and update or delete) multiple corresponding Argo CD Application CRs.
1025+
1026+
ApplicationSets offers a number of improvements via automation. With ApplicationSets you can:
1027+
- automatically deploy to multiple cluster at once, and automatically adapt to the addition/removal of clusters
1028+
- handle large deployments of Argo CD Applications from a single mono-repository, automatically responding to the addition/removal of new applications to the repository
1029+
- enable development teams to manage large groups of applications securely, via self-service, without cluster administrator review, on a cluster managed via Argo CD.
1030+
- have applications managed by the ApplicationSet controller, which can be managed by only a single instance of an ApplicationSet custom resource (CR)... that means no more juggling of large numbers of Argo CD Application objects when targeting multiple clusters/repositories!
1031+
1032+
When you create, update, or delete an ApplicationSet resource, the ApplicationSet Controller responds by creating, updating, or deleting one or more corresponding Argo CD Application resources.
1033+
1034+
The ApplicationSet Controller does:
1035+
- create, update, and delete Application resources within the default `gitops-operator` namespace.
1036+
- ensure that the Application resources remain consistent with the defined declarative ApplicationSet resource
1037+
1038+
The ApplicationSet Controller does NOT:
1039+
- Create/modify/delete Kubernetes resources (other than the Application CR)
1040+
- Connect to clusters other than the one Argo CD is deployed to
1041+
- Interact with namespaces other than the one Argo CD is deployed within
1042+
- Manage the application resources of the applications it creates (Argo CD itself is responsible for the actual deployment of the generated child Application resources, such as Deployments, Services, and ConfigMaps)
1043+
1044+
The ApplicationSet Controller can thus be thought of as an Application 'factory', taking an ApplicationSet resource as input, and outputting one or more Argo CD Application resources that correspond to the parameters of that set.
1045+
1046+
Creation, update, or deletion of ApplicationSets will have a direct effect on the Applications present in the Argo CD namespace. Likewise, cluster events (the addition/deletion of Argo CD cluster secrets, when using Cluster generator), or changes in Git (when using Git generator), will be used as input to the ApplicationSet controller in constructing Application resources.
1047+
1048+
Because it is such a powerful tool, **only admins may create, update, and delete ApplicationSets.**
1049+
1050+
Note: All ApplicationSet resources and the ApplicationSet controller must be installed in the same namespace as Argo CD. ApplicationSet resources in a different namespace will be ignored.
1051+
1052+
### ApplicationSets Getting Started and Installation
1053+
1054+
In all current supported versions of GitOps Operator, there is no extra installation needed to install the ApplicationSet controller with your existing GitOps Operator Installation.
1055+
1056+
The name of the ApplicationSet Controller that installs as part of the OpenShift GitOps Operator is `openshift-gitops-applicationset-controller`.
1057+
1058+
The following options are available for the Argo CD ApplicationSet Controller in the ArgoCD CR under `.spec.applicationSet`:
1059+
1060+
| Name | Default | Description |
1061+
| -------- | ------- |------- |
1062+
| env | Empty | Env lets you specify environment for applicationSet controller pods |
1063+
| extraCommandArgs | Empty | ExtraCommandArgs allows users to pass command line arguments to ApplicationSet controller. They get added to default command line arguments provided by the operator. Please note that the command line arguments provided as part of ExtraCommandArgs will not overwrite the default command line arguments. |
1064+
| image | Empty | Image is the Argo CD ApplicationSet image (optional) |
1065+
| logLevel | ArgoCDDefaultLogLevel | The log level to be used by the ArgoCD Application Controller component. Valid options are `debug` , `info` , `error` , and `warn`. |
1066+
| resources | Empty | Resources defines the Compute Resources required by the container for ApplicationSet. |
1067+
| version | Empty | Version is the Argo CD ApplicationSet image tag.(optional) |
1068+
| webhookServer | Empty | WebhookServerSpec defines the options for the ApplicationSet Webhook Server component.|
1069+
| scmRootCAConfigMap | Empty | SCMRootCAConfigMap is the name of the config map that stores the Gitlab SCM Provider's TLS certificate which will be mounted on the ApplicationSet Controller (optional)|
1070+
<!--- Add "enabled" option once it's functional- either v1.11.1 or v1.12? -->
1071+
<!--- | enabled | true | Enabled is the flag to enable the Application Set Controller during ArgoCD installation. (optional) | -->
1072+
1073+
1074+
**ApplicationSet ExtraCommandArgs:**
1075+
1076+
The example shows how a user can add command arguments to the ApplicationSet Controller.
1077+
1078+
```
1079+
apiVersion: argoproj.io/v1beta1
1080+
kind: ArgoCD
1081+
metadata:
1082+
name: example
1083+
spec:
1084+
applicationSet:
1085+
extraCommandArgs:
1086+
- --foo
1087+
- bar
1088+
```
1089+
1090+
The following describes all the available fields of an ApplicationSet:
1091+
```
1092+
apiVersion: argoproj.io/v1alpha1
1093+
kind: ApplicationSet
1094+
metadata:
1095+
name: hello-world-appset
1096+
namespace: openshift-gitops
1097+
spec:
1098+
# See docs for available generators and their specs.
1099+
generators:
1100+
- list:
1101+
elements:
1102+
- cluster: https://kubernetes.default.svc
1103+
# Determines whether go templating will be used in the `template` field below.
1104+
goTemplate: false
1105+
# Optional list of go templating options, see https://pkg.go.dev/text/template#Template.Option
1106+
# This is only relevant if `goTemplate` is true
1107+
goTemplateOptions: ["missingkey="]
1108+
# These fields are identical to the Application spec.
1109+
template:
1110+
metadata:
1111+
name: hello-world-app
1112+
spec:
1113+
project: my-project
1114+
# This sync policy pertains to the ApplicationSet, not to the Applications it creates.
1115+
syncPolicy:
1116+
# Determines whether the controller will delete Applications when an ApplicationSet is deleted.
1117+
preserveResourcesOnDeletion: false
1118+
# Alpha feature to determine the order in which ApplicationSet applies changes.
1119+
strategy:
1120+
# This field lets you define fields which should be ignored when applying Application resources. This is helpful if you
1121+
# want to use ApplicationSets to create apps, but also want to allow users to modify those apps without having their
1122+
# changes overwritten by the ApplicationSet.
1123+
ignoreApplicationDifferences:
1124+
- jsonPointers:
1125+
- /spec/source/targetRevision
1126+
- name: some-app
1127+
jqPathExpressions:
1128+
- .spec.source.helm.values
1129+
```
1130+
1131+
### ApplicationSet Generators
1132+
1133+
Generators are responsible for generating parameters, which are then rendered into the template fields of the ApplicationSet resource.
1134+
1135+
There are currently (as of GitOps v1.11.0) 9 types of Generators:
1136+
- List Generator: The List generator allows you to target Argo CD Applications to clusters based on a fixed list of any chosen key/value element pairs.
1137+
- Cluster Generator: The Cluster generator allows you to target Argo CD Applications to clusters, based on the list of clusters defined within (and managed by) Argo CD (which includes automatically responding to cluster addition/removal events from Argo CD).
1138+
- Git Generator: The Git generator allows you to create Applications based on files within a Git repository, or based on the directory structure of a Git repository.
1139+
- Matrix Generator: The Matrix generator may be used to combine the generated parameters of two separate generators.
1140+
- Merge Generator: The Merge generator may be used to merge the generated parameters of two or more Generators. Additional generators can override the values of the base generator.
1141+
- SCM Provider Generator: The Source Code Management (SCM) Provider generator uses the API of an SCM provider (eg GitHub) to automatically discover repositories within an organization.
1142+
- Pull Request Generator: The Pull Request generator uses the API of an SCMaaS provider (eg GitHub) to automatically discover open pull requests within an repository.
1143+
- Cluster Decision Resource Generator: The Cluster Decision Resource generator is used to interface with Kubernetes custom resources that use custom resource-specific logic to decide which set of Argo CD clusters to deploy to.
1144+
- Plugin Generator: The Plugin generator makes RPC HTTP requests to provide parameters.
1145+
1146+
NOTE: Clusters that you put into any generators must already be defined in Argo CD in order to use them with ApplicationSets. The ApplicationSet controller does not create clusters within Argo CD as it does not have credentials to do so.
1147+
1148+
1149+
### ApplicationSet Resource Updating, Pruning & Deletion
1150+
1151+
The ApplicationSet Controller creates/modifies/deletes Argo CD Application resources, based on the contents of the ApplicationSet CR.
1152+
1153+
Once again, please note that **only admins may create, update, and delete ApplicationSets.**
1154+
1155+
All Argo CD Application resources created by the ApplicationSet controller (from an ApplicationSet) will contain:
1156+
- A `.metadata.ownerReferences` reference back to the parent ApplicationSet resource
1157+
- This means that when the ApplicationSet is deleted, the Applications will automatically be deleted.
1158+
- An Argo CD `resources-finalizer.argocd.argoproj.io` finalizer in `.metadata.finalizers` of the Application (only applies if `.syncPolicy.preserveResourcesOnDeletion` is set to false)
1159+
- This means that when the Application is deleted, all of the child resources of that Application will be deleted.
1160+
1161+
When an applicationSet is deleted, the following will occur:
1162+
1. The ApplicationSet itself is deleted
1163+
2. Any Applications that were created from the now-deleted ApplicationSet will be deleted (or Applications that have an owner-reference to the deleted ApplicationSet)
1164+
3. Any deployed resources (Deployments, Services, Configmaps, etc), from the deleted Applications will also be deleted.
1165+
1166+
However, the ApplicationSet controller has settings that can limit its ability to make changes to Applications it has generated, for example, preventing the controller from deleting child Applications.
1167+
1168+
Some of these settings are…
1169+
1170+
#### 1. Dry-run: Prevent ApplicationSet from creating/modifying/deleting all Applications
1171+
1172+
To prevent the ApplicationSet controller from creating, modifying, or deleting any Application resources, you may enable dry-run mode. This essentially switches the controller into a "read only" mode, where the controller reconcile loop will run, but no resources will be modified.
1173+
1174+
To enable dry-run:
1175+
1176+
1. Add `--dryrun true` to the ApplicationSet Deployment's container launch parameters with `kubectl edit deployment/openshift-gitops-applicationset-controller -n openshift-gitops`
1177+
2. Locate the `.spec.template.spec.containers[0].command` field, and add the required parameter(s):
1178+
```
1179+
spec:
1180+
# (...)
1181+
template:
1182+
# (...)
1183+
spec:
1184+
containers:
1185+
- command:
1186+
- entrypoint.sh
1187+
- openshift-gitops-applicationset-controller
1188+
# Insert new parameters here, for example:
1189+
# --dryrun true
1190+
# (...)
1191+
```
1192+
#### 2. Using Managed Application Modification Policies
1193+
1194+
The ApplicationSet Controller spec allows you to add a `syncPolicy.applicationsSync` field, like so:
1195+
```
1196+
apiVersion: argoproj.io/v1alpha1
1197+
kind: ApplicationSet
1198+
spec:
1199+
# (...)
1200+
syncPolicy:
1201+
applicationsSync: create-only
1202+
```
1203+
The options for the `applicationsSync` field are:
1204+
- create-only: Prevents ApplicationSet controller from modifying or deleting Applications.
1205+
- create-update: Prevents ApplicationSet controller from deleting Applications. Update is allowed.
1206+
- create-delete: Prevents ApplicationSet controller from modifying Applications. Delete is allowed.
1207+
- sync (default): Update and Delete are allowed.
1208+
1209+
You could also do the same via the `--policy` addition to the ApplicationSet controller Deployment.
1210+
Run `kubectl edit deployment/gitops-applicationset-controller -n openshift-gitops` and then edit the `.spec.template.spec.containers[0].command` to have the following:
1211+
```
1212+
spec:
1213+
# (...)
1214+
template:
1215+
# (...)
1216+
spec:
1217+
containers:
1218+
- command:
1219+
- entrypoint.sh
1220+
- openshift-gitops-applicationset-controller
1221+
# Insert new parameters here, for example:
1222+
# --policy create-update
1223+
# (...)
1224+
```
1225+
When `--policy` is set as it is above, it does take precedence over the previously mentioned `.spec.syncPolicy.applicationsSync` field.
1226+
1227+
Note: this does not prevent deletion of Applications if the ApplicationSet itself is deleted.
1228+
1229+
#### 3. Ignoring certain changes to Applications
1230+
1231+
The ApplicationSet spec has an `ignoreApplicationDifferences` field which allows you to specify which fields of the ApplicationSet should be ignored when comparing Applications. Multiple ignore rules can be used and each ignore rule may specify a list of either `jsonPointers` or `jqPathExpressions` to ignore. You may optionally also specify a name to apply the ignore rule to a specific Application, or omit the name to apply the ignore rule to all Applications.
1232+
1233+
```
1234+
apiVersion: argoproj.io/v1alpha1
1235+
kind: ApplicationSet
1236+
spec:
1237+
ignoreApplicationDifferences:
1238+
- jsonPointers:
1239+
- /spec/source/targetRevision
1240+
- name: some-app
1241+
jqPathExpressions:
1242+
- .spec.source.helm.values
1243+
```
1244+
**Temporarily Ignoring Certain Changes to Applications**
1245+
1246+
If you have an ApplicationSet that is configured to automatically sync Applications, there may come a time when you may want to temporarily disable auto-sync for a specific Application. You can do this by adding an ignore rule for the spec.syncPolicy.automated field:
1247+
```
1248+
apiVersion: argoproj.io/v1alpha1
1249+
kind: ApplicationSet
1250+
spec:
1251+
ignoreApplicationDifferences:
1252+
- jsonPointers:
1253+
- /spec/syncPolicy
1254+
```
1255+
1256+
**Limitations of `ignoreApplicationDifferences`: **
1257+
1258+
When the ignored field is part of a list, such as when you have an Application with multiple sources, changes to other fields or in other sources will cause the entire list to be replaced, and the related fields will be reset to the value defined in the ApplicationSet. This is due to how ApplicationSets generate MergePatches, which state that “existing lists will be completely replaced by new lists" when there is a change to the list. So in the case of having multiple sources, if we have an ApplicationSet that looks like this:
1259+
```
1260+
apiVersion: argoproj.io/v1alpha1
1261+
kind: ApplicationSet
1262+
spec:
1263+
ignoreApplicationDifferences:
1264+
- jqPathExpressions:
1265+
- .spec.sources[] | select(.repoURL == "https://git.example.com/org/repo1").targetRevision
1266+
template:
1267+
spec:
1268+
sources:
1269+
- repoURL: https://git.example.com/org/repo1
1270+
targetRevision: main
1271+
- repoURL: https://git.example.com/org/repo2
1272+
targetRevision: main
1273+
```
1274+
We can change the `targetRevision` of the first repo source and the ApplicationSet controller will not overwrite the changes.
1275+
```
1276+
apiVersion: argoproj.io/v1alpha1
1277+
kind: Application
1278+
spec:
1279+
sources:
1280+
- repoURL: https://git.example.com/org/repo1
1281+
targetRevision: fix/bug-123
1282+
- repoURL: https://git.example.com/org/repo2
1283+
targetRevision: main
1284+
```
1285+
But if we change the targetRevision of the second repo source, the ApplicationSet will overwrite the entire sources field.
1286+
```
1287+
apiVersion: argoproj.io/v1alpha1
1288+
kind: Application
1289+
spec:
1290+
sources:
1291+
- repoURL: https://git.example.com/org/repo1
1292+
targetRevision: main
1293+
- repoURL: https://git.example.com/org/repo2
1294+
targetRevision: main
1295+
```
1296+
1297+
#### 4. Preventing an Application’s child resources from being deleted when the parent application is deleted
1298+
1299+
By default, when an Application resource is deleted by the ApplicationSet controller, all of the child resources of the Application will be deleted as well. To prevent this behavior and keep the child resources, add the `preserveResourcesOnDeletion: true` field to the syncPolicy of the ApplicationSet:
1300+
```
1301+
apiVersion: argoproj.io/v1alpha1
1302+
kind: ApplicationSet
1303+
spec:
1304+
# (...)
1305+
syncPolicy:
1306+
preserveResourcesOnDeletion: true
1307+
```
1308+
Using the `preserveResourcesOnDeletion` policy will prevent the finalizer from being added to Applications, thus stopping their deletion.
1309+
1310+
Another way of achieving this behavior would be to delete the ApplicationSet using a non-cascading delete:
1311+
```
1312+
kubectl delete ApplicationSet (NAME) --cascade=orphan
1313+
```
1314+
1315+
Please note though that when using the non-cascaded delete, the `resources-finalizer.argocd.argoproj.io` is still specified on the Application. This means that when the Application is deleted, all of its deployed resources will also be deleted. If you don’t want this behavior, use the `preserveResourcesOnDeletion` policy above instead.
1316+
1317+
1318+
#### 5. Preventing an Application’s child resources from being modified
1319+
1320+
When an Application resource is updated by the ApplicationSet controller, all of the child resources of the Application will be updated accordingly. However, by default (and as a safety mechanism), automated syncs will not delete resources when Argo CD detects the resource is no longer defined in Git. You can "pause" updates to cluster resources managed by the `Application` resource by unsetting the `spec.template.syncPolicy.automated`. As an extra precaution, you could also have `spec.template.syncPolicy.automated.prune` set to false to prevent any unexpected changes.
1321+
```
1322+
apiVersion: argoproj.io/v1alpha1
1323+
kind: ApplicationSet
1324+
spec:
1325+
# (...)
1326+
syncPolicy:
1327+
automated:
1328+
prune: false
1329+
```
1330+
1331+
#### 6. Preserving Application annotation and labels
1332+
1333+
If you have annotations and/or labels that you have applied to Applications after their generation by an ApplicationSet, to keep those annotations/labels when syncing you must use the `preservedFields` property like below:
1334+
```
1335+
apiVersion: argoproj.io/v1alpha1
1336+
kind: ApplicationSet
1337+
spec:
1338+
# (...)
1339+
preservedFields:
1340+
annotations: ["my-custom-annotation"]
1341+
labels: ["my-custom-label"]
1342+
```
1343+
By default, the Argo CD notifications and the Argo CD refresh type annotations are also preserved.
1344+

0 commit comments

Comments
 (0)