How to Fix “Too long: must have at most 262144 bytes” in ArgoCD

This error message in ArgoCD:

one or more objects failed to apply, reason: CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

Now with ArgoCD > 2.5, we have finally an option to deploy large CRDS without any workaround like using ArgoCD applications. It is called: Server-Side Apply.

What is Server-Side Apply?

Server-Side Apply (SSA) has been generally available in Kubernetes since the v1.22 release in August 2021. It’s a strategy for declarative resource management that improves diff calculations and warns about merge conflicts by moving the logic of the kubectl apply command onto the server.

What is Client-Side Apply?

We talk about client-side apply, when we run kubectl apply -f xyz.yaml. The declarative configuration you wrote is stored in the field metadata.annotations.kubectl.kubernetes.io/last-applied-configuration on the object.

If you then change something in the yaml file and run kubectl diff -f xyz.yaml you will see the diff of what gets changed. You will not get a diff if someone has run an imperative command like kubectl scale deployment xyz --replicas=3 because it is not stored in the last-applied-configuration field. So that makes it hard to track where the value is coming from.

When you run kubectl apply -f xyz.yaml on an existing object, the client-side apply will make a strategic merge and update the values that change from last-applied-configuration. The configuration will not be replaced!

Server-Side Apply in action

The idea behind server-side apply is to have a simple way to track changes to objects and see who is the owner of the field.

When using server-side apply, the declarative configuration is stored in the field metadata.managedFields on the object. managedFields enables conflict detection so Kubernetes knows when multiple actors are trying to change the same field. An actor can be a user, a controller, or another component of the system.

To use server-side apply, you need to use the kubectl apply -f xyz --server-side flag.

How to use Server-Side Apply with ArgoCD?

To use server-side apply with ArgoCD, you need to add to the syncOptions in the ArgoCD application the value ServerSideApply=true.

The new Application yaml looks like this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  name: kube-prometheus-stack
  namespace: argocd
spec:
  destination:
    namespace: monitoring
    server: https://kubernetes.default.svc
  project: default
  source:
    chart: kube-prometheus-stack
    helm:
      values: |-
        prometheus-node-exporter:
           hostRootFsMount:
             enabled: false
        prometheusOperator:
           admissionWebhooks:
             failurePolicy: Ignore
    repoURL: https://prometheus-community.github.io/helm-charts
    targetRevision: 45.8.1
  syncPolicy:
    syncOptions:
    - ServerSideApply=true
    - CreateNamespace=true

Check more troubleshooting Articles on: K8s Troubleshooting Archives — FoxuTech