fix(infrahub-enterprise): give OpenShift pods a writable home directory

Merged
FA
6d ago
+100-22

Private CA settings have no effect on OpenShift

A customer deploying on OpenShift reported two problems. The task workers could not create /opt/infrahub/.gitconfig, and git certificate verification failed until they moved that file to /tmp. They also saw the same permission error for /.prefect.

Running the 1.11.5 enterprise image with an OpenShift-style uid reproduces both:

  • The uid has no entry in /etc/passwd, so HOME is /. Prefect puts its home directory at ~/.prefect (source), so it tries to create /.prefect and fails.

  • At startup the task worker writes its global git configuration to git.global_config_file, which defaults to /opt/infrahub/.gitconfig (source). The image owns that file and its directory as uid 1000, so every write fails. The worker logs each failure and keeps running (source):

    Failed to set git http.sslCAInfo: error: could not lock config file /opt/infrahub/.gitconfig: Permission denied

  • The image's copy of that file already holds the git user and credential helper, so cloning still works. But the http.sslCAInfo and http.sslVerify keys, set from git.tls_ca_file and git.tls_insecure, are never written. Git then checks HTTPS remotes against the system certificates only.

Changes to values.openshift.yaml

  • The Infrahub server, task worker and upgrade job get HOME=/home/infrahub on an emptyDir, which the SCC's fsGroup makes writable. The Prefect chart already gives its own pods HOME=/home/prefect on an emptyDir (source).
  • The task worker gets INFRAHUB_GIT_GLOBAL_CONFIG_FILE=/home/infrahub/.gitconfig. The worker writes every key at startup, including the git user and credential helper, so nothing from the image's copy is lost.
  • An emptyDir at /prom_shared replaces PROMETHEUS_MULTIPROC_DIR=/tmp on the server and the task worker. The overlay already handles Prefect this way. /tmp can stop the server container:
    • Each gunicorn worker deletes the entries of the metrics directory when it starts, and fails with IsADirectoryError on a subdirectory (source).
    • Infrahub creates the subdirectory /tmp/infrahub-tls when a CA bundle is supplied as PEM text (source).
    • Gunicorn builds each worker in its main process before forking (source). The error therefore stops the whole server container the next time a worker restarts.

Not decided

  • Whether the Infrahub image should make these paths writable by group 0, as OpenShift recommends for images that run with an arbitrary uid (OKD docs). That would remove the need for these overrides. It is not part of this PR.

Related pull requests

  • #89 added the overlay.

Documentation and changelog updates

  • The OpenShift section of the chart README (README.md.gotmpl and the generated README.md).
  • charts/infrahub-enterprise/changelog/+openshift-writable-home.fixed.md.

Tests run

Done locally:

  • Two new tests in tests/e2e/test_infrahub_enterprise_openshift.py:
    • The server and the task worker can write to $HOME, /prom_shared and /opt/infrahub/git as the assigned uid.
    • The task worker writes its git configuration into a directory it can write.
  • In a vcluster, the full module passes with this change (6 tests). Against the previous overlay, the 3 new test cases fail, and the task worker logs the same could not lock config file errors.
  • In the 1.11.5 enterprise image, run as uid 1000670000 with git.tls_ca_file set, the task worker's set_git_global_config fails on every key before this change. With this change, it writes http.sslCAInfo, and Prefect creates /home/infrahub/.prefect.
  • helm template with the medium preset and the overlay: the server, the task worker and the upgrade job get the new environment variables and volumes.
  • helm lint, yamllint, and a towncrier build --draft preview of the changelog fragment.

CI does not run this module, because it has only the manual marker. These changes have not been tested on an OpenShift cluster: in the vcluster, the SCC-simulation values file assigns the uid.

Generated with Claude Code