Prow Secrets Management
Secrets in prow service/build clusters are managed with Kubernetes External
Secrets, which is responsible for one-way syncing secret values from major
secret manager providers such as GCP, Azure, and AWS secret managers into
kubernetes clusters, based on
ExternalSecret custom resource defined in
cluster (As shown in example below).
Note: the instructions below are only for GCP secret manager, for authenticating with other providers please refer to https://github.com/external-secrets/kubernetes-external-secrets#backends
Set Up (Prow maintainers)
This is performed by prow service/build clusters maintainer.
- In the cluster that the secrets are synced to, enable workload identity by
config/prow/cluster. The deployment file assumes using the same service account name as used in step #1
- [Optional but recommended] Create postsubmit deploy job for managing the deployment, for example post-test-infra-deploy-prow.
Usage (Prow clients)
This is performed by prow serving/build cluster clients. Note that the GCP project mentioned here doesn’t have to, and normally is not the same GCP project where the prow service/build clusters are located.
In the GCP project that stores secrets with google secret manager, grant the
roles/secretmanager.secretAccessorpermission to the GCP service account used above, by running:
gcloud beta secrets add-iam-policy-binding <my-gsm-secret-name> --member="serviceAccount:<same-service-account-for-workload-identity>" --role=<role> --project=<my-gsm-secret-project>
The above command ensures that the service account used by prow can only access the secret name
<my-gsm-secret-name>in the GCP project owned by clients. The service account used for prow.k8s.io (aka
test-infra-trustedbuild cluster) is defined in
trusted_serviceaccounts.yaml, and the secrets are defined in
kubernetes_external_secrets.yaml. The service account used for
k8s-prow-buildscluster(aka the default build cluster) is defined in
build_serviceaccounts.yaml, and the secrets are defined in
Create secret in google secret manager
Create kubernetes external secrets custom resource by:
apiVersion: kubernetes-client.io/v1 kind: ExternalSecret metadata: name: <my-precious-secret-kes-name> # name of the k8s external secret and the k8s secret namespace: <ns-where-secret-is-used> spec: backendType: gcpSecretsManager projectId: <my-gsm-secret-project> data: - key: <my-gsm-secret-name> # name of the GCP secret name: <my-kubernetes-secret-name> # key name in the k8s secret version: latest # version of the GCP secret # Property to extract if secret in backend is a JSON object, # remove this line if using the GCP secret value straight property: value
Within 10 seconds (determined by
POLLER_INTERVAL_MILLISECONDS envvar on deployment), a secret will be created automatically:
apiVersion: v1 kind: Secret metadata: name: <my-precious-secret-kes-name> namespace: <ns-where-secret-is-used> data: <my-kubernetes-secret-name>: <value_read_from_gsm>
Secret will be updated automatically when the secret value in gsm changed
ExternalSecret is changed. when
ExternalSecret CR is deleted from the
cluster, the secret will be also be deleted by kubernetes external secret.
(Note: deleting the
ExternelSecret CR config from source control doesn’t
result in deletion of corresponding
ExternalSecret CR from the cluster as the
postsubmit action only does
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.