Life of a Prow Job
NOTE: This document uses 5df7636b83cab54e248e550a31dbf1e4731197a6 (July 21, 2021) as a reference point for all code links.
Let’s pretend a user comments
/test all on a Pull Request (PR).
In response, GitHub posts this comment to Prow via a webhook.
See examples for webhook payloads.
Prow’s Kubernetes cluster uses an ingress resource for terminating TLS, and routes traffic to the hook service resource, finally sending the traffic to the hook application, which is defined as a deployment:
- This document describes the configuration for the ingress resource.
- This document describes the configuration for the hook service.
- This document defines the pods for the hook application.
The pods for hook run the hook executable.
hook listens for incoming HTTP requests and translates them to “GitHub event objects”.
For example, in the case of the
/test all comment from above, hook builds an
Afterwards, hook broadcasts these events to Prow Plugins.
Prow Plugins receive 2 objects:
- a GitHub event object, and
ClientAgent object contains the following clients:
- GitHub client
- Prow job client
- Kubernetes client
- BuildClusterCoreV1 clients
- Git client
- Slack client
- Owners client
- Bugzilla client
- Jira client
These clients are initialized by hook, during start-up.
hook handles events by looking at
X-GitHub-Event, a custom HTTP header.
ConfigAgent object, initialized during hook’s startup, selects plugins to handle events.
See githubeventserver.go for more details, and check plugins.yaml for a list of plugins per repo.
Going back to the example, hook delivers an event that represents the
/test all comment to the Trigger plugin.
The Trigger plugin validates the PR before running tests.
One such validation is, for instance, that the author is a member of the organization or that the PR is labeled
The function called
handleGenericComment describes Trigger’s logic.
If all conditions are met (
ok-to-test, the comment is not a bot comment, etc.),
handleGenericComment determines which presubmit jobs to run.
The initial list of presubmit jobs to run (before being filtered down to those that qualify for this particular comment), is retrieved with
Next, for each presubmit we want to run, the trigger plugin talks to the Kubernetes API server and creates a
ProwJob with the information from the PR comment.
ProwJob is primarily composed of the
Pod details aside, a sample ProwJob might look like this:
apiVersion: prow.k8s.io/v1 kind: ProwJob metadata: name: 32456927-35d9-11e7-8d95-0a580a6c1504 spec: job: pull-test-infra-bazel decorate: true pod_spec: containers: - image: gcr.io/k8s-staging-test-infra/bazelbuild:latest-test-infra refs: base_ref: master base_sha: 064678510782db5b382df478bb374aaa32e577ea org: kubernetes pulls: - author: ixdy number: 2716 sha: dc32ccc9ea3672ccc523b7cbaa8b00360b4183cd repo: test-infra type: presubmit status: startTime: 2017-05-10T23:34:22.567457715Z state: triggered
prow-controller-manager runs ProwJobs by launching them by creating a new Kubernetes pod. It knows how to schedule new ProwJobs onto the cluster, responding to changes in the ProwJob or cluster health.
When the ProwJob finishes (the containers in the pod have finished running), prow-controller-manager updates the ProwJob.
crier reports back the status of the ProwJob back to the various external services like GitHub (e.g., as a green check-mark on the PR where the original
/test all comment was made).
A day later, sinker notices that the job and pod are a day old and deletes them from the Kubernetes API server.
Here is a summary of the above:
- User types in
/test allas a comment into a GitHub PR.
- GitHub sends a webhook (HTTP request) to Prow, to the
- The request gets intercepted by the ingress.
- The ingress routes the request to the hook service.
- The hook service in turn routes traffic to the hook application, defined as a deployment.
- The container routes traffic to the hook binary inside it.
- hook binary parses and validates the HTTP request and creates a GitHub event object.
- hook binary sends the GitHub event object (in this case
handleGenericCommentEventsends the data to be handled by the
- The data in the comment gets sent from hook to one of its many plugins, one of which is trigger. (The pattern is that hook constructs objects to be consumed by various plugins.)
- trigger determines which presubmit jobs to run (because it sees the
- trigger creates a ProwJob object!
- prow-controller-manager creates a pod to start the ProwJob.
- When the ProwJob’s pod finishes, prow-controller-manager updates the ProwJob.
- crier sees the updated ProwJob status and reports back to the GitHub PR (creating a new comment).
- sinker cleans up the old pod from above and deletes it from the Kubernetes API server.
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.