This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.


A brief guide about Prow and its ecosystem


Prow is a Kubernetes based CI/CD system. Jobs can be triggered by various types of events and report their status to many different services. In addition to job execution, Prow provides GitHub automation in the form of policy enforcement, chat-ops via /foo style commands, and automatic PR merging.

See the GoDoc for library docs. Please note that these libraries are intended for use by prow only, and we do not make any attempt to preserve backwards compatibility.

For a brief overview of how Prow runs jobs take a look at “Life of a Prow Job”.

To see common Prow usage and interactions flow, see the pull request interactions sequence diagram.

Functions and Features

  • Job execution for testing, batch processing, artifact publishing.
    • GitHub events are used to trigger post-PR-merge (postsubmit) jobs and on-PR-update (presubmit) jobs.
    • Support for multiple execution platforms and source code review sites.
  • Pluggable GitHub bot automation that implements /foo style commands and enforces configured policies/processes.
  • GitHub merge automation with batch testing logic.
  • Front end for viewing jobs, merge queue status, dynamically generated help information, and more.
  • Automatic deployment of source control based config.
  • Automatic GitHub org/repo administration configured in source control.
  • Designed for multi-org scale with dozens of repositories. (The Kubernetes Prow instance uses only 1 GitHub bot token!)
  • High availability as benefit of running on Kubernetes. (replication, load balancing, rolling updates…)
  • JSON structured logs.
  • Prometheus metrics.


Getting started

More details


The stability of prow is heavily relying on unit tests and integration tests.

Useful Talks

KubeCon 2020 EU virtual

KubeCon 2018 EU

KubeCon 2018 China

KubeCon 2018 Seattle


Prow in the wild

Prow is used by the following organizations and projects:

Jenkins X uses Prow as part of Serverless Jenkins.

Contact us

If you need to contact the maintainers of Prow you have a few options:

  1. Open an issue in the kubernetes/test-infra repo.
  2. Reach out to the #prow channel of the Kubernetes Slack.
  3. Contact one of the code owners in prow/OWNERS or in a more specifically scoped OWNERS file.

Bots home

@k8s-ci-robot lives here and is the face of the Kubernetes Prow instance. Here is a command list for interacting with @k8s-ci-robot and other Prow bots.

1 - Architecture

Prow is made up of a collection of microservices (aka “Prow components”) that work together in a service cluster, leveraging one or more build clusters to schedule Prow Jobs (or just “jobs”).

Prow in a Nutshell

Prow creates jobs based on various types of events, such as:

  • GitHub events (e.g., a new PR is created, or is merged, or a person comments “/retest” on a PR),

  • Pub/Sub messages,

  • time (these are created by Horologium and are called periodic jobs), and

  • retesting (triggered by Tide).

Jobs are created inside the Service Cluster as Kubernetes Custom Resources. The Prow Controller Manager takes triggered jobs and schedules them into a build cluster, where they run as Kubernetes pods. Crier then reports the results back to GitHub.

flowchart TD

    classDef yellow fill:#ff0
    classDef cyan fill:#0ff
    classDef pink fill:#f99

    subgraph Service Cluster["<span style='font-size: 40px;'><b>Service Cluster</b></span>"]
        PCM[Prow Controller Manager]:::cyan
            subgraph Hook
                WebhookHandler["Webhook Handler"]
                PluginCat(["'cat' plugin"])
                PluginTrigger(["'trigger' plugin"])

    subgraph Build Cluster[<b>Build Cluster</b>]

    style Legend fill:#fff,stroke:#000,stroke-width:4px
    subgraph Legend["<span style='font-size: 20px;'><b>LEGEND</b></span>"]
        direction LR
        k8sResource[Kubernetes Resource]:::yellow
        prowComponent[Prow Component]:::cyan
        hookPlugin([Hook Plugin])

    Prowjobs <-.-> Deck <-----> |Serve|
    GitHub ==> |Webhooks| WebhookHandler
    WebhookHandler --> |/meow| PluginCat
    WebhookHandler --> |/retest| PluginTrigger
    Prowjobs <-.-> Tide --> |Retest and Merge| GitHub
    Horologium ---> |Create| Prowjobs
    PluginCat --> |Comment| GitHub
    PluginTrigger --> |Create| Prowjobs
    Sinker ---> |Garbage collect| Prowjobs
    Sinker --> |Garbage collect| Pods
    PCM -.-> |List and update| Prowjobs
    PCM ---> |Report| Prowjobs
    PCM ==> |Create and Query| Pods
    Prowjobs <-.-> |Inform| Crier --> |Report| GitHub


Note that Prow can also work with Gerrit, albeit with less features. Specifically, neither Tide nor Hook work with Gerrit yet.