CI Pipeline Outcomes Sankey Diagram: Trigger to Failure Cause
Trace pipeline runs from trigger to result and the cause of each failure: flaky tests, real regressions, config and environment. Replace the sample counts with your CI history.
A CI pipeline Sankey shows how builds end by trigger and why they fail. It separates failures that mean something (a regression) from the ones that do not (a flaky test).
The example has 900 pipeline runs from three triggers: pushes to main, pull requests and nightly builds. 760 pass every stage and 140 fail, at build (37), tests (96) or deploy (7). The last column gives the cause: flaky test, real regression, code error, dependency or config, and test environment.
Pass rate hides the mix. In the example, 35 of the 96 test failures are flaky, which is a trust problem for the suite, and 45 are real regressions, which is the pipeline working. Counts are runs, not commits, so a retry is a run. For the review stage before it, see /sankey/pull-request-review-flow, and /guides/how-to-find-bottlenecks-in-a-software-release-process for the wider process.
How to read this diagram
- 760 of 900 runs (84%) pass every stage; 140 (16%) fail.
- Tests are the most common point of failure: 96 of 140 failures (69%).
- 35 failures are flaky tests, 25% of all failures, and 45 are real regressions.
- Pull request runs pass 82% of the time (410 of 500), main 90% (270 of 300).
The data behind it
Invented figures for illustration. Not data about any real company, team or system.
| From | To | Amount (pipeline runs) |
|---|---|---|
| Push to main | Passed all stages | 270 |
| Push to main | Failed at build | 8 |
| Push to main | Failed at tests | 17 |
| Push to main | Failed at deploy | 5 |
| Pull request | Passed all stages | 410 |
| Pull request | Failed at build | 25 |
| Pull request | Failed at tests | 65 |
| Nightly | Passed all stages | 80 |
| Nightly | Failed at build | 4 |
| Nightly | Failed at tests | 14 |
| Nightly | Failed at deploy | 2 |
| Failed at build | Code error | 15 |
| Failed at build | Dependency or config | 22 |
| Failed at tests | Flaky test | 35 |
| Failed at tests | Real regression | 45 |
| Failed at tests | Environment | 16 |
| Failed at deploy | Dependency or config | 3 |
| Failed at deploy | Environment | 4 |
The same data as CSV
source,target,value
Push to main,Passed all stages,270
Push to main,Failed at build,8
Push to main,Failed at tests,17
Push to main,Failed at deploy,5
Pull request,Passed all stages,410
Pull request,Failed at build,25
Pull request,Failed at tests,65
Nightly,Passed all stages,80
Nightly,Failed at build,4
Nightly,Failed at tests,14
Nightly,Failed at deploy,2
Failed at build,Code error,15
Failed at build,Dependency or config,22
Failed at tests,Flaky test,35
Failed at tests,Real regression,45
Failed at tests,Environment,16
Failed at deploy,Dependency or config,3
Failed at deploy,Environment,4How it works
Export pipeline runs for one period
Include the trigger, the first failing stage and the result. Keep reruns as separate runs.
Label the cause of each failure
Use a short list: flaky, regression, code error, config, environment. A failure with no cause goes to Unknown, which is worth showing.
Count trigger to result to cause
Count each run once, by its first failing stage.
Check the balance
Each failing stage must leave as many runs as entered. Passed ends at the second column.
Frequently asked questions
How do I tell a flaky test from a regression?
A common rule is that a test that fails and then passes on retry with no code change is flaky. Use whichever rule your team already has and keep it the same each period.
Why count runs and not commits?
Runs are what the CI system records. If you want commits, count each commit once by its final result and say so in the data note.
Edit this CI pipeline Sankey
Open the example as your own chart and enter your pipeline counts.