contents
Git is a tool — it does not dictate, inform or even suggest how developers organise their work amongst themselves, how miscellaneous development efforts ought to be be combined together and how the maintenance for released versions is related to ongoing development work. Often software development projects tend to do these things either on an ad-hoc basis, or impose stringent external project management rules and regulation. The Git-flow branching scheme strives towards a middle ground between both extremities and places the evolving code centre stage as it is recorded in Git branches and denominated by Git tags. This is achieved by introducing a consistent naming scheme together with some simple rules on when and how to branch, and when and how to merge.
A branching and code integration model is part of a project’s organisation and social structure. Some code bases and projects are plain and simple to the degree that they do not even need any formalised structure — while other projects are highly complex and tied to external responsibilities.
The essence of software is that it can be changed. People and organisations may rely on its workings, leading to new avenues of usage and the desire to adapt the software to evolving circumstances, while retaining all achievements, insights and the resolution of former conflicts embodied into the formulation of the software’s behaviour. As a collaborative effort, code can easily accumulate interdependencies to a degree surpassing the abilities and capacity of a single person or even a coherent group of people in charge of maintaining the application. Mistakes will be made, causing damage and conflict. An increasing amount of time and effort is required to both keep the system in a usable shape, and, at the same time, to retain the ability for change — and this effort in itself conflicts with the demand to expand the software’s abilities.
If a system encounters such a situation loaded with tension and conflict, dedicated care and consideration is required to avoid the generation and amplification of accidental complexity — which may lead to a loss of control and corrupt and nullify the software’s usability. A formalised process based on practical experience may help to mitigate this danger — allowing different persons to focus on some aspects in isolation, while creating controlled and predictable points of conflict resolution. This is the idea behind any kind of project management method, development scheme or integration model: allowing to focus on one problem at a time, and to resolve issues and conflicts in several steps, without the need to connect everything with everything.
One person or group of people must be able to focus on a single aspect, requirement or problem, while explicitly disregarding other requirements and circumstances existing at the same time. Any achievement requires some degree of simplification.
Yet aspects disregarded for the sake of focus must be considered in a second step, incrementally building up a solution and a state that is acceptable, and sustainable.
Any non trivial work requires time, and some efforts must be started in advance and carried out concurrently with other work to meet deadlines. Bugs must be fixed quickly and often external commitments (e.g. a release date) must be met.
It must be possible to prioritise work and move some tasks past one another, without incurring excessive integration efforts due to a modified schedule.
Whenever effort is invested to solve some problem, the results must be retained, even while it may not be possible to integrate these results immediately; it is easy to forfeit a solution under the pressure of ongoing development and different circumstances.
There should be a simple and clear scheme on how to label effort and deliverables, to connect changes to causes, and to tag versions and CI build artefacts.
The classical (and in a way, naive) approach to organisation and management is to track and control matters extraneously, from the outside, from “above”. Thereby, some people acquire the power to watch and check the actions of other people — which, surprisingly enough, seems attractive both for the watcher and the watched. A prudent solution however places the onerous of coordination onto those people doing the actual work, because these are the only ones able to see the essential nuances required for good judgement.
The success of Git (and similar distributed systems) is based on this insight; instead of locking, rejecting and selectively permitting changes from a system placed above people, people are given the tools to collaborate, and the history emerges as a result of this real-world collaboration. Git-flow attempts to expand on this approach to handle the intricacies involved in software releases. It is essentialy a set of conventions allowing for changes to flow from development to production release, while mitigating the friction caused by conflicts.
Development, features, integration, testing, QA, bugfixes and delivery cover the space between development and delivery of production-ready code. Git-flow translates this into two permanent ongoing branches: the development-branch and the production-branch. The development process is represented by the former, while the latter tracks the history of production-ready code. A release marks a transition between these spheres, and is thus modelled by a release-branch, which originates at development and is finally merged into production. And a bugfix is caused by an issue related to the system in productive use and thus originates from the production branch, is kept separate from development, and is merged back into production after resolving the issue. The production branch itself never carries direct changes, but rather receives tested and approved changes through merging.
The names for all these branches are a matter of convention, yet a final decision on nomenclature
should be made, documented and never changed thereafter. The development branch is often called
develop or integration, or main, while the traditional master is repurposed as
production branch in most cases — which has the benefit of any users unfamiliar with
Git-flow usage will not hesitate to check out master, which happens to be the correct
choice for casual contributions or bugfix investigations.
The field of software development and project management has seen a trend towards an increase in releases and delivery in recent years. Beginning with an effort to gradually automate more and more stages of the delivery process, at some point even the usefulness of marked releases and the adequacy of stable software selections and distributions is called into question. New methods of work organisation were promoted with much creed and dogmatism, at times leading to a false dichotomy to place the “agile trunk based development” in opposition to pretty much any other established form of structured work organisation. The driving force behind this tendency however is a gradual move towards industrialisation within the field of data technology and software development.
In the light of this perception shift, it seems advisable to clarify and disentangle some of the terms, practices and handling patterns related to development, work organisation and delivery of software. A trunk based development style can be used by a seasoned team of experienced developers, working in a well confined environment and within a largely stabilised state of a system. It should not be confused with trunk based releases, where some code from the tip of the development branch is directly published into the production system with the help of a highly automated delivery pipeline. None of these methods is inherently superior; their applicability largely depends on the finer points of the given circumstances.
Decisions in software development can be subject to wider considerations and forces beyond the technical realm. External liabilities may arise once people start to rely on data technology for any purpose or goal of significance. Yet, at the same time, a change in the patterns of usage, a shift of focus or even an improved understanding of the problem domain can lead to the desire for, or even the necessity of deep changes, to a degree that the very heart of the software system must be upended.
The practice of working towards release milestones, to create a vetted software stack and stabilised selections and distributions of code can be seen as a strategy to cope with such a situation and to handle demands surpassing the problem solving capacity of a small and coherent group of people. Establishing such a release will protect the users and enables the emergence of a down-stream ecosystem of connected tools and usages. And it helps to create a safe space for development to focus on resolving tough problems and non-trivial challenges.
As regards to development, Git-flow can accommodate various ways of collaboration and does not favour any specific style of development. Work can be done by a single person, by a tightly knit group of experienced developers, or by an elaborate, structured organisation, with possibly even several teams, including a formalised review and approval process.
In a highly conditioned and uniform environment, where infrastructure can be controlled, mapped and redefined by configuration code (“infrastructure as code”), it is possible to deliver every arbitrary state of code any time into the production environment. This setup works well to the degree that requirements are strict and can be formalised into automated testing. Essentially the purpose and usage of the system as such must be focused, limited and always clear, secondary cross dependencies should be minimised, and the latency of any effect of change should ideally be small. Furthermore, since mistakes will be made nonetheless, it helps if the scope can be limited and the consequences of any defect are not particularly severe.
The effort to establish all required prerequisites (notably the strict formalisation of requirements and the conditioning of the environment) should not be underestimated. However, when either the situation is mostly stable and without the conflicts and tensions related to the evolution of software as detailed above — or, quite to the extreme, when the goal is to move fast and break things, then a formalised release vetting process, or even any kind of maintenance can and should be avoided. Using Git-flow in such a situation would just generate accidental complexity.
When most changes can be achieved with a single step or broken down into an incremental scheme while the overall system remains fully functional during all stages, a well-practised team of experienced developers is able to act without much formalisation, collaboratively, on a single shared development branch. Working in such an environment can be quite rewarding and result in a high degree of productivity, assuming that all participants know their own limits.
Such a trunk centric working style does not necessarily also imply continuous delivery; quite on the contrary, it tends to work best when a well organised additional release and QA process is present as an additional safety net. A collaboration style where work mostly happens on the main development branch can thus be perfectly well combined with Git-flow for release and maintenance aspects.
However, any change that touches the essence of a system — and also any change which is explorative or innovative in nature will inevitably break some other aspects of the setup. It is this very breakage which enables transformation and allows to gain new insights. When other groups of people need to work on other aspects of the system at the same time, they must be shielded from the ripple effect of such a breakage. This can be achieved in various ways:
Feature toggles allow to activate the new functionality at runtime, dynamically, while retaining the former stable state of the system. This works to such a degree that the breakage can be confined to a local area of functionality, while retaining an uncompromised code structure. Otherwise, structures would have to be duplicated, which makes the code hard to understand and difficult to work with. Constantly maintaining a set of feature toggles incurs significant cost and effort, especially when the changes are far reaching.
Feature branches are the only viable solution whenever the overall structure of the code must be broken temporarily, and when a new solution has to be developed often in gradual phases. Any other development work that yet again diverges and occurs at the same time as regular maintenance activity will cause additional effort when it comes to integrating the reworked and altered shape of the system. Care should be taken to re-integrate frequently, at least in one direction, i.e. the new and partially broken code should strive to integrate the ongoing regular development work so that the new form of the code is able to replace the exiting code once it is completed. This can be achieved by performing frequent re-bases.
Another reason to favour development branches is to allow various levels of proficiency and to accept incomplete and immature contributions, which require time to achieve maturity. This situation is very common in Open-Source development communities. A complex integration process is also required for large and intricately structured systems: changes will first be accommodated within a subsystem, which is followed by joining subsystem branches into a new state of integration. Code Review is most effectively used when it is a means of support and collaboration: you can move more confidently when you know that another experienced developer will stand by you. Reviews by gatekeepers tend to be more problematic; which can easily turn into a superficial ritual, or cause a bottleneck and threaten to overload the most experienced developers with gatekeeper work, which is often demanding, time-consuming and prevents them from exercising their more able capabilities. Nevertheless, reviewing and merging should occur on a frequent basis, and integrated with the coding work itself, so that the code on the development branch is already well reviewed and scrutinised, with bold moves supported by automated testing. All the various flavours of working with branches can be combined with Git-flow for the release and maintenance work. To which degree the landing of branches is conducted as a formalised review process is a matter of style in collaboration within the development team and thus beyond the scope of this discussion.
One aspect however is crucial for any successful release (irrespective of the actual methodology): All participants must be aware of the release and its goals. Care must be taken to allow for the overall code base to settle down prior to cutting the actual release. The work of the whole development crew should be synchronised to the degree that everyone is aware of the time frame and the objectives of a pending release. Potentially dangerous changes might either be completed early in the release cycle, or otherwise be held back on feature branches until there is agreement that the risks of landing them are acceptable. Furthermore, developing a significant change on a feature branch, always kept in mergeable state through frequent re-bases provides the additional benefit that it is possible to back out and undo the landing, by applying a squashed version of the branch in reverse, should it turn out that stabilisation seems risky within the given time frame before a pending release.
Well written code does not need much additional comments for explanation, since the code itself tells the story.
Git-flow expands the same idea onto release organisation: Instead of a release planning document and release
tracking meetings and spreadsheets with percentage numbers, just look at the git history (and the tickets)
to see what is going on. The work on the code itself, the changes, are arranged in a way that they tell
the release story. First there is the convergence phase, which is obvious from the landing of feature
branches and lots of smallish tweak and fixup commits. When the code seems stable, the release is cut:
A new release branch is created at that point, and the immediate next commit on development mainline
will bump the version number of the code base. The kind of formalised scheme applied here is what allows
us to forego external “organisation work”: At any time, at maximum one release branch should exist in
the repository, named with a prefix and the version number (e.g. release-4.20 or rel/4.20 or whatever
convention was established).
A version number scheme should be well documented and consistent, but it is essentially arbitrary. It is crucial however that the release tag should use the same number as was embedded into the release branch name, and build artefacts will also be marked with the same number. It is a good idea not to get overly obsessed with version numbers; to achieve that, it can be helpful to have a separate »public« version number, while using a consistent build-number scheme internally. Also, API versioning (and SONAMEs) should be kept entirely separate from official version numbers, because especially for APIs (and really, only for APIs) the Semantic Versioning scheme is well defined and fully deterministic, so that no room is left for bikeshedding. Generally speaking, you do not want to allow considerations of marketing or communication to interfere with your level of API compatibility.
For Git-flow to work well, the (technical) version number should be predictable, because we want to bump the version on development mainline immediately after cutting a release. Furthermore, it should be noted that Git-flow branches will be merged by a well defined scheme, so possible merge conflicts due to conflicting version bumps must be taken into account. A related issue arises from the practice to mark the in-between development stage with a special suffix; such a suffix is typically not visible on the tags in Git, but is embedded into the version definition in-tree, which is picked up by the build system. If a commit is needed to remove this development-version marker prior to starting the release build, then care must be taken to handle possible conflicts which may ensue from the back-merge to the development branch afterwards, because a version bump was also done on the latter directly after forking the release branch. While it is possible to auto-resolve certain kinds of merge conflicts with clever scripting, doing so is tricky and risks auto-resolving and thus possibly discarding actual code conflicts erroneously — conflicts, which actually should have stopped automatic processing and require a manual inspection of the situation might have been auto-optimised away.
The code for a release originates from a stabilised state of the development line; after the fork, developers are free to break the code base again, to work towards the future. At the same time, only bugfixes, stability issues and minor tweaks should be applied to the release branch. Often, global reviews of the overall situation are conducted here, or the builds from the branch are handed over to an external QA team. It is not uncommon for actual full-scale integration testing to be very expensive, often requiring to spin up a dedicated wide-scale integration testing environment, which is connected to external test-networks and other facilities beyond the control of the local development team. Another related aspect is public beta testing. Smaller institutions and Open-Source projects typically do not have the manpower to perform an industrial-strength QA and testing process. Furthermore, when software is used in creative ways, a public beta test is the only way to identify problems of those surprising kinds the developers never could have imagined. Honestly, while you could just release and wait for the systems to blow up, you’d better not try that for software which actually matters.
So the release branch reflects activities beyond the scope of pure development and connected to extended human interactions. However, it is not unusual to uncover serious problems at this stage, leading to further significant development efforts. And while not ideal, in practice some last-minute rushed features and changes can be encountered, based on overly zealous commitments made for the release. Furthermore, the time a release takes to reach maturity is also typically used to prepare examples and documentation — and to augment the tests. During that stage, teams often split up into a part concerned with getting the release out of the door, while another part has already moved on to the next topics of development. Having a dedicated branch and distinct version numbering and CI jobs allows for both parts to proceed without interference.
The release will invariably be at some point in time ready: Final polishing touches made to the release notes, possibly the development (or RC) marker will be removed with one final commit, and then the release code will be merged into the production-branch. Due to the well defined branching scheme, together with the fact that commits will never be done on the production branch directly, this merge will always succeed without conflict. At that point, the release version tag can be set, which often also triggers an automated build-and-delivery pipeline. So the production branch will comprise of only a sequence of merge commits with release version tags, and any commit here will trigger delivery to production.
A naive branching scheme would call it a day here, and maybe keep the release branch around, for later bug fixes. Not so with Git-flow! Because we have completed significant work on the release branch, we must be careful not to loose any findings and achievements. Thus, in the Git-flow scheme, after the actual release is done, also a final back merge to development is performed. This merge often leads to merge conflicts, which must be resolved manually. Doing this is important work, and typically requires to consult with those developers who have already progressed to future work on the development branch. Resolving such conflicts together is an effective means of knowledge exchange, and often spurs an effort to re-implement release fixes in a more robust, future proof way.
These back-merges are the most prominent distinguishing factor of the Git-flow branching scheme. When significant work was done on the release branch, it is prudent even to perform such a merge much sooner, to allow resolving conflicts with ongoing development earlier. Sometimes you’ll even find developers cherry-picking important fixes immediately from the release branch to the development mainline, yet any of these additional integration measures are optional. Because the final merge after release will ensure that no code change can be lost (which is a quite common issue when using a traditional, naive branching scheme, where it is up to the individual developer to care for getting bug fixes also into mainline).
Another distinguishing feature of Git-flow is the fact that the release branch will be deleted afterwards. Only a single release is allowed to be “in flight” at any time, and keeping the release branch around afterwards would serve no real purpose. With Git-flow, the branch structure aims to reflect actual activity, and after the release, there is no release-work left. The released code is available at the tip of the production branch, because it is this code which is now the latest version in active use. And any fixes to that code, would not be considered regular release work, but rather urgent maintenance.
If a failure occurs in a production system, or with code distributed to many users and in use
for real work, then immediate action and quick response is required. Changes done to this code,
however, should be as minimal and risk-free as possible. Git-flow uses a specific branch pattern for such a
»Hotfix« or »Patch« release: it is forked off from the code in production, which is at the tip
of the current production branch, and named with a suitable prefix (patch-4.20.1 or fix/4.20.1
or whatever the actually chosen convention is). The first commit on such a bugfix branch will set
the version number (typically incrementing the last component of the version, which represents a
revision or patch number). After implementing and testing the fix, it is released with a merge
back into the production branch. This merge is tagged with the bugfix version.
And similar to a regular release, a back merge of that patch is then performed, to integrate the change with development mainline. A special situation arises when a bug is discovered during the release preparation phase; if the fix is urgent, a bugfix must be delivered immediately, with the isolated fix only, and a bugfix version number smaller than the release version currently in preparation. However, the back-merge has to go into the release branch in this case, otherwise the release will cause a regression. Later, the back-merge after the release will also cause that fix to be integrated into the main development line.
Similar to releases, bugfix branches will also be deleted when complete — and there can be only a single bugfix branch in active preparation at any one time.
It should be evident by now that Git-flow is not a tool or technology — it is a pattern of behaviour, time-proven in many real-world projects to deal with liabilities, conflicts and constraints related to the release and maintenance of software which actually matters in some way or another. Git-flow is an elegant solution to these problems, since it re-orders the activities related to releases and bugfixes, so that the history of the code itself, as captured by the Git version management, is enough to guide and document the structure of the process.
To start using Git-flow, the only thing that is necessary is to discuss and settle for a clear
and simple naming convention for branches, tags and version numbers. Anything beyond that can
easily be done manually, using just git commands, once you have understood the reasoning
behind that branching scheme.
Yet branching and release activities in accordance with the Git-flow scheme can very well be automated further — which can be very helpful when releases are frequent or when integration with CI/CD tooling is required. Git-flow support has been integrated into various IDEs and tools. Notably there is a set of Unix-style commandline tools available, which can be adapted to specific needs easily.
the original author created a first implementation in 2011,
which however was not maintained beyond 2012
a most widely used fork is the git-flow AVH Edition
the latter is even packaged in Debian and can be installed with apt install git-flow
Git-flow allows for some variability regarding the placement of tags and when to bump version numbers; the latter can be relevant to avoid merge conflicts and for the fine points regarding integration with the build system and continuous integration. Version numbers are often used to control which build artefacts are used for automated testing, are stored and retained in some artefact repository or delivered for publication and installation. Version numbers and notably an additional build number is typically used in collaboration with a QA team, test documentation systems or distribution platforms.
Sometimes, further down-stream activities are chained behind the release tag on the production branch.
A common approach known to work well is to use separate Git repositories and dedicated branches for
such activities. A good example is the
git-buildpackage tooling for Debian,
which uses a dedicated debian branch and a merge commit after each release point. Maintaining a
DEB package is formalised and can be automated to a high degree, yet requires some manual review
and approval steps by a package maintainer, which is responsible for that DEB. Notably this
person is responsible to sign a packaged version with GPG, which allows to prevent many kinds
of tampering, code-injection and supply-chain attacks by powerful entities.
Another example of down-stream activities could be the maintenance of platform-specific additional tweaks, or the long-term support for older versions. Keeping this additional infrastructure in additional Git-repositories helps to reduce complexity and distribute the burden of maintenance work to several people, without breaking the chain of automated and reproducible publication.