Daniel Siepmann - Coding is Art

Blog Post

Create Git mono repository

Published: , Updated:

Topics: git

Introduction

Some years ago we got a new customer with a lot of similar TYPO3 installations. All had their own Git repository. We decided to migrate this setup to composer and split the existing TYPO3 extensions out of the repositories, into their own. This year we noticed that was a bad idea, and we will migrate all installation into a single Git repository, together with all extensions.

I'll try to explain the initial idea, why we think this was a bad idea, and how we create the new mono repository.

Initial setup

I'll first explain the original setup we encountered when we started to work with the customer.

The customer has multiple clients, each with his own TYPO3 installation. Each installation had his own git repository. Most installations had the same set of custom extensions, those extensions were part of each installation git repository.

The bad thing was: Changes to an extension for one client, needed to be transferred to all git repositories for each client. That made maintenance a nightmare. Also setting up testing, linting and continuous integration (=CI) was difficult, as each client repository needed the same setup, slightly adjusted to the set of extensions.

Split for composer

As we already were used to Composer, we thought we will use the most up to date client repository and do a git subtree split to generate individual git repositories for each extension. That would erase all the issues. One could require a specific version of an extension for a client. Each extension could have its own setup within CI regarding testing.

You can search the Internet on how to do the subtree split, I'll not explain that here.

Learnings

Two years later, we were suffering from the bad idea. It turned out that our setup didn't fit to the chosen solution. Our workflow in order to release a small hotfix for a client most often involved the following steps:

  1. Make adjustments in local development environment.
  2. Create pull requests for the adjustments.
    Most often more than one extension is involved, leading to pull requests in at least two extensions and the client git.
  3. Merge pull requests and create releases.
    Most often multiple extensions need to be released, resulting in concrete version requirements between those extensions and the client.

That was very time-consuming. We learned that our real workflow should be:

  1. Make adjustments in local development environment.
  2. Create a single pull request for the changes.
  3. Merge and release the change.

Beside those issues, we still had many clients running old versions of our setup. Keeping migration wizards for all possible setups compatible turned out to be hard with the old TYPO3 version we were running.

Therefore, we decided to migrate our setup into a mono repository. Our goal was a single git repository which contains everything we need, all extensions, a single TYPO3 installation and each client setup.

Developers could switch the active client, and the active client would be defined during building the setup for deployment. A general rule for our work with this client is: Keep changes to a minimum, but always create a better environment then you found. As this is a legacy project existing for several years, this is very important. Without this rule we would constantly being changing stuff and no one would be able to work anymore. Or we would keep the legacy stuff and would not be able to add new features or keep fixing bugs.

Creating mono repository

One of the great benefits of Git is the flexibility. Git is not in your way, it is easy to use, but yet powerful. One power is the possibility to merge histories of unrelated repositories. That way one can create a new repository while keeping history of each file from foreign repositories.

That's what we did, we merged all client repositories and extension repositories into a single new mono repository. The new folder setup looks like this:

├── composer.json
├── composer.lock
├── dependency-checker.json
├── fileadmin
├── .git
├── .gitattributes
├── .gitignore
├── index.php -> vendor/typo3/cms/index.php
├── local_packages
├── patches
├── .phan
├── phpcs.xml.dist
├── phpunit.functional.xml.dist
├── phpunit.xml.dist
├── typo3 -> vendor/typo3/cms/typo3
├── typo3conf
├── typo3temp
├── uploads
└── vendor

Composer setup was simplified as well, as we didn't have to set up Satis, Packagist or VCS as foreign sources. The CI was also simplified as there is only a single setup to test. We hope this will suite us longer than two years and reduce the daily work. The main goal is to focus on fixing issues and delivering features faster.

We created a small script, so we can create the mono repository from current state.

The first task is to create the new repository:

mkdir -p monorepo
cd monorepo
git init && git commit -m "INIT" --allow-empty

Next we will add the extensions from their repositories:

repos=""
for repo in $repos; do
    remoteRepoPath="ssh://domain/pathtoextensions/${repo}.git"
    git remote add $repo $remoteRepoPath
    git fetch $repo master
    git merge -s ours --no-commit --allow-unrelated-histories $repo/master
    git read-tree --prefix=local_packages/$repo -u $repo/master
    git commit -m "Imported $repo as a subtree."
    git remote remove $repo
done

The repos variable would contain a whitespace separated list of the repository names, which correspond to the extension keys.

Once done, we repeat the same for each client git repository:

repos=""
for repo in $repos; do
    remoteRepoPath="ssh://domain/pathtoclients/${repo}.git"
    git remote add $repo $remoteRepoPath
    git fetch $repo master
    git merge -s ours --no-commit --allow-unrelated-histories $repo/master
    git read-tree --prefix=local_packages/$repo -u $repo/master
    git mv "local_packages/$repo/local_packages/$repo" "local_packages/client_$repo"
    mkdir -p "local_packages/client_$repo/Configuration/AdditionalConfiguration/"
    git mv "local_packages/$repo/typo3conf/AdditionalConfiguration_dev.php" "local_packages/client_$repo/Configuration/AdditionalConfiguration/Development.php"
    git mv "local_packages/$repo/typo3conf/AdditionalConfiguration_live.php" "local_packages/client_$repo/Configuration/AdditionalConfiguration/Production.php"
    rm -rf "local_packages/$repo"
    git add .
    git commit -m "Imported $repo as a subtree."
    git remote remove $repo
done

Each client had two configuration files which are placed in a new location. We will add a small class which will load the expected files. As most of the time, we try to keep changes to a minimum and easy to understand for everyone.

Last but not least we do the same for the most up to date client, where we also use git mv for some more files like composer.json. Once done, we apply some patches and composer updates to get a working and up to date system. Those patches involve path adjustments and adding files for CI, e.g. phpunit.xml.dist.

Migrate existing branches

Especially in larger teams there might be feature branches. Until now, we have seen how to integrate the main branch, still called master in the examples. There is also a way to integrate further branches.

The following will work beginning with Git v2.22.0.

git switch "master"
git remote add $repo $remoteRepoPath
git fetch $repo $branch
git switch -t $repo/$branch
git -c merge.directoryRenames=true -c merge.renamelimit=3000 rebase -m --strategy recursive master
git remote remove $repo

First switch to the master branch. Then add the remote repository again, fetch the branch, switch to it, and rebase it considering directory renames. Once done, remove the remote repository to keep things clean.

You might need to tweak the -c merge.renamelimit=3000 option. Also replace $repo and $remoteRepoPath as well as $branch with expected values. We created a small bash script and call it for each branch during migration.

Acknowledgements

Thanks to our customer reuter.de for the friendly and cooperative environment. They are still looking for further developers, check out their job website at jobs.reuter.de.

Also thanks to Codappix for employing me and allowing me to work with such customers.