Hey everybody,
this is a status update of the GitHub Migration – after the discussions we had in Munich and letting the concept be improved for a while in my head.
UPDATE from 6.08.2015
- please read RFC: The GitHub Move -- Vote Needed! - #18 by sebastian
- please vote at http://doodle.com/9evuwuretertkpqm
UPDATE from 16.07.
(discussion with @christopher, @kdambekalns, @daniellienert, @sebastian)
- We will go for two separate repositories instead of one (longer discussion on that)
- We’ll try it out now!
tl;dr
We think there is no obstacle blocking the migration to GitHub and we should definitely move there, as this will tremendously increase the visibility of the project and ease of contribution.
We propose to migrate to GitHub very shortly after the 2.0 Release, so prepare everything upfront.
Background: One Team, Two Products, Two Repositories
Basically, after the discussions in Munich around mid-May with the team, we wanted to clearly position Flow as being the framework which is the basis for Neos; and we’d like to make it as simple as possible to contribute and to work for us.
We want to have only two big, shared repositories – one for Neos and one for Flow packages – basically for everything the team manages and releases at once.
We want to create read-only subtrees as separate git repositories from the individual packages – this will be what composer/packagist uses and what is being used in production. These packages however contain the full history and all tags. They will live underneath a separate GitHub organization, in order to have them “filtered” by default. This topic is low-risk as it is done by many other projects already.
- the full git history is kept
- git blame will work as expected
- only drawback: all git commits from the past get a new SHA1, because they get rewritten as if they had existed inside another path (this is needed in order for git blame to work).
I have worked on the migration towards the git repositories, on scripts which can be found at GitHub - skurfuerst/neos-github-converter: Helper Scripts aiding in moving to GitHub (mangling our git repositories forth and back) (They have not been updated yet for the shared repository).
There are some issues to solve when doing this, which are listed below.
Challenge 1: Different versioning scheme for Flow and Neos
(solved because we’ll use two repositories instead of one)
Challenge 2: Ability to release Flow patch level releases separately from Neos
(solved because we’ll use two repositories instead of one)
State so far
At this point, we have explained how the shared repository structure should look like, and how we can create individual subtree splits for Packagist.
Next, we’ll describe how the setup for local development should be.
Challenge 3: Not all packages should be installed/active at the same time
(solved because we’ll use two separate repositories instead of one)
TODO: I am still unsure whether we should check out the merged distribution through composer or not. That’s something I don’t really know at all yet. Comments?
Challenge 4: Dependencies of packages
When we do not load e.g. the package TYPO3.Flow through composer anymore, but when it is part of a big, shared repository, by default, the transitive dependencies of this package (e.g. doctrine/orm) won’t be picked up by composer anymore.
I suggest that during the checkout process, we add the transitive dependencies to the main composer.json of the distribution; and place a separate file composer_dev.json in the root of the project, being an “extended” copy of the composer.json. During development, we need then to use: COMPOSER=composer_dev.json composer install.
We cannot directly add the dependencies to the main composer.json, because this would also change the production setup (when you use this directly).
Current Status – Development Workflow
Daniel worked on the infrastructure around Composer and the workflow a developer will use when starting to contribute to Neos. Furthermore he checked lots of other projects like OwnCloud or Symfony for their contribution workflow. Our proposed workflow for Neos/Flow currently is the following:
- You install a Flow package which contains helper commands setting up your environment. Basically, instead of writing lots of documentation which will be outdated soonish, the idea is rather to create a command
- Then, using a command, you can convert your “production” environment to a “Development” environment, checking out the merged git repository instead of the per-package ones. We have some ideas how this might work in practice and will try out which of these works best.
- Furthermore, this command will also create a Fork of the public repositories on GitHub; making sure you are all set up for contribution. It will contain commands to keep this fork up to date, etc.
- Furthermore, there are commands which convert Gerrit Review Requests to Pull Requests on Github – for migration of pending changesets. This is already tested and works.
Daniel’s current state of the package can be found at GitHub - neos/Neos.Contribute: Tools helping contribution to Flow and Neos
Please leave your comments:
- What do you think of the repository structure idea?
- Comments you have to the general approach?
If you agree this is the direction to go, I’ll update the scripts to create this single, shared repository.
Greets, and thanks for reading,
Sebastian
So +1 for automating this!