RFC: The GitHub Move -- Vote Needed!

Hey everybody,

this is a status update of the GitHub Migration – after the discussions we had in Munich and letting the concept be improved for a while in my head.

UPDATE from 6.08.2015

UPDATE from 16.07.
(discussion with @christopher, @kdambekalns, @daniellienert, @sebastian)

  • We will go for two separate repositories instead of one (longer discussion on that)
  • We’ll try it out now!

tl;dr

We think there is no obstacle blocking the migration to GitHub and we should definitely move there, as this will tremendously increase the visibility of the project and ease of contribution.

We propose to migrate to GitHub very shortly after the 2.0 Release, so prepare everything upfront.

Background: One Team, Two Products, Two Repositories

Basically, after the discussions in Munich around mid-May with the team, we wanted to clearly position Flow as being the framework which is the basis for Neos; and we’d like to make it as simple as possible to contribute and to work for us.

We want to have only two big, shared repositories – one for Neos and one for Flow packages – basically for everything the team manages and releases at once.

We want to create read-only subtrees as separate git repositories from the individual packages – this will be what composer/packagist uses and what is being used in production. These packages however contain the full history and all tags. They will live underneath a separate GitHub organization, in order to have them “filtered” by default. This topic is low-risk as it is done by many other projects already.

  • the full git history is kept
  • git blame will work as expected
  • only drawback: all git commits from the past get a new SHA1, because they get rewritten as if they had existed inside another path (this is needed in order for git blame to work).

I have worked on the migration towards the git repositories, on scripts which can be found at https://github.com/skurfuerst/neos-github-converter (They have not been updated yet for the shared repository).

There are some issues to solve when doing this, which are listed below.

Challenge 1: Different versioning scheme for Flow and Neos

(solved because we’ll use two repositories instead of one)

Challenge 2: Ability to release Flow patch level releases separately from Neos

(solved because we’ll use two repositories instead of one)

State so far

At this point, we have explained how the shared repository structure should look like, and how we can create individual subtree splits for Packagist.

Next, we’ll describe how the setup for local development should be.

Challenge 3: Not all packages should be installed/active at the same time

(solved because we’ll use two separate repositories instead of one)

TODO: I am still unsure whether we should check out the merged distribution through composer or not. That’s something I don’t really know at all yet. Comments?

Challenge 4: Dependencies of packages

When we do not load e.g. the package TYPO3.Flow through composer anymore, but when it is part of a big, shared repository, by default, the transitive dependencies of this package (e.g. doctrine/orm) won’t be picked up by composer anymore.

I suggest that during the checkout process, we add the transitive dependencies to the main composer.json of the distribution; and place a separate file composer_dev.json in the root of the project, being an “extended” copy of the composer.json. During development, we need then to use: COMPOSER=composer_dev.json composer install.

We cannot directly add the dependencies to the main composer.json, because this would also change the production setup (when you use this directly).

Current Status – Development Workflow

Daniel worked on the infrastructure around Composer and the workflow a developer will use when starting to contribute to Neos. Furthermore he checked lots of other projects like OwnCloud or Symfony for their contribution workflow. Our proposed workflow for Neos/Flow currently is the following:

  • You install a Flow package which contains helper commands setting up your environment. Basically, instead of writing lots of documentation which will be outdated soonish, the idea is rather to create a command
  • Then, using a command, you can convert your “production” environment to a “Development” environment, checking out the merged git repository instead of the per-package ones. We have some ideas how this might work in practice and will try out which of these works best.
  • Furthermore, this command will also create a Fork of the public repositories on GitHub; making sure you are all set up for contribution. It will contain commands to keep this fork up to date, etc.
  • Furthermore, there are commands which convert Gerrit Review Requests to Pull Requests on Github – for migration of pending changesets. This is already tested and works.

Daniel’s current state of the package can be found at https://github.com/daniellienert/Neos.Contribute

Please leave your comments:

  • What do you think of the repository structure idea?
  • Comments you have to the general approach?

If you agree this is the direction to go, I’ll update the scripts to create this single, shared repository.

Greets, and thanks for reading,
Sebastian

2 Likes

Mmmh, that might cause problems in the IDE and especially with XDEBUG. There’s always a lot of trouble when mixing symlinks with non-symlinked files there (at least I always had problems with TYPO3 and symlinked typo3_src).

So just having all the packages inactive is no option? In the end it should be a matter of the package manager to not load Neos packages, and that can be easily scripted to update your PackageStates.php.

This is too bad but the proposed solution might be our only choice. Maybe we could auto-generate the list of dependencies by putting together all the (active) inner package dependencies. Because maintaining two lists hardly works.

Hey Christopher,

We could of course also do this. I currently often do not deactivate packages at all; but if that works reliably, sure let’s do that!

Yeah, exactly, I thought we’ll auto-generate this!

All the best, thanks for your comment,
Sebastian

IMHO the ideas regarding the activation of packages are not practicable yet as far as I see it. Changing the global composer.json just because we do not install our packages via composer is a no go. Creating a solution to resolve inner dependencies is just another level of complexity and source of errors then.
I would rather like to find a more integrated solution.

How about using composer to install the big repository?

I could imagine the big repository having a composer.json that says it replaces all the packages inside. Our installer could install this into a directory that is not picked up by our package management. Then do symlinks or whatever from there. Just a quick idea, but not using composer won’t work for me.

Hey everybody,

In order to make the GitHub move a reality, I really need help from more people trying out different approaches, if the solution I suggested is not the one to chose. I played back-and-forth lots of ideas also on how composer shall/might be used, but installing the big repository through composer also has large drawbacks – especially because the full transitive dependencies are then loaded even if you’re only running e.g. Flow instead of Neos.

We can ofcause arrange a hangout where we can discuss these matters.

Greets, Sebastian

Would love to discuss this to avoid any “homebrewn” solution to this. I think we will extremly complicate matters if we go this route (moving dependencies up, collecting dependencies etc.)

I would rather come to the conclusion that we have two repositories (one for Flow and one for Neos). Also loading the transitive dependencies is a non issue for me, it will happen only for developers and you can still deactivate the packages for now (and in theory those dependencies should do no harm).

Hey Christian,

thanks for your feedback… I still favor going single-repository, as we are one team with one “flagship” product - and it is one place where everything is, without possible segmentation. Actually, I think collecting dependencies etc would be quite easy as it is all just simple JSON files I guess.

Anyways - I had another idea: Could we maybe create a composer plugin (I think you played around with this?) – which handles the “one-repository-developer-repo” correctly and adds the dependencies to the composer internals correctly? Just as an idea how we could continue using composer in a (hopefully clean) way and on the other hand keep a single git repository.

All the best,
Sebastian

1 Like

Good idea, lets check that.

I just realize that this would probably be a task that jenkins could solve for us, isn’t it?

Basically the task is: collect all composer.json in the repository, collect dependencies and dev-dependencies, remove packages that are actually in the repository, create composer.json for the big repository listing all dependencies of the sub packages. Ideally it would check if the result is the same as before and only if not create a commit.

Yes, sure, I wouldn’t do it by hand :slight_smile: So +1 for automating this!

All the best,
Sebastian

I rather meant that this means we do not need to touch the users (the distribution) composer.json.

Hey, as I am doing a big rewrite of the PackageManager currently I found an issue with Flow that will definitely make this harder.

As we wanted to allow setups like distribution + site package we cannot rely on composer to get a list of installed packages, so we try to find them ourselfs. This is done by traversing Packages/ and finding composer.json files which will then usually be treated as packages. Now as charming as that is, it will break with a big repository having one composer.json for the whole repository and separate packages with their package composer.json inside. Only the first composer.json (of the big repository) would be picked up as a package, definitely breaking everything as stuff like Package.php and resource path resolution (and more) won’t work anymore.

Now this logic could be changed of course and I would assume that a vanilla Neos install won’t have any problems, but I am not sure if due to this approach there are installs out there that might be VERY broken.

We could introduce some configuration in the extra part of the composer.json that tells the package manager to basically ignore the big repository composer.json and dig deeper. That should be backwards compatible. If you think that could be a good approach I can add it to my refactoring.

Hey everybody,

just wanted you to update on this topic that Christian and myself had a phone call yesterday on the approach, and it seems the following will work out nicely:

So we’ll experiment into this direction and try it out!

All the best,
Sebastian

2 Likes

Hey everybody,

we just had a discussion together with @kdambekalns, @christopher, @daniellienert where we went in-depth whether we want to go the two-repositories or single-Repositories route. After quite some thinking forth and back, we are again proposing to split to two repositories; namely because of the following reasons:

  • it is two separate products
  • we don’t have to deal with long, combined version numbers (for flow/neos)
  • easier to do separate releases
  • so far, we have very rare dependencies between Neos and Flow anymore
  • easier to handle just for Flow-developers (who do not use Neos)

We discussed some details of the move; and we’ll try it out soon again!

All the best,
Sebastian

1 Like

Too bad I missed the discussion, would have loved to join you.
Could you add some more details on what lead you to that decision?

Hey Bastian,

updated the reply above yours; maybe it is more clear now :slight_smile:

Here’s one more benefit to maintaining two separate repositories for Neos and Flow: In doing so, we will be setting the standard for other projects based on Flow.

When an agency has a large project that requires multiple packages to achieve the appropriate level of cohesion (and, therefore, maintainability), they should be able to use the same setup that we use: Keeping all of the packages in one repository.

Maybe we should keep that in mind in developing our composer plugin. Other projects will want to have multiple packages in one repository, let’s make that easy to do.

2 Likes

Status Update – Final Migration Proposal

tl;dr: Please read the Git Repository Structure Proposal and vote at Please vote at http://doodle.com/9evuwuretertkpqm

Hey everybody,

I just took the time to re-generate the shared Neos and Flow repositories a (from my side) final time:

Shared Repositories

These should be located inside github.com/neos organization.

Read-only-subsplits per-package will be located inside github.com/neos-readonly organization

Separate Repositories

These should be located inside github.com/neos organization

The following packages would be versioned as separate Git repositories:

  • TYPO3.Party
    • versioned roughly like Flow; but Flow is already independent of Party
  • TYPO3.Form
    • versioned completely separately; 2.0 already released
  • TYPO3.Imagine
    • versioned completely separately; 2.0 already released
  • TYPO3.Neos.SEO
    • versioned completely separately; 1.0.5 most up-to-date version
  • TYPO3.Setup
    • versioned completely separately; 1.0.3 most up-to-date version
  • TYPO3.Twitter.Bootstrap
    • versioned completely separately; 2.2.0 most up-to-date version

Comments

I personally have discovered https://github.com/pulls – which can show pull requests across repository boundaries, i.e. inside one GitHub organization. I think this is a totally fine way to have a great overview about which pull requests are in which package(group).

Voting

Please vote at http://doodle.com/9evuwuretertkpqm if you like this proposal or not!

I think it is fine that everybody in the community is allowed to vote here to express his/her opinion and concerns. “Binding” are the votes of the core team, though.

All the best,
Sebastian

2 Likes

What about the demo site? Should this be delivered with the Neos bundle or as a stand alone?

Two things:

  • TYPO3.Neos.GoogleAnalytics is missing as an independent package
  • Latest TYPO3.Setup version is 2.0.0