Bug 11792 – Investigate migrating to a meta repo

Currently our github projects assume they live in the same directory and access each other by using ../projname/stuff. We should formalize this by migrating everything to a meta repo and making them as submodules. The migration should be minimally disruptive and preserve history etc.

I think it's pretty simple. - Create a 'DPL' or 'dlang' repo at D-Programming-Language - Include all submodule - Write a README.md on how to checkout and git init submodule everything - Write github hooks that update the meta repo branches for every submodule commit. - Start to populate the meta repo with meta Makefiles :).

That's simple to setup but tedious to use. I made some tests in https://github.com/Geod24/MetaD This is my feedback: 1) Starting from 1.8.2, git can track a branch, not only a commit. When you checkout it's still set to a specific commit though, you need to pass the `--remote` option to `git submodule update` So I think we could make the repo point to the latest release, and our contributor just need to keep in mind to passe `--remote`. This design void the necessity of a hook. 2) I was wondering what the workflow would be with submodules. I'm currently still testing and updating the README accordingly. The most annoyance so far is that submodule HEAD gets detached quite easily. The other problem is that your repository gets "dirty" by using --remote. I think we should try to lower the need of git submodule commands, as using them is tricky and often leads to confusion, like commiting in the wrong order. We should also take care not to accept a PR with a modif to .gitmodules unless it's a stable version update. 3) I'll take a look at the merging strategy soon. By default submodules are just binary object, but git provide a way to merge/rebase them too as part of the update process. I'll keep experimenting with it, any comment / example of your normal workflow is very welcome.

This sounds similar to my D.git experiment: http://forum.dlang.org/post/[email protected] The repository is automatically generated from the component repos' histories and periodically updated (currently hourly). > - Write a README.md on how to checkout and git init submodule everything "git clone --recursive" will fetch all submodules as well, so there's no need to use "git submodule init". > 1) Starting from 1.8.2, git can track a branch, not only a commit. That would preclude importing history, and would make it unusable for regression testing. > We should also take care not to accept a PR with a modif to .gitmodules > unless it's a stable version update. Why would anyone send a pull request to the meta repository, unless it's e.g. to add/fix makefiles?

(In reply to comment #3) > This sounds similar to my D.git experiment: > > http://forum.dlang.org/post/[email protected] > > The repository is automatically generated from the component repos' histories > and periodically updated (currently hourly). > I am getting 404 on the mentioned repository (but D-dot-git works). > > 1) Starting from 1.8.2, git can track a branch, not only a commit. > > That would preclude importing history, and would make it unusable for > regression testing. > Submodules are not really meant for importing history. Actually, looking at the man page kind of advise you against doing this: https://www.kernel.org/pub/software/scm/git/docs/git-submodule.html (2nd paragraph of the description). > > We should also take care not to accept a PR with a modif to .gitmodules > > unless it's a stable version update. > > Why would anyone send a pull request to the meta repository, unless it's e.g. > to add/fix makefiles? In my approach, that would be to update the pointed commit (you still have a pointed commit with --remote). Now, how do we handle testing PR that require Makefiles changes ? If I do a PR to Phobos, and one to the superproject which modify Makefiles, how does the auto-tester will possibly handle it ?

(In reply to comment #4) > I am getting 404 on the mentioned repository (but D-dot-git works). As mentioned later in the thread, the repository was moved to Bitbucket to prevent GitHub from "helpfully" generating mention notifications: https://bitbucket.org/cybershadow/d/commits > Submodules are not really meant for importing history. Actually, looking at the > man page kind of advise you against doing this: > https://www.kernel.org/pub/software/scm/git/docs/git-submodule.html > (2nd paragraph of the description). I don't see how this applies to a meta-repository. It is also not possible to meaningfully merge the history of several projects with a non-linear history. > In my approach, that would be to update the pointed commit (you still have a > pointed commit with --remote). In what circumstances would that happen? > Now, how do we handle testing PR that require Makefiles changes ? If I do a PR > to Phobos, and one to the superproject which modify Makefiles, how does the > auto-tester will possibly handle it ? This problem is no different from any changes that require patches on multiple components at once. It is currently not handled.

Another attempt via https://github.com/Dicebot/TestDlangAggregated (also http://forum.dlang.org/post/[email protected]) Outlining some differences: 1) use of git 1.8.2+ feature of bounding submodule to remote branches for simple updating to HEAD 2) providing top-level script (dlang.d) for managing typical tasks without much investigation 3) installing toolchain into single ./bin folder inside same repo which can be added to system PATH to immediately start using new stuff 4) stated goal for moving meta-tools from tools repo to meta-repo and replacing makefiles with D scripts for better cross-platform development experience

On the forum, I posted ([1]): > I would like to add that, however, it might be worth considering > moving everything to a single repository at the same time as the > switch to DDMD. DDMD by itself is a big change, so aggregating > other changes with big wolkflow impact (but net benefit in the > long run) would make sense. This would also be a good time to replace makefiles with a D build tool. Here are some thoughts on how this can be done: - Create the repo with an experimental status ahead of time, so that it can be extensively tested. - Decide on a date on which the switch will occur. - Add comment to all open pull requests some time before, that the changes will need to be rebased on the new repository. - Use git-subtree to add the existing repositories to the new unified repository. - Add a commit to the existing repositories which replaces the README (and possibly all content) with a link to the new repository. - Move the old repositories to a new organization (e.g. D-Programming-Language-Old) so they don't clutter D-P-L organization's project list. TODO: Check if GitHub creates redirects (we need redirects). Although the change is big, the gain is also big: we will have one single repository using a self-hosted compiler and build system. By doing these all together, we will need to break things only once. [1]: http://forum.dlang.org/post/[email protected]

I am of very bad opinion of subtree approach (single "physical" repo). It would destroy natural separation of areas of interest for pull request reviewers (need to add "dmd"/"phobos" labels manually for each PR), make commit history so noisy it is impossible to follow for few specific projects, make impossible to finely tune team permissions to different repos. Only benefit is reducing amount of repositories under D-Programming-Language organization. I don't see it a problem that needs solution, not at this cost at least.

(In reply to Dicebot from comment #8) > It > would destroy natural separation of areas of interest for pull request > reviewers (need to add "dmd"/"phobos" labels manually for each PR), It would not be hard to write a bot to apply these labels automatically. > make > commit history so noisy it is impossible to follow for few specific > projects, It is easy in git to see the log of a specific directory, "git log phobos" or "gitk phobos". > make impossible to finely tune team permissions to different repos. Valid argument, but I'm not sure how important it is. > Only benefit is reducing amount of repositories under D-Programming-Language > organization. I don't see it a problem that needs solution, not at this cost > at least. There are more benefits: - Code which affects multiple projects can be done as one commit. - Makefiles and build scripts no longer need to assume (but allow overriding) location of other subcomponents - their location will be guaranteed. This means they can be simplified. - It will be easy to build D at a specific point in its development. - It will no longer be necessary to use specialized tools such as Digger to bisect regressions.

> It is easy in git to see the log of a specific directory, "git log phobos" or "gitk phobos" I was referring to following on GitHub, not exploring plain git history. For example I am currently subscribed to all changes in Phobos but not in DMD and druntime - combined together it is simply too much stuff to me to follow. > - Makefiles and build scripts no longer need to assume (but allow overriding) location of other subcomponents - their location will be guaranteed. This means they can be simplified. This is addressed by my proposed scheme (with submodules) too. As for the other points - you propose to improve experience for very small group of developers (those who need to do history bisection and/or do changes on multiple repos at once) at the cost of harming experience for more casual one (anyone who does D development / reviewing or simply wants to know what happens in more details). I think having specialized tool (Digger) for specialized tasks is more appropriate that having specialized tool (tagging bot + custom activity feed filters) for common tasks.

Just ran into this issue. I'm using this: https://github.com/lionello/d-meta

THIS ISSUE HAS BEEN MOVED TO GITHUB https://github.com/dlang/dmd/issues/18742 DO NOT COMMENT HERE ANYMORE, NOBODY WILL SEE IT, THIS ISSUE HAS BEEN MOVED TO GITHUB