Git migration status update

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Git migration status update

George (Foswiki)
Hi Foswikians,

The current status: The migration is still being worked on,  Subversion is still readonly with some exceptions for the conversion process and some outstanding work that was pending check-in.  Two key points:
  • It is still critical that nobody pushes to the github repositories.
  • I've encountered some issues that may change the git repository layout - in particular default extensions and submodules.

To date:

  • A complete audit of the code in subversion compared to the github was completed.  A sha1sum of all svn files was compared to both the monolithic git repo, and the "repo-per-extension" repositories.   A few differences were identified and resolved.
    • Modules and Topics using $Rev$ and $Date$ keyword expansions resulted in sha1 differences.
    • Several recent extension repositories had been incorrectly cloned and were corrected.
  • is now rebuilt and running from a github checkout.
  • The web, email and irc hooks were installed in all 610 repositories.   Changes on github result in an email and irc msg .  And the web hook invokes a rest handler on  It logs the event and will eventually update the tasks.
  • A plugin was written to map from a svn rev to one or more git commit URLs.

Now for the problem.   With the proposed repository layout, we lose history on any files that were ever moved from core to an extension.   I have a solution, but if anyone has a better way, please let me know.  Here are the details:

The "git log" command operates differently from subversion when a file is moved to a different location.

  • With svn, the history follows the file.
  • With git, a new history starts at the new location, and the old history remains at the original (deleted) location.
    • Some commands, such as "git blame", automatically follow the history to the old location and it appears seamless
    • "git log" requires the --follow option to report the changes committed at the old location.

The big problem in the conversion from the monolithic repo to the per-extension repos.  The history is in the original location, and is lost.  For individual extensions, this generally isn't an issue, but for the core+default, where features are restructured from core to extensions, this is a huge issue.  The best way to explain is an example.

Version 1.2 makes the Store subsystem pluggable.  The Store functions were all moved from core into the RCSStoreContrib.
  • "git log core/lib/Foswiki/Store/"   reports file not found   (it's moved to the contrib)
  • "git log -- core/lib/Foswiki/Store/"  will show the history of the original file up to the move.
  • "git log RCSStoreContrib/lib/Foswiki/Store/"  shows a single commit the file was moved.
  • "git log --follow RCSStoreContrib/lib/Foswiki/Store/"  shows the complete history

However,   when we split RCSStoreContrib out into it's own repository,   the history is lost.  --follow and git blame have no "old location" to access.

During the git conversion, history is lost whenever a file was moved from core to an extension,  or when a file was moved between two extensions. 

I believe that this means we cannot use the "repo per extension" strategy for the core+default extensions.  We'll need to:

  • have a repository that is core+default.
  • continue with per-extension repositories for non-default extensions, with possible exceptions

This would be the revised layout of github repositories:

  • FoswikiDistribution
    • core
    • CommentPlugin
    • ... <all default extensions>
  • AccessStatsPlugin
  • ActionTrackerPlugin
  • ... <repository per extension>

If there are families of related extensions where files have historically moved between extensions and revision history is important,  then it might also make sense to group them into a single repository.  But that does complicate the pseudo-install process.   It would then need a map to relate extension to repository rather than just by extension name.

I apologize for the delay and confusion here, but some things were just not discovered until the full audit.  I was hoping we would be all done by now. Unfortunately I'm going to be tied up the next few days so this isn't going to be completed this weekend.    In the meantime if anyone has any ideas on how to find all files that were ever moved between core + extension or between extensions,  and to reattach the lost history when the extension is split to a different repository, that would be a great help.


George Clark


Foswiki-discuss mailing list
[hidden email]