systemtap Fedora dist-git

[Roland McGrath <roland at redhat dot com>]

This is all about maintaining the systemtap packages in Fedora.  If
you aren't interested in helping with Fedora maintenance, then there
is nothing about systemtap here to follow.

This week has been some mayhem in Fedoraland, converting from dist-cvs
to dist-git.  You should follow devel@lists.fedoraproject.org for general
details about the whole new world order for package maintainers.

You'll want to start with:

	# yum upgrade --enablerepo=updates-testing fedora-packager

Versions before 0.5.1.0 don't work.  (I presume that by next week
sometime Fedora updates proper will have a good version.)

If you have an old CVS checkout from /cvs/pkgs for systemtap, then go
look at it.  The CVS servers are still there for read-only use, so you
can do a cvs update and make sure there is nothing in your old checkout
that you wanted to keep.

Start in the new world with:

	$ fedpkg clone systemtap

This is actually just doing:

	$ git clone ssh://pkgs.fedoraproject.org/systemtap

but you don't have to remember the URL.

The new scheme is the same as the old scheme in essence: the .spec file,
any patch files, and a "sources" file (md5sum list) are in this
repository.  Things you previously did with make targets are now done
with fedpkg subcommands, mostly of the same names as teh old make
targets.  It's all very much still being worked on, so go talk to Fedora
folks on mailing lists or IRC (#fedora-admin) if you hit problems.
'fedpkg help' will get you started, and each subcommand has a --help.

Unlike the old dist-cvs scheme, each package (i.e. systemtap) has just
one actual repository and the different maintenance streams are on
normal proper git branches.  This is what I want to tell you about in a
little detail.

	$ git branch -r -v
	  origin/HEAD       -> origin/master
	  origin/f10/master 0b8a1ce dist-git conversion
	  origin/f11/master 0e4ee2b dist-git conversion
	  origin/f12/master 3031bf9 Merge branch 'f13/master'
	  origin/f13/master 3031bf9 Merge branch 'f13/master'
	  origin/f14/master 3031bf9 Merge branch 'f13/master'
	  origin/f7/master  724adb9 dist-git conversion
	  origin/f8/master  79889bd dist-git conversion
	  origin/f9/master  2a51003 dist-git conversion
	  origin/master     3031bf9 Merge branch 'f13/master'

The "master" branch is for rawhide.  The fN/master branches are for the
primary maintenance for each Fedora release stream.  The reason they are
called fN/master instead of just fN is that some package maintenance
teams might have other branches called fN/something going for new side
versions meant for fN being worked on before they are ready to put those
in the main fN-updates stream.  I don't think that sort of scenario will
ever come up for us with systemtap, so just know the branch naming
convention but don't worry about it.

We have generally maintained Fedora's systemtap packages in lock-step,
where we just update each Fedora live stream to each new "upstream"
release we make all at the same time, so all the live branches are
identical.  Heretofore that has been annoyingly informal with the
dist-cvs non-branch multi-stream scheme.  Now we can just use normal git
method to do this, and it will be much, much nicer.  But that does
require some care and understanding of "normal" git for branch juggling.

After the conversion, we had separate identical trees for rawhide and
f13, but the branches did not share any history in git's eyes.
Likewise, we had an f12 branch that had drifted only very slightly (with
one tiny .spec change) but git thought it entirely separate.

I've unified the branches with some appropriate git merge commands.
That gets the result you see above, where the f1[234]/master branches
and master are all actually the very same commit id.

Here's what I did:

	$ fedpkg clone systemtap
	$ cd systemtap

That's just getting started.

	$ git checkout f12/master

Done the first time (when no local f12/master branch exists), this is
the same as 'git checkout -b f12/master origin/f12/master', and --track
is implicit there by default git settings.  That means you get a local
f12/master branch that matches the f12/master on the server
(i.e. "origin"), and later plain 'git pull' while on this local branch
will merge in the server's f12/master branch to yours.  Now that this
branch exists locally, 'git checkout f12/master' later will simply
switch your checkout to that branch.

	$ git diff f12/master..master

This shows that they were not identical trees to begin with.
The difference was the 1.3-1 -> 1.3-2 spec change:
	+%ifarch ppc # crash is not available
	+%{!?with_crash: %global with_crash 0}
	+%else
	 %{!?with_crash: %global with_crash 1}
	+%endif
et al.  I checked the crash.spec from f13/rawhide and verified that we
really do want this change not just for f12, but for later ones too.  So
here we have an example where in the old system we changed f12 only but
really wanted to change f13 and rawhide too, but didn't because the old
system made it an unnatural hassle requiring extra thought and legwork.

So, we really want the f12/master version of things to be what we have
on master and f13/master (and now f14/master, just branched yesterday).

	$ git checkout master

This just switches me back to the local 'master' branch, which already
existed by default from 'git clone' (or 'fedpkg clone').

	$ git diff master..origin/f13/master

This shows that the trees are actually identical for the f13 and rawhide
branches already.  So there is nothing concrete to worry about, we just
want to get git to understand their relationship after the conversion.

	$ git merge origin/f13/master

This merges f13/master into rawhide.  The trees are identical already,
so this just works automatically.  Now git knows that the master and
f13/master branches share history, so future merges will work with the
normal intelligent behavior for nonidentical trees.

	$ git merge f12/master

Now we're merging the f12/master branch into the master branch.  Since
git didn't think they shared any history, it shows that one change as a
merge conflict, even though a similar case in the future would just
merge automatically.  I just fixed this up by hand, since that is the
safest way to be sure you know what you are getting before you get it.
That is, editting systemtap.spec, 'git add -u', 'git commit'.  (Here
when we know it's what we want, we could have also done 'git merge -s
recursive -Xtheirs' and no manual steps required.  But that is dangerous
magic best left to gurus--I might be a guru, and even I didn't actually
use it.)

The f11 and earlier branches are actually different (never got 1.3),
so I didn't touch those.  These branches are effectively dead, since
we are not doing package updates for those Fedora releases any more.

	$ git checkout f12/master
	$ git merge master

Now we switch back to the f12 branch and merge master back into it.
This is a no-op on the tree, but it makes git realize the shared
history.  In fact, it is a "fast-forward" merge, meaning that it just
resets the f12/master head to the new master head, because git noticed
that the old f12/master head was already fully merged in here.

	$ git checkout f13/master

Now we're getting a local f13/master branch, which we hadn't yet
bothered with.  (We as well could have done this earlier, and then
used f13/master in place of origin/f13/master above.)

	$ git merge master

Now we merge master into f13/master.  Just like with f12/master, it
notices that we're fully merged with the old f13/master head (thanks to
'git merge origin/f13/master' above), so this is a "fast-forward" too.

	$ git checkout f14/master
	$ git merge master

Finally, we remember that there is now an f14/master branch (as of
today, rawhide is technically targetting f15) and do the same there.
Since the old f14/master head was the old master head, it's fully merged
up and we get another fast-forward.

	$ git branch -v
	  f12/master 3031bf9 Merge branch 'f13/master'
	  f13/master 3031bf9 Merge branch 'f13/master'
	  f14/master 3031bf9 Merge branch 'f13/master'
	* master     3031bf9 Merge branch 'f13/master'

Now we're at the state seen above.  All the branches are identical.

	$ git push

Now that's on the server too, for all to share.  (If you look at the git
history, you might see signs that some of the merges actually went in
different directions from what I've listed here.  I did it all piecemeal
before writing this up, and I modified the description to match what I
might have done if I'd thought it all through before I did anything.
But the material effect of the final state of all the branches is the same.)

Now that we've gotten to a good state after the conversion, how will
"normal" procedures look in the future?  Say, when 1.4 is ready.

The first thing to remember is that there might be other changes to the
package repository made by Fedora folks.  For anything substantive, they
should presumably have talked to a systemtap package maintainer so it
won't be a surprise.  But it's likely there will be things like a bump
of the Release: number for a rebuild.  So let's imagine this has
happened on f14/master, e.g. 2 changed to 3 and a %changelog entry
that says "- rebuilt for blah blah".

Now f14/master is not the same as master (or f13/master, etc).
But we have fresh new stuff to replace it with.  What to do?

	$ git checkout master
	$ git pull

It's reasonable practice to always work on rawhide first.  So let's
start our "lock-step" procedure with updating rawhide and then making
everything else lock into step with that.  The 'pull' makes sure you are
looking at the current state on the server.  Your local branches won't
update by themselves.  If you want to keep from getting confused, you
probably want to do:

	$ git checkout master && git pull
	$ git checkout f14/master && git pull
	$ git checkout f13/master && git pull
	$ git checkout f12/master && git pull

before you start thinking about merges.  Now all your local branches
match the remote ones, and you can verify that with 'git branch -v -a'
to be a little paranoid.

	$ git checkout master
	$ git merge f14/master

We're merging in those .spec changes that got done on Fedora branches.
This is not the only choice for the situation, but IMHO it is the good
way to go.  It keeps the %changelog history of the old Fedora releases
in the new most-current systemtap.spec, which lets those branches both
become identical to master again and not lose any of their %changelog
history that talks about Fedora builds that exist out in the world.

It won't hurt to make that:

	$ git merge f12/master
	$ git merge f13/master
	$ git merge f14/master

for all the still-live "lock-step" branches.  It's unlikely that any but
the most recent one got touched by other Fedora folks, but just in case.

Really, it would be the best idea to do this *before* cutting the next
systemtap release.  Then we can take the merged-up systemtap.spec and
fold it back into the systemtap.spec in the systemtap source tree
proper.  That way we wind up with identical systemtap.spec in Fedora and
in our sourceware releases.

	$ fedpkg new-sources systemtap-1.4.tar.gz
	$ cp .../systemtap-source/systemtap.spec .

That's really only cp if they were made identical before the 1.4 release
got made.  Edit by hand and verify by eyeball as needed.

	$ git commit -a -m"Update to 1.4"

Or however you like to do it, get the changes committed.  Before this
you presumably did some build tests of some sort.  Or you did:

	$ fedpkg local

which does a local rpmbuild of the srpm made from what's in your working
directory, so you can test it out beforehand.

	$ git push
	$ fedpkg build

This runs koji for you, so you can see the rawhide build succeed before
moving on to other branches.  Of course, if you are confident about
things, you don't need to wait for that build to finish successfully
before doing other git work.

	$ git checkout f12/master
	$ git merge master
	$ git push
	$ fedpkg build

This returns f12/master to lock-step (same commit id) as master, and
builds it too.  Repeat for f13, f14.  (Or do f14, then f13, then f12,
or whatever floats your boat.)

Note that 'fedpkg build' looks at which origin/* branch your current
local branch is tracking, and uses the fN/* naming convention to decide
which build target to point your koji build at.  So actually the name of
the local branch doesn't matter (if you want to do fancier things in
your local git repo), but the local branch has to be "tracking"
(i.e. 'git config branch.<branchname>.merge' setting) the right Fedora
server branch.  Using 'git checkout --track -b mybranch origin/somebranch'
is the usual way to get that set up right from the start when you make a
new branch.

I hope this helps make the next rounds of systemtap releases and
associated Fedora shenanigans go smoothly and pleasantly.  I am really
quite pleased with the improvements of the git-based system over the old
CVS-with-insane-convention-choices system, but all of that new goodness
does essentially come from being able to do these arcane things with
git.  So I imagine it's all still rather daunting to anyone who has less
experience with juggling branches in git.  As git goes, what we want to
do here is really quite simple, but that leaves it at pretty subtle and
nonobvious in the absolute sense.


Thanks,
Roland