A new method of storing package data base information, proposingpackages, and announcing updates

[Daniel Reed <n at ml dot org>]

This method was discussed briefly several months back, but I feel it may be
time to discuss, finalize, and move forward.

THIS WILL REQUIRE ALL PACKAGE MAINTAINERS TO CHANGE THEIR BEHAVIOR. It
should be a small change: just the addition of a few lines to each
setup.hint, but the new method will only be useful if all package
maintainers are happy with the change and are willing to ensure the new
method is followed.


Currently, I store all package information in a large XML document which I
update by hand whenever a new package is ITPd, an existing package is
updated, or some other change must be made against the live repository. XSL
transformations convert this XML document into several other formats,
including the plaintext PPL, MPL, and make files to automate downloading,
reviewing, and pushing packages into the live repository.

If nothing else, the XSL transformations on the XML document take a lot of
CPU time, which makes it slow to push packages, generate the PPL, etc.
Additionally, package information is spread between setup.hint files and the
XML file, making it hard for other Cygwin contributors to keep everything
synchronized.

The proposal from several months back was to extend the format of the
setup.hint files to include *all* package-related information. Currently,
setup.hint files only contain information necessary to build the master
setup.ini file--that is, information that would show up in setup.exe, or is
otherwise needed to prepare new packages for distribution by setup.exe.

In order to ease the in-place transition, it was agreed that the non-
setup.exe information should be stored as specially-formatted comments in
setup.hint. That is, information that is not used to generate setup.ini will
stay hidden from the utilities that generate setup.ini (so these utilities
will not need to be modified ahead of time).

One proposal was to use the sharp/bang notation traditionally used to denote
executable script files, leaving all other lines beginning with a sharp (aka
hash or pound) as un-interpreted comments. Since these files are not truly
executable files (and can not be made executable without modifying the
utilities that generate setup.ini), it will be important to ensure that no
setup.hint begins with a sharp/bang special comment.

For example:

release/foomaster/setup.hint
  sdesc: "The Finite Object Organization visualization tool"
  ldesc: "The Finite Object Organization visualization tool allows visual
  navigation of organizations of finite objects."
  requires: cygwin libfoo
  category: Utils
  #!maintainer Daniel Reed
  #!bin http://www.example.net/products/foomaster/Cygwin/foomaster-%{version}-%{release}.tar.bz2
  #!src http://www.example.net/products/foomaster/Cygwin/foomaster-%{version}-%{release}-src.tar.bz2
  #!hint http://www.example.net/products/foomaster/Cygwin/foomaster.hint

release/foomaster/libfoo/setup.hint
  sdesc: "The Finite Object Organization library"
  ldesc: "The Finite Object Organization library provides service routines to
  organize finite objects."
  category: Libs
  #!maintainer Daniel Reed
  #!related-to foomaster
  #!bin http://www.example.net/products/foomaster/Cygwin/libfoo-%{version}-%{release}.tar.bz2
  #!src http://www.example.net/products/foomaster/Cygwin/libfoo-%{version}-%{release}-src.tar.bz2
  #!hint http://www.example.net/products/foomaster/Cygwin/libfoo.hint

release/foomaster/libfoo-devel/setup.hint
  sdesc: "libfoo development headers"
  ldesc: "The Finite Object Organization library provides service routines to
  organize finite objects."
  requires: libfoo
  category: Devel Libs
  external-source: libfoo
  #!related-to foomaster
  #!bin http://www.example.net/products/foomaster/Cygwin/libfoo-devel-%{version}-%{release}.tar.bz2
  #!hint http://www.example.net/products/foomaster/Cygwin/libfoo-devel.hint


The procedure [on my end] for pushing new packages would be:
  1) Wait for the announcement that a new package is available.
  2) Grab the %{hint} file from the URL in the existing .hint file. This will
allow a package maintainer to move URL bases around with the only backward-
compatibility requirement being that a stub .hint file be left in the prior
location. (If the value of %{hint} changes between the existing .hint and the
newly-downloaded .hint, step 2 will be repeated at least once.)
  3) Using URL bases in the new .hint, grab the %{bin} and %{src} packages.
  4) Perform automated sanity checks against the bin package.
  5) Push the new hint, bin, and src packages into the live repository.

After step 5, the automatic utilities will notice the new package and regenerate
setup.ini with the new .hint's information.

I envision the syntax for a push to be something as simple as:

  cd release/.util; make upgrade NAME=foomaster CURR=2.0-3 PREV=1.3

%{bin}, %{src}, and any other URLs will be evaluated with %{version} = 2.0 and
%{release} = 3 in this case. If CURR had been simply 2.0, %{version} would be
2.0 and %{release} would default to 1.

PREV would only be used to tell the script which previous version to leave; all
others would be removed.

If no CURR is provided, the script will expect the to-be-downloaded .hint file
to include at least an appropriate curr: and possibly a prev: line. This will be
the prefered procedure when upgrading a package (to minimize the chance of the
package coordinator manually typing the wrong value). The syntax would then
simply be:

  cd release/.util; make upgrade NAME=foomaster

Or possibly even:

  cd release/foomaster; make -f ../netrel.mk upgrade


For pending packages--those packages that have been created for Cygwin but have
not been formally accepted into the Cygwin distribution--a non-release/
repository will be maintained in a non-live location. In addition to holding
operational values like #!bin and #!maintainer, the .hint files stored in
non-release/ will include values like #!vote, #!problem, #!review, and any other
information required to generate the PPL. Once a package has been formally
accepted these values may be kept for historical purposes but, as the package
maintainer will control the contents of the .hint at that point, they may be
replaced with other values or removed altogether.

(In practice, these PPL-specific values may be held in a separate file in
the same directory and simply merged with setup.hint for processing.)

To propose a new package, one would simply send the proposed initial
contents of the package's .hint file to the cygwin-apps mailing list. An
automated script can be used to periodically scan the archives looking for
these messages. If the .hint section of the email is guaranteed to always
begin with "@ packagename", no other special formatting should be required
for these proposal messages. That is, the following would be acceptable for
automatic purposes:

  From: Daniel Reed <n@example.net>
  To: cygwin-apps@cygwin.com
  Subject: hi

  Hi. This is a cool package.

  @ foomaster
  curr: 1.3
  sdesc: "The Finite Object Organization visualization tool"
  ldesc: "The Finite Object Organization visualization tool allows visual
  navigation of organizations of finite objects."
  requires: cygwin libfoo
  category: Utils
  #!bin http://www.example.net/products/foomaster/Cygwin/foomaster-%{version}-%{release}.tar.bz2
  #!src http://www.example.net/products/foomaster/Cygwin/foomaster-%{version}-%{release}-src.tar.bz2
  #!hint http://www.example.net/products/foomaster/Cygwin/foomaster.hint

  And this one!

  @ libfoo
  curr: 1.1.5
  sdesc: "The Finite Object Organization library"
  ldesc: "The Finite Object Organization library provides service routines to
  organize finite objects."
  category: Libs
  #!bin http://www.example.net/products/foomaster/Cygwin/libfoo-%{version}-%{release}.tar.bz2
  #!src http://www.example.net/products/foomaster/Cygwin/libfoo-%{version}-%{release}-src.tar.bz2
  #!hint http://www.example.net/products/foomaster/Cygwin/libfoo.hint

  And this!

  @ libfoo-devel
  sdesc: "libfoo development headers"
  ldesc: "The Finite Object Organization library provides service routines to
  organize finite objects."
  requires: libfoo
  category: Devel Libs
  #!bin http://www.example.net/products/foomaster/Cygwin/libfoo-devel-%{version}-%{release}.tar.bz2
  #!hint http://www.example.net/products/foomaster/Cygwin/libfoo-devel.hint

  Thanks.

However, keep in mind that this example is intended to show the strength of the
message format, specifically its resilience to human neglect. It is advised that
proposal messages should adhere to existing Cygwin standards for informative
Subject content and introductory paragraphing.

If no #!maintainer is supplied, the value will be pulled from the From field.

For messages containing multiple .hint file contents, all packages after the
first will be treated as "related-to" the first unless an explicit
#!related-to is given. #!related-to may be provided and left blank to force
the subsequent package to reside at the top level of the repository.

In the interests of simplicity, it may make sense to allow the @ line to
contain a path instead of just a package name. That is, "@ libfoo-devel"
followed by "#!related-to foomaster" could be simplified to just
"@ foomaster/libfoo-devel". The final, live setup.hint would return to
"@ libfoo-devel" in the foomaster/libfoo-devel/ directory.

If no #!src is provided, external-source will be required in the final
.hint. If no external-source is provided when one is needed, the field will
be set to the first "complete" package preceeding the current one. That is,
in the above example, libfoo-devel will have an external-source of libfoo.
If a fourth package, libfoo-demo, was provided without #!src nor
external-source, it too would be given an external-source of libfoo
(skipping over libfoo-devel). For truly binary-only packages (such as
packages composed completely of shell scripts), #!src may be provided and
left blank.

Long URLs and other data that exceed 80 characters will have to run past the
80th column; it will be an error to split long lines. Furthermore, "@" and
"#!" must start in the first column; it will be an error to start any line
in a .hint file with any leading spacing. Every effort will be made to
ensure that extra spaces between the #!-section and the value, as well as
any trailing spaces are ignored.


I have not decided on an optimal machine-readable format for announcing
updates. If the hint URL (#!hint) has not changed, all that is needed for
automation purposes is the new CURR and possibly a new PREV. It may be
easiest in practice to allow package updates to be announced by again
posting the .hint file in the above format.

However, if the package maintainer's online .hint file contains curr: and
prev: entries, even that information is not strictly required in the
announce message; simply announcing that a new version is available would be
sufficient. This would make it easy to construct an automatic web form or
special-purpose email address for package maintainers to cause new versions
to be downloaded, auto-reviewed, and pushed to the live repository without
any human intervention.


If this proposal is accepted, I will begin writing the required backend
utilities. Once they have been written, the exact non-setup.exe syntax can
be finalized and the documentation can be updated. Offhand, I believe only
the setup.hint section of http://cygwin.com/setup.html will need to be
updated.

I think this is a good proposal, but I am open to any suggestions.
Maintaining the PPL and assorted support files has become very time-
consuming lately; I am hopeful this new method will allow us to redistribute
some of the coordination load and help streamline Cygwin maintenance.

-- 
Daniel Reed <n@ml.org>	http://naim-users.org/nmlorg/	http://naim.n.ml.org/
1832 Savior214: that sucks that one day your just gonna die and all that
work you did learning stuff just gets a rm -rf