XIOS-2 Code Repo Maintenance

Set-up

The /ocean/sallen/hg_repos/XIOS-2-hg-mirror repository is an svn checkout of https://forge.ipsl.fr//ioserver/svn/XIOS/trunk/ at revision 1066 and also a read-only Mercurial repository. It was initialized with:

$ cd /ocean/sallen/hg_repos
$ svn co -r 1066 https://forge.ipsl.fr/ioserver/svn/XIOS/trunk XIOS-2-hg-mirror
$ hg init XIOS-2-hg-mirror
$ cd XIOS-2-hg-mirror
$ cat > .hgignore
syntax: glob
.svn
ctrl-d
$ hg add
$ hg ci -m"Initialize XIOS-2 svn mirror at r1066 of ^/trunk."

svn v1.8.8 was used on skookum for the svn part of the initialization.

Doug maintains an XIOS-2-mirror-merge repo on his laptop. That repo is used to merge changes from the upstream Subversion repository that are brought in via the /ocean/sallen/hg_repos/XIOS-2-hg-mirror repo, and from the Salish Sea team XIOS-2 repo on Bitbucket. The XIOS-2-mirror-merge repo was created by cloning the /ocean/sallen/hg_repos/XIOS-2-hg-mirror repo:

hg clone ssh://skookum.eos.ubc.ca//ocean/sallen/hg_repos/XIOS-2-hg-mirror XIOS-2-mirror-merge

and setting the paths in its .hg/hgrc to:

[paths]
bb = ssh://hg@bitbucket.org/salishsea/xios-2
default-push = ssh://hg@bitbucket.org/salishsea/xios-2
mirror = ssh://skookum.eos.ubc.ca//ocean/sallen/hg_repos/XIOS-2-hg-mirror

Those paths mean that the repo for hg pull and hg incoming commands must be specified explicitly. The bb and mirror paths are provided to facilitate pulling from xios-2 on Bitbucket and /ocean/sallen/hg_repos/XIOS-2-hg-mirror, respectively. hg push and hg outgoing commands will act on the xios-2 repo on Bitbucket, unless otherwise specified.

The Salish Sea team XIOS-2 repo on Bitbucket was created via the Bitbucket web interface and populated there by an hg push from Doug’s XIOS-2-mirror-merge repo.

A working copy was then created by cloning the Salish Sea team XIOS-2 repo on Bitbucket as XIOS-2.

Getting the Salish Sea team XIOS-2 repo on Bitbucket to a state in which team members can clone it and easily build XIOS is annoyingly fiddly. The difficulties largely stem from the fact that the svn checkout includes compressed tarballs in the archive/ directory. On the first build, those tarballs are decompressed into extern/ and tools/FCM/. Subsequent builds check to see if the tarballs have been decompressed, so the tarballs can’t be removed, only their compressed forms.

Here are the steps that were done in a working copy on salish cloned from the Salish Sea team XIOS-2 repo on Bitbucket:

  • Tag XIOS-2r1066

  • Add README.rst by copying and editing the one from the XIOS-1 repo

  • Symlink the XIOS-ARCH/UBC-EOAS/arch-GCC_SALISH.* files into arch/

  • Run make_xios

  • hg forget tools/archive/FCM.tar.gz tools/archive/blitz.tar.gz tools/archive/boost.tar.gz tools/archive/rapidxml.tar.gz

  • Add tools/archive/*.tar.gz to .hgignore

  • hg add tools/archive/*.tar tools/FCM/ extern/blitz/ extern/boost/ extern/rapidxml/

  • hg commit -m"Replace tools/archive compressed tarballs with their extracted contents."

  • hg push

  • Clone the Salish Sea team XIOS-2 repo on Bitbucket on orcinus

  • Confirm that XIOS-2 builds successfully

  • Return to the working copy on salish

  • Add the following lines to .hgignore (it is critical that this not be done earlier because it will prevent needed files from being pushed to Bitbucket):

    .cache/
.void_file
Makefile
arch.*
bin/
cfg/
config.fcm
done/
etc/libxios.cfg
extern/netcdf4
fcm_env.*
flags/
inc/
lib/
obj/
ppsrc/

  • hg commit -m"Don't track build products files and directories."

The workflow sections below explain how these 4 repo clones are used to pull changes from upstream and merge them with changes that Salish Sea team members push to Bitbucket.

../../../_images/XIOS-2CodeRepoMaint.svg

XIOS-2 code repositories and workflow to update and merge SVN and local changes

Workflow to Pull Changes from XIOS-2 svn Repo

The workflow to pull changes from the master XIOS svn repo and commit them to our XIOS-2-hg-mirror repo is somewhat automated by the Marlin – Salish Sea NEMO svn-hg Maintenance Tool.

#. Review the upstream changes in the source browser at https://forge.ipsl.fr/ioserver/log/ to select a range of changes to be pulled into our XIOS-2-hg-mirror repo.

  1. Working on salish in the /ocean/sallen/hg_repos/XIOS-2-hg-mirror repo with an activated virtualenv in which marlin is installed:

    $ ssh salish
$ workon marlin
(marlin)$ cd /ocean/sallen/hg_repos/XIOS-2-hg-mirror

  2. Use marlin incoming information about the next SVN revision that will be pulled from upstream and confirm that it is the expected revision:

    (marlin)$ marlin incoming
r1062 2017-02-23 17:32:17 UTC
  Bug fix in interpolation for cell boundaries generation.

    The --limit option can be used to see more incoming revisions; see marlin help incoming for details.

  3. Use marlin update to update the working copy to the next upstream commit and commit the SVN update as a Mercurial changeset with the SVN commit message as the body of the Mercurial commit message and echo that message:

    (marlin)$ marlin update
Update to svn r1062.

Bug fix in interpolation for cell boundaries generation.

    The --to-rev option can be used to apply a series of upstream updates, committing them to Mercurial one at a time; see marlin help update for details.

Workflow to Merge XIOS-2 svn Repo and Salish Sea Revisions

Merging changes from NEMO svn and the Salish Sea team XIOS-2 repo on Bitbucket is done in a repo that is used for only that purpose. Doug does the merges on his laptop. The repo in which the merging is done was created by cloning the /ocean/sallen/hg_repos/XIOS-2-hg-mirror repo as described in the Set-up section.

After the Workflow to Pull Changes from XIOS-2 svn Repo has been completed the workflow to merge those changes with Salish Sea MEOPAR project revisions is:

  1. Pull and update recent changes from the Salish Sea team XIOS-2 repo on Bitbucket into XIOS-2-mirror-merge:

    cd XIOS-2-mirror-merge
hg pull --update bb

  2. Pull and update the changes from /ocean/sallen/hg_repos/XIOS-2-hg-mirror into XIOS-2-mirror-merge:

    hg pull mirror

  3. Because the changesets pulled from the Salish Sea team XIOS-2 repo on Bitbucket are public a branch merge is necessary:

    hg merge
hg commit -m"Merge svn updates."

  4. Push the result of the updates and merges to the Salish Sea team XIOS-2 repo on Bitbucket:

    hg push bb

    If other users have pushed changes to the Salish Sea team XIOS-2 repo on Bitbucket while merge conflicts were being handled hg pull --rebase can be used to bring in those changes and deal with any additional merge conflicts.

  5. Notify team members of the upstream merge.