Casa-Distro

Download Casa-Distro

Just click here:

I - Overview

What is CASA ?

CASA = CATI + BrainVISA

What is Casa-Distro ?

It’s a development environment which helps to provide developers with a working development environment for BrainVISA and CATI tools.

It provides Singularity and Docker images compatible with BrainVISA distributions with all needed setup to build custom toolboxes, package them, and test them, in the same environment.

Casa-distro is the metronome and swiss knife for the management of compilation and publication of CASA software distributions. It is a metronome because the distribution procedures, and especially the distribution frequency, is defined in casa-distro documentation (bioproj wiki). It is a swiss knife because it provides a tool for the management of the whole distro creation pipeline (configuration source retrieval, compilation, packaging, publication, etc.).

Use cases

  • I develop toolboxes, I need to build and release them as binary compatible with the official BrainVisa distrtibutions
  • I am a contributor of Cati/BrainVisa environment, and need to get started quickly
  • I am release maintainer of BrainVisa and need to produce a new release yesterday

What Casa-Distro is not

  • Casa-Distro is not a build sytem: this is the job of BrainVisa-Cmake. Casa-distro brings a wrapper for the build system: it provides Docker images of Linux systems which contain BrainVisa-Cmake and many other thirdparty development tools and libraries, which are already setup for development and distribution of tools.

    The user will be able to use the build system (bv_maker), inside Docker, in a way that ensures to build, run, and ship software that is compatible with public distributions of BrainVisa.

  • Casa-Distro is not a binary distribution of BrainVisa. Even if it could be used that way. Binary distributions provide binaries that try to work on the wider possible variety of systems. Casa-distro is one (or a small set) of these systems with development, distribution, and run environment.

II - Background: distributions, versions and build workflows.

Casa (BrainVISA / CATI) is composed of many versioned software components (more than 50 at the time of this writing). Each one has its own time line for development and release. Casa team is not big enough to follow the release of all projects and make sure that good practices are followed (for instance make sure that the project version is increased whenever the project source are changed between releases). It was therefore decided to schedule casa-distro versions at a frequency that is independent of project versions. This distribution frequency is a fundamental element of casa-distro; every first monday of each month, a distribution is created. The creation of a distribution is a series of steps that leads to the creation of build workflows and packages. Build workflows are all the directories used by developers who need to create or modify software. Packages are single file archives used to distribute all the component of a distro to end users.

Distributions

A distribution (or distro) is a selection of software components that can be shared with collaborators. All the distro are compiled, tested and packaged the same way and at the same time. But each distro have its own software content, targeted audience and license agreement. At the time of this writing, the following distros are planned to be technically managed by CASA :

  • opensource: the open-source subset of projects managed by BrainVISA.
  • brainvisa: the historical BrainVISA distribution. It combines open source and close source software. It can be downloaded by anyone and used freely for non profit research.
  • cati_platform: this distro contains the sowtware necessary to operate CATI platform. It is managed by CATI and distributed only to CATI members.
  • cea: this distribution is based on the BrainVISA distribution but contains also some toolboxes that are not distributed to the whole community. It is installed in several CEA labs : Neurospin, MIRCen and SHFJ.

Distribution frequency

(in the future)

A distribution is created every first monday of each month. Normally, any modification done after a distribution must wait for the next distribution to be released. However, in case of emergency, it is possible to add exceptional distributions.

Distribution versioning

It has been chosen to use a classical version numbering convention : <major>.<minor>.<patch> where :

  • <major>: is a number that is increased whenever a major modification is done on significant components inducing an incompatibility with latest distribution.
  • <minor>: is a number that is increased whenever important features or modifications are done but without introducing incompatibilities.
  • <patch>: is a number that is increased whenever routine modifications are done.

Distribution creation

The creation of a distribution is a three steps process :

  1. Create release plan: selection of the software components that are going to be upgraded in the next release of the distribution.
  2. Apply release plan: update the version of generic branches (latest_release, bug_fix and trunk) in the source code of all projects according to the release plan.
  3. Compilation, testing and packaging: creation of build workflows, tests execution and packages creation for the new distribution.
  4. Distribution deployment: make build workflows and packages available for users.

All these steps are done by CASA admin team using casa_distro command. The release plan is discussed with distros managers and eventually modified before being applied. The current status of the release plan can be found on a BioProj wiki page.

Build workflow

A build workflow represent all what is necessary to work on a distro at a given version for a given operating system. In casa_distro, a build workflow is identified by three variables:

- distro: the identifier of the distro (e.g. opensource, brainvisa, cati or cea)
- branch: the name of the virtual branch used to select sofware component sources (latest_release, bug_fix or trunk).
- system: the operating system the distro is build for (e.g. ubuntu-12.04, ubuntu-16.04, win32 or win64).

The hardware representation of a build workflow is a set of directories :

- conf: configuration of the build workflow (BioProj passwords, bv_maker.cfg, etc.). The content of this directory is the input of the compilation, packaging and testing steps. There are some predefined content for conf that can be generated with command casa_distro create_build_workflow but the whole build workflow can be customized by editing files in this directory.
- src: source of selected components for the workflow. The content of this directory is first created by bv_maker from within a Docker container running the targeted operting system. Simply call casa_distro bv_maker for this. The same command can be used to update sources to latest revision or to recompile when source code has been modified.
- build: build directory used for compilation. Like the src directory, the content of this directory is created by commands such as casa_distro bv_maker.
- pack: directory containing distribution packages. Packages that are created by bv_maker are stored in that directory.
- test: directory used during testing. Typically reference data will be downloaded in this directory and compared to test data generated, in this directory, by test commands.

III - casa_distro installation and setup

Requirements

To use Casa-Distro, a user (or rather a developer) must have a system with the following characteristics:

  • Either Singularity or Docker must be installed and setup for the user on the building system. These container technologies only runs on reasonably recent Linux systems, recent Mac systems, and Windows.
  • Python >= 2.7 is necessary run the casa_distro command.

The rest takes place inside containers, so are normally not restricted by the building system, (as long as it has enough memory and disk space).

For instance, to use Singularity on Debian based Linux systems (such as Ubuntu), the following packages must be installed :

# System dependencies
sudo apt-get install python build-essential

# Singularity install
VERSION=2.4.5
wget https://github.com/singularityware/singularity/releases/download/$VERSION/singularity-$VERSION.tar.gz
tar xvf singularity-$VERSION.tar.gz
cd singularity-$VERSION
./configure --prefix=/usr/local
make
sudo make install

For more details about installation, setup, and troubleshooting, see Installation and setup

Install latest release

This is the simplest and the only recommended way to use casa_distro. Simply download the latest release with the following link : casa-distro.zip.

Then, provided that Python is installed on your system, you can use casa-distro.zip directly. For instance, to get the help of the command, do:

python casa-distro.zip --help

Install with brainvisa-cmake

This advanced method can be used by people that are familiar with the use of BrainVisa-Cmake. Casa-distro belong to the standard projects of brainvisa-cmake so in most case, there is no specific modification to do on bv_maker.cfg. casa_distro can be downloaded, configured and built like any other brainvisa-cmake components. Once built, it can be used as other commands compiled by bv_maker. For instance :

<path_to_build_dir>/bin/bv_env_host casa_distro --help

Building CASA projects

First time

Casa-distro is pre-setup to handle CATI/BrainVisa open-source projects. In this situation, once casa_distro has been downloaded or installed (it can be the .zip file), a user has to:

  • create a build workflow, which will contain the sources repository for BrainVisa projects, a build directory, etc.
python casa-distro.zip -r /home/me/casa create distro_source=opensource branch=bug_fix system=ubuntu-12.04

This command specifies to setup a build workflow for the open-source projects set (distro_source=opensource is actually the default and can be omitted), for the bug_fix branch (default is latest_release), using a container system based on Ubuntu 12.04. Directories and files will be created accordingly in the repository directory, here /home/me/casa (default without the -r option is $HOME/casa_distro).

  • build everything
python casa-distro.zip -r /home/me/casa bv_maker distro=opensource branch=bug_fix system=ubuntu-12.04

If distro, branch, or system are not provided, all matching build workflows will be processed.

Additional options can be passed to the underlying bv_maker command, which will run inside the container. Typically, the documentation can be built, testing and packaging can be performed.

Updating projects

To update to the most recent versions of the projects sources, and rebuild, it is simply a matter of re-running python casa-distro.zip bv_maker (with corresponding options, if needed).

Customizing projects

It is possible to customize the projects list to be retreived and built. It is done by editing the bv_maker.cfg file in the build workflow, which can be found in the directory <repository>/<distro>/<branch>_<system>/conf/

where <repository> is the base casa-distro repository directory (passed as the -r option of casa_distro, <distro> is the projects set name (opensource, brainvisa, cati_platform), <branch> is the version branch identifier (latest_release, bug_fix, trunk), and <system> is the system the build is based on.