In modern unix systems, the sheer amount of options in most of the cli tools is plain staggering. Often we have use cases that matter to us, but in order to use them effectively we find ourselves writing a wrapper around them. The real goal is to create a new ‘interface’ with sensible defaults. Now, instead of thinking in solving our original problem, we are thinking how to solve implementation details of an imperative language.(do we need to escape that space?, enclose that string in double quotes?)
Regardless of the language, our tool have a set of common interfaces: argument processing, documentation, configuration files, and variables. loadconfig is meant to unify the management of these interfaces with simple, descriptive yaml strings.
#!/usr/bin/env python '''usage: loadconfig [-h] [-v] [-C CONF] [-E STR] [args [args ...]] loadconfig 0.2.6 generates envvars from multiple sources. positional arguments: args arguments for configuration optional arguments: -h, --help show this help message and exit -v, --version show program's version number and exit -C CONF, --conf CONF Configuration file in yaml format to load -E STR, --str STR yaml config string "key: value, .." Make a list of envvars from config file, yaml strings and cli args. Keywords: check_config: python code for config key validation. clg: Specify command line interpretation. As convention, keys are lowercase with underscore as space. Full documentation: web: https://loadconfig.readthedocs.org pdf: https://readthedocs.org/projects/loadconfig/downloads ''' from loadconfig import Config, __version__ import sys conf = """\ clg: description: $prog $version generates envvars from multiple sources. epilog: | Make a list of envvars from config file, yaml strings and cli args. Keywords: check_config: python code for config key validation. clg: Specify command line interpretation. As convention, keys are lowercase with underscore as space. Full documentation: web: https://loadconfig.readthedocs.org pdf: https://readthedocs.org/projects/loadconfig/downloads options: version: short: v action: version version: $prog $version conf: short: C default: __SUPPRESS__ help: Configuration file in yaml format to load str: short: E default: __SUPPRESS__ help: 'yaml config string "key: value, .."' args: args: nargs: '*' default: __SUPPRESS__ help: arguments for configuration""" def main(args): c = Config(conf, args, version=__version__) print(c.export()) if __name__ == '__main__': main(sys.argv)
Following loadconfig’s philosophy, its script implementation is in itself straightforward. All imperative programming aspects are kept to minimum. As we can see, all keywords and concepts of our conf python variable were already introduced in CLI interface. When using a clg key, loadconfig defines the $prog attribute using args. This allows to decouple the program name from sys.argv. Sometimes, it is nice to get the program version from an external source (eg: another module) and feed it into Config. The convenient Config version parameter is used in this case. At this point, we are familiar with the Config class. c.export is just a Config method that iterates over all keywords defined, making them uppercase, replacing space by underline and prepending the word export. Want to take a guess? We will see shortly why. Finally, all the actual commands are enclosed in the main function as good organizational practice and as it allows for easy testing.
Ok Daniel, all of this looks fine. Did I miss something?
This seamlessly simple script hides really well its true expressiveness power when combined with some shell scripting. Here we go.
Lets assume we have an application that renders sphinx html documentation, detects changes in the documentation sources in real time and controls a browser. There is a wonderful project called docker that encapsulates incredibly well all the application pieces (libraries, fonts, programs) in a single unit called image and offers a neat cli to interact with the operating system. Now, there are lots of possibilities to precisely control how docker will communicate with the filesystem, with the video and audio subsystems, with the network ... Wait! we just want to run our application, remember? Sure! Docker makes the task trivially simple in just one line... one looong line:
docker run -d -u admin -v /data/rst:/data/sphinx -e DISPLAY=:0.0 \ -v /tmp/.X11-unix:/tmp/.X11-unix reg.csl/sphinx
The point is that although it is an incredible simple interface to interact with the operating system, typing those ‘lines’ are not exactly fun. loadconfig allow us to take back the command line interface, defining the defaults we want in configuration files or within a wrapper and to leave the command line for the variable arguments we care the most. In this case, most of the docker run command is setup. The only ‘interesting’ variable part, is the path of our sphinx source documents, in this case /data/rst.
#!/bin/bash CONF=$(cat << 'EOF' version: 0.1 desktop_args: -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix docker_args: -d -u admin -v $(realpath $sphinx_dir):/data/sphinx \ $DESKTOP_ARGS reg.csl/sphinx clg: prog: $prog description: $prog $version is a documentation server. epilog: | Build sphinx docs, launch a browser for easy reading, detect and render doc changes with inotify. options: version: short: v action: version version: $prog $version debug: short: d action: store_true default: __SUPPRESS__ help: show docker call args: sphinx_dir: nargs: '?' default: /data/rst help: | directory holding sphinx conf.py and doc sources (default: %(default)s) check_config: | import os, sys if not os.path.isfile('$sphinx_dir/conf.py'): sys.exit('Error: $sphinx_dir/conf.py not found.') EOF ) set -e ENV=$(loadconfig -E="prog: $(basename $0)" -E="$CONF" "$@") eval "$ENV" [ $DEBUG ] && echo "docker run $DOCKER_ARGS" cid=$(docker run $DOCKER_ARGS) docker wait $cid docker rm $cid >/dev/null
As this is a full application, there are plenty of details to see. Still, with a simple glance, we can see 2 distinctive sections. A config section with just one shell variable, CONF, and an executable section. Most of the ‘code’ happens on the CONF variable. The executable section is driven by loadconfig script, docker and the shell interpreter.
version is a literal string, just as the ones on Basic Config Tutorial
desktop_args is another literal string with a twist. It contains the shell environment variable DISPLAY. The shell will expand it later.
docker_args is also a multiline literal string (separated by \) with a big twist.
- docker_args: -d -u admin -v $(realpath $sphinx_dir):/data/sphinx \
- sphinx_dir is the path we want loadconfig to load as a cli argument.
As such, it is declared within clg. loadconfig will expand sphinx_dir after it runs. We saw loadconfig expansion on the Intermediate tutorial
$(realpath ... ):/data/sphinx is a literal for loadconfig. After loadconfig runs the shell will see $(realpath /data/rst):/data/sphinx assuming the default defined in clg and will expand $()
DESKTOP_ARGS is also a literal for loadconfig. It will be expanded by the shell
clg was covered on CLI interface except for %(default)s with is expanded by clg with /data/rst.
check_config is a special loadconfig keyword. It makes loadconfig exec the declared python string with the primary purpose of validating the configuration. In this case, it checks that a conf.py file exist within the sphinx_dir path
This is where the ‘action’ happens.
- set -e makes the shell to stop when a command does not succeed. This is good shell programming practice and loadconfig takes advante of it.
- ENV=$(loadconfig -E=”prog: $(basename $0)” -E=”$CONF” “$@”) executes loadconfig which will interpret our CONF variable and the command line arguments. Remember that loadconfig printed export lines with the each key of config? This output is assigned to the ENV shell variable. There are two cases where loadconfig will not print envars: when passing the options -h or -v. -h is controlled by clg and -v by the version action on the CONF variable. In these cases loadconfig script exits with 1 which signals the shell to stop as we just saw. The version and the help are printed to the standard error so they can be seen instead of being taken as ENV content.
- eval “$ENV” is what makes those text exported strings become shell environment variables and as such leverage shell commands like docker in this case.
The rest of the lines are simple shell commands:
- [ $DEBUG ] && echo “docker run $DOCKER_ARGS” ouputs the docker call in case we pass the -d option for debugging purpose
- cid=$(docker run $DOCKER_ARGS) launches the docker image reg.csl/sphinx and assigns the container id (sort of a process in normal shell) to the cid shell variable.
- docker wait $cid will wait for the container to stop before returning control to the shell
- And finally docker rm $cid >/dev/null does the cleanup removing the container
Docker is just one (very good) use case example. François Ménabé, the author of CLG, shows us how to leverage KVM virtual machines on his CLG examples. Pretty much all functionality and examples from CLG work unmodified in loadconfig, including CLG execute keyword. There is plenty of CLG and argparse documentation to make the most of the cli.