Handle Guile modules using Automake

Introduction

For a number of reasons, I had found myself using the Guile Scheme programming language to embed a scripting language into a larger application. Essentially, there was a fixed core written in a low level language, exposing various features to the user, and then Guile on top of, allowing the user to actually use those features by combining them, effectively programming the platform.

The platform uses Autotools (Autoconf, Automake and Libtool) as its build system, so naturally I wanted to use them to also handle the Guile part of the project, as Automake will also handle proper installation of the program(s).

However, the platform had its own set of modules; in fact, it's mostly written in Guile, as the Scheme codebase ended up being larger (and in some parts more complex) than the core. Due to how Guile handles modules, it's not very straightforward, for Automake, to properly deal with them. This article presents the solution used for this specific project.

As a small disclaimer, I don't know if this is the proper way to handle Guile modules: this solution is based on the requirements of the platform I was working on and it might be completely different than your standard Guile project.

The structure of the project

This section is just a short description of the part of the platform using Guile. It describes the directory tree and some design choices.

The project tries to stay as contained as possible: executables and shared libraries are installed in the standard locations (so that the rest of the system can find them), but everything else is placed under a single directory in the “data directory”, as defined by Automake's pkgdatadir variable. That way, be it a file containing an initial configuration (and thus not something that users should edit, but rather override with a separate file) or a Guile module, it will be found and loaded by searching in the same place (modulo the internal directory tree, e.g. modules might be under the modules directory.)

Effectively, the files in the “data directory” are files that are not executed nor are supposed to be handled by users, but rather are loaded (for some definition of loaded) in memory and, at best, handled by the scripting system.

Lastly, the modules will be installed as pre-compiled code, instead of the source code. There is essentially one reason why: if the module is installed as-is (i.e. the .scm file), the embedded Guile will compile it at startup. By itself it wouldn't be too terrible, aside maybe an extended startup time the very first time (which to be honest is awful, especially as there are many modules), but the main issue is that the program has an interactive prompt and messages about modules being compiled would appear any time the program is updated or the Guile cache directory is emptied, before the prompt is shown. That's hardly good user experience in my opinion (in fact, I found them to be quite annoying), thus, pre-compiled modules.

Setting up Automake

This section isn't really about installing Automake or anything like that as I assume it's alread installed. Rather, it's about telling Automake some informations about the structure of the Guile code.

First, we define the directory where the code should go during installation:

modulesdir = $(pkgdatadir)/guile-modules

The directory name is actually important. Later on, it will be explained in more details.

Now, Automake needs to know which files are going to be installed in that directory. For this article I'll just use two files, foo.scm and bar.scm.

modules_DATA = foo.go bar.go
CLEANFILES = $(modules_DATA)

The files are considered _DATA because Automake doesn't handle Guile files that well. By defining them as plain data, they will simply copied in the modulesdir directory, without trying to operate on them in some other magical way.

The CLEANFILES variable is set so that commands like make clean will remove the generated compiled files, like it happens for the other built-in supported languages.

Lastly, we have to tell Automake which files have to be included in the distribution archive when running make dist:

EXTRA_DIST = $(subst .go,.scm,$(modules_DATA))

Dependencies

It's inevitable, in large projects, for modules to have dependencies on other modules. There are countless programs out there with modules (or equivalent) called “utils”, “misc”, “helpers” or something along those lines. Because it also happened in this project, a method to handle dependencies had to be devised.

The concept of dependency is very simple: if file A depends on file B, then the file B must be compiled before file A. Similarily, if file B depends on the files C and D, both C and D have to be compiled before B, so they will also be compiled before A. Additionally, if D is modified, then B must be recompiled (which most likely will trigger a compilation of A.)

Most of that process is already handled by Make, but as stated already, Automake doesn't really handle Guile. In particular, dependencies in Guile are mostly artificial, as the compiling process happening at startup takes care of that automatically.

Since we can't rely on either Automake nor the startup compiler, this is what we'll do: first, a “class” of variables are created, following Automake's conventions; then, these variables are used to inform the compiler about the module's dependencies; finally, the standard Make behaviour will take care of the rest.

The variables follow this pattern: first the file name, including its extension (the .go part) is “canonicalized” using Automake's rules, that is, each character that is considered invalid in an identifier is replaced by an underscore character; thus, the name foo.go will become foo_go. After transforming the file name, the suffix _DEPS is appended.

Automake does have a standard _DEPENDENCIES class, but it will interpret its content as something target the C language (or C++ or the other few built-in languages), so it can't be used.

Assuming that the file foo.go depends on bar.go, the Automake file will have something like this:

foo_go_DEPS = bar.go

Compiling Modules

Now let's examine how to compile modules. If you tried running the generated Makefile using the definitions written so far, you should've noticed that modules are not actually being compiled at all, even if you try to override the compiler.

The reason is that the Makefile doesn't have any rule specifying how to generate each .go file from the respective .scm file; it likely doesn't even know which is which.

At a very basic level, we'd have to write the following rule, here written only for the foo.go file, for each of the involved modules:

foo.go: foo.scm $(foo_go_DEPS)
    $(AM_V_GEN)$(GUILD) compile $(guild_flags) -o $@ $<

The GUILD variable is taken from the environment (like the CC variable that is pretty much standard in Makefiles), while guild_flags is the following:

guild_flags = $(GUILD_FLAGS) -W unused-variable -W unused-toplevel -W unbound-variable -W arity-mismatch

In which GUILD_FLAGS is, again, taken from the environment.

Writing that recipe for each new module is unfeasible, as we'd have to first add it to the modules_DATA variable, then add a _DEPS list of files and finally copy the code above and edit it to reflect the file being compiled.

In the long run it will become unmanageable, especially if the command has to be changed for every file being compiled (e.g. to add an echo command before the call to the compiler.)

Fortunately, Automake (or maybe just Make?) provides a way to define “templates” which can be used to generate code:

define SCMCOMPILE_template =
 $(1): $(subst .go,.scm,$(1)) $$($(subst .,_,$(subst -,_,$(1)_DEPS)))
     $(AM_V_GEN)$(GUILD) compile $(guild_flags) -o "$$@" "$$<"
endef

The above command is fairly magical and unreadable, but in short, it does the following: it takes the value of the first argument and uses it as the target of the recipe; it substitutes the .go extension with the .scm string and uses the result as the first dependency; lastly, it changes each dot and dash into an underscore, it appends the _DEPS suffixs and use it to generate a “variable use”, that is, a string in the form $(file_name_DEPS). Depending on the actual file names, you might need to also add the + character (e.g. if a file is called foo++.go) or any other symbol that can't be part of an indentifier.

The second line is the command as shown above, with the only difference being in the double dollar signs.

Now if you try to run make all, you'll see that it will finally try to compile the foo.scm file. Except that it will complain about not being able to find bar.go!.

This is because the compiler needs to be told where to find the dependencies (in this case the bar module.)

Let's say that foo.scm contains the following commands:

(define-module (my-special-project foo))

(use-modules (ice-9 textual-ports) (my-special-project bar))

In this case, we can see that the module foo resides in the my-special-project “namespace” and depends on (ice-9 textual-ports) and (my-special-project bar).

The textual-ports module comes packaged with Guile itself, so we'll ignore it.

The my-special-project part is important because Guile will search the load path for two files: my-special-project/foo.scm and my-special-project/foo.go, that is, it will treat the “namespace” as the name of a directory and the module name as the name of the file containing the module itself.

Because the default load path doesn't contain the my-special-project directory, we have to inform the compiler about it by using the -L flag.

Since the compiler places the generated .go files in the same directory as the source code, the argument to the flag would then be the parent directory, as it would then search for $PWD/../my-special-project/ for the module files, where PWD is the standard UNIX environment variable.

There is one last change to make: since the directory being searched is my-special-project and the compiler will also scan .. for modules, we must change the name of the directory containing the modules to my-special-projects. In the shell:

$ mv guile-modules my-special-project

And in the Makefile:

modulesdir = $(pkgdatadir)/my-special-project

If you now run make all, you'll see that bar.go is compiled before foo.go and that dependencies are found correctly.

Conclusion

This usecase is rather specific and I'm sure barely anybody will ever need it, but I had to and I'm sure I'll have to read this article again at some point in time, so here it is.