Emacs: writing a project.el backend


Lately I've seen on the web, people talking about project.el, the solution to handle “projects” which comes built-in with GNU Emacs 27.1

I decided to experiment with it too, but I've met two issues:

  • The file coming with the base installation of the editor is outdated, at least on 27.1; you still need to install the latest from ELPA.
  • The project I wanted to use it with is sort-of incompatible with the built-in settings.

The first is easy to solve, but it was surprising at first. The next GNU Emacs version should have the updated version, looking at the commit history.

The second is the one which caused more difficulties and will be the subject of this article.

How project.el deals with projects by default

This description was written when ELPA had version 0.5.3 of the project (i.e. project.el) package. While unlikely, it is possible for maintainers to change the default behaviour in future versions.

The package is structured around functions defined with cl-defgeneric and cl-defmethod, so that the correct behaviour is performed by dispatching on the structure of the data type identifying the project (more on this later.)

The “entry point”, term used a bit improperly, is the generic function project-root: this function must return the root directory of the project, that is, the directory containing each file of the project and the parent directory of which is not related to the project.

As a quick example, assuming the currently visited file is ~/MyProject/src/utils/magic.c, project-root will return ~/MyProject/

The generic implementation will throw an error when executed, because according to the bits of documentation at the beginning of the file:

;; `project-root' must be defined for every project.

After obtaining the root directory, the remaining functions will return the appropriate values relative to this directory.

The interested functions are the following: project-files will return all the files in the project, so that it's possible to visit them easily and with completion at the prompt; project-ignores will return a list of patterns to remove from the set returned by project-files; project-external-roots will return a list of directories that are not part of the project itself, but which are still related to it in one way or another.

The package also provides a backend, instead of just the generic functions. This backend is based on VC, the Emacs built-in system to deal with different VCS.

Due to the ubiquity of VCSs, this backend, as basic as it might be, will satisfy the majority of use cases. Even though VC doesn't support every VCS out there by default, there are external packages to extend it.

However, the project I was dealing with was not managed by any VCS! Therefore, I couldn't use the full power of project.el, making it essentially useless for this specific case. Still, the codebase has a particular structure suitable for a new project.el backend.

Writing a new backend

Before we begin, let's describe the structure of the project at hand: starting at the root directory, the project is made of either files or other directories. Some directories must be ignored, for example because they contain artifacts generated during the build process. Similarily, some files must also be ignored according to some pattern, like the file extension.

The defining characteristic of the project is that each directory has 0 or 1 Makefile; the root directory will always have a Makefile.

Because of this, the new backend will describe a project according to these Makefiles.

A “backend” is a set of methods dispatching on a specific project instance. As such, the first step is to define a new structure: this structure must abide these rules:

;; - Choose the format of the value that represents a project for your
;; backend (we call it project instance).  Don't use any of the
;; formats from other backends.  The format can be arbitrary, as long
;; as the datatype is something `cl-defmethod' can dispatch on.  The
;; value should be stable (when compared with `equal') across
;; invocations, meaning calls to that function from buffers belonging
;; to the same project should return equal values.

You can read which data types cl-defmethod can dispatch on by reading its documentation. The simplest data type, however, is a cons (or a list), so we'll be using that to define our instance. The built-in VC backend also uses a cons, though with different contents.

Specifically, the format will be a list whose car is the symbol makefile and whose cadr is the root directory as a string.

For reference, the VC-based backend uses an actual cons, with the car being the symbol vc and the cdr the root directory, but for reasons listed later we're going to use a list.

Now that we have our format, we can specialize project-root accordingly:

(cl-defmethod project-root ((project (head makefile)))
  (car (cdr project)))

Now, whenever the project.el internal API receives an instance of a Makefile-based project, it will correctly return the root directory.

But how does project.el know wether a project is based on Makefiles or is using a VCS? The answer is in a special hook, project-find-functions.

This hook will execute all the functions listed within and will use the first non-nil result as the project instance.

The function called by the hook must take one argument: the current directory the user is in. The function must then build the project instance based only on this argument.

(defun project-makefile-try (dir)

We are going to use locate-dominating-file to search for the topmost Makefile, allowing us to quickly skip directories not containing any.

However, some projects, most notoriously those built using autotools, have recursive Makefiles, meaning that the path returned by locate-dominating-file is not necessarily the root directory.

As such, we're going to traverse the directory tree backwards until no more Makefiles are found:

  (let ((dominating (locate-dominating-file dir "Makefile")))
    (when dominating
      (let* ((above (file-name-directory (directory-file-name dominating)))
	     (dominating2 (locate-dominating-file above "Makefile")))
        (while dominating2
	  (setq dominating dominating2
	        above (file-name-directory (directory-file-name dominating))
	        dominating2 (locate-dominating-file above "Makefile"))))

This function will look only for files called ‘Makefile’, but it should be trivial to extend it to also search for e.g. ‘GNUMakefile’

Now here is a new thing: as said above, the backend can also deal with ignored files and external root directories. In fact, the main reason why I started writing a new backend was exactly because the generic functions could not be extended to include other elements in additions to the default values.

For reference, the VC-based backend uses the relative VCS (git, hg, etc.) to create a list of ignored files and a special function to list the external roots.

Because we are basing the projects on Makefiles only, we can't do like the VC backend, at least for ignored files, so we'll have to make do with something else. What we'll do is to save within the project instance the list of patterns to ignore and the list of external roots.

Initially, I wanted to use dir-local variables to perform this task, but unfortunately I met a couple of bugs I could not resolve, so for the time being we'll make due with an optional file called .project, where each pattern or directory is on a single line and is prefixed by : (a colon) and if the colon is also prefixed by # (a hash), then the pattern or directory will be placed in the list of ignored files. Implementing reading the list of ignored files from dir-locals-file is left as an excercise for the reader.

      (let ((igns nil) (extr nil) (dotfile (concat dominating ".project")))
        (when (and (file-exists-p dotfile) (file-readable-p dotfile))
	  (let ((linep t)
		(line ""))
	      (insert-file-contents-literally (concat dominating ".project"))
	      (goto-char (point-min))
	      (while linep
		(setq line (buffer-substring-no-properties
		(let ((split (split-string line ":")))
		  (when (= 2 (length split))
		    (if (string= (car split) "#")
			(setq igns (cons (car (cdr split)) igns))
		      (setq extr (cons (car (cdr split)) extr)))))
		(setq linep (= 0 (forward-line 1)))))))

Now that we have our lists, we can finally return the project instance:

        (list 'makefile dominating igns extr)))))

The list of ignored patterns is the third element and the list of external roots the fourth element of the list.

Now that our instance is ready, we can specialize the remaining methods:

(cl-defmethod project-external-roots ((project (head makefile)))
  (car (cdr (cdr project))))

(cl-defmethod project-ignores ((project (head makefile)) _dir)
  (append (car (cdr project)) grep-find-ignored-files))

The grep-find-ignored-files variable contains some common patterns, like *~ or *.o, which are normally ignored in most circumstances, essentially giving us a default list of patterns in case the user does not create this “.project” file (or equivalent.)

This backend does not specialize project-files, but that's ok. Even though the documentation warns us that the generic function will get slower the more files the project has, we really can't do any better: it will use the “find” tool to scan the filesystem, using the list of patterns returned by project-ignores to exclude the files matching those patterns.

Unlike the VC-based backend, this Makefile-based project can't really use tools operating on data formatted in a particular way like VCSs do, so the only way to make it faster would be to use a tool faster than “find”.

The final step to make it all work is to place this call somewhere, like the init file:

(add-hook 'project-find-functions #'project-makefile-try t)

You should strive to keep the default VC-based backend first, as this Makefile-based instance will conflict in some cases, giving you undesirable results.


All in all, extending project.el is trivial and the only thing lacking is a broader user-facing API, but that's something that will eventually come to be the more people use it.

On the other hand, repeating what I said earlier, thanks to the fact that VCSs are everywhere, there's not really a need to write new backend but rather it would be more useful to write extensions to Emacs's VC package instead.