Writing a test library for Common Lisp

Origin
Todo
First draft

Published October 2016, work in progress

Origin

I've been trying to write a test library. I have no will to publish it: it's just meant as an exercise on a problem I understand. This project stems from a conversation I had on twitter with shinmera, who was requesting comments on a test library design he was working on. As usual, I want to document what I find as I proceed.

If you are actually interested in a complete CL test library, you may want to have a look at Parachute.

Todo

[X] Replacing eval with funcall
[ ] Test results presentation
[ ] Test organizations (see notes below)

First draft

Here what I have so far:

(in-package #:bait)

(defparameter *tests* (make-hash-table :test 'eql))

(defclass test ()
  ((name :initarg :name :accessor test-name)
   (body :initarg :body :accessor test-body)))

(defmacro define-test (name &body body)
  `(setf (gethash ,name *tests*)
        (make-instance 'test
                       :name ,name
                       :body ',@body)))

(defmethod run ((current-test test))
  (eval (test-body current-test)))

(defun run-tests ()
  (loop
     for test being the hash-values of *tests*
     collect (run test)))

Here a usage example:

BAIT> (define-test 'mytest2 (= (+ 2 1) 3))
#<TEST {10065F4CC3}>
BAIT> (run-tests)
(T)

I believe the code is pretty straightforward in its intentions, but I'm not sure about the style and the techniques I'm using:

I'm using *tests* to maintain a set of tests, meant to be running via the invocation of run-tests. My problem is *tests* is in the bait namespace, so it will collect tests defined for every system I may want to test in a session. This sounds prone to errors, if not completely wrong. What is a more reasonable strategy would be?
Another approach could be something like (defclass suite ...) (I'm not going to write the actual implementation), then somehow make it current, and the define-test macro would add tests to the current suite. This seems slightly better, but again I'm tampering with a global variable.
(This is a consequence of what I'm talking about in the first question) The define-test macro is not functional, since it contains a big side-effect on *tests*.

Notes and comments from the chat on #lisp

I presented the above questions on #lisp, here are my notes from the answers and suggestions I received.

choosing a correct :test configuration for hashes is fundamental. More that one user asked about what I was using as *tests* keys (strings or symbols).
another observation is about the use of eval instead of funcall, which would be more apt to the ask
- it should only be a matter of using :body (lambda () ,@body)
one suggests to use test collections based on package
- if a test suite is all in the same package, one could use *package* as a base for constructing a key in the tests collection
  - I think another nuance has been suggested in the same line of thought, with the following words "you can 'tie' the test sequence state to a package by storing it in the symbol-plist of the package. If you want to group tests by package, a better idea would be to have hash-table where the key is the package and the value the test sequence".
another one suggests to use asdf, but I'm not sure what it means
- but another one argues "it might be non-trivial to figure out which asdf system a test belongs to".
another option is to group tests by file using a file local variable
- file local variables are not actual part of CL spec, but there are modules implementing them
another user suggests that define-test could create entirely new classes, each one with a method that is the body of the test
another different approach would be to be more declarative implementing a with-test-suite macro, although I'm not sure what its semantic would be.

Writing a test library for Common Lisp

Table of Contents

Origin

Todo

First draft

Notes and comments from the chat on #lisp