Making my tests multi-bound

To multi-bind a test is to make it so that a single test enforces the same semantic requirement against different portions of a system.

This has been going on for a long time but not everyone noticed it. Teams that use the Gherkin syntax to define a behavior are likely to exploit those definitions in two ways:

1. Bind automatically with Specflow or Cucumber.
2. Treat as a manual test script.

This is an early form of multi-binding.

I'm talking about something more than that and it's the foundation of stability in all my pipelines. So we need to discuss it, first.

In my environment, any given test can be executed all of the following ways:

1. As a unit test.
2. As an integration test with no networking (everything runs in a single process).
3. As an integration test for the backend with mocked data.
4. As a gate between the blue and green environments of the backend.
5. As an integration test for the Android client, with a local server and a mocked database.
6. As an integration test for the WebGL client, with a local server and a mocked database.
7. As a gate between the blue and green environments of one of my WebGL deployments.

There's nothing to stop me from adding more ways to run tests if I need them. This seems pretty good for what amounts to a one-man show.

Advantages of Using Acceptance Criteria as Sole Definition of Scope

A while back, I wrote about using acceptance criteria as the sole definition of scope for a software product. I wanted to expand on how that helps development teams.

It gives developers a clear-cut way of ensuring that all the code they write services some business goal. If you can get rid of it and still clear all your acceptance criteria, you don't need it.

It actually makes writing requirements easier, too, once you get used to it. There's only one way to explain what you want and only one place to look to see what you already have.

Acceptance Criteria: Eliminate Ambiguity

One of the problems that people have when trying to build real, meaningful acceptance criteria for their work is that the criteria end up being ambiguous. There are several reasons why this might be and I won't get into them, in this post.

Any such force can be at least partially counteracted by introducing one of two simple rules.

1. Any way to interpret an AC as met means it is met.
2. Any way to interpret an AC as unmet, it is unmet.

You might think these rules are horribly unfair and one-sided. Yet, that's why they work so well. They eliminate the potential for ambiguity by automatically selecting the most relaxed or strictest version of any AC.

At the same time, they provide an incentive to drive ambiguity out of an acceptance criterion's wording. The easiest way to ensure nobody uses the "wrong" version of an acceptance criterion is to make sure there is only one interpretation.

Over time, the criteria will justify themselves and no incentive will be required to make them unambiguous, but something like this can help at first.

Acceptance Criteria: All or Nothing

Commitment works both ways. Teams that commit to delivering on little pieces of acceptable work based upon agreed-to acceptance criteria need to know that Product commits to accepting based upon those same requirements.

Once an acceptance criterion is "in-flight", it should not be adjusted without unanimous consent by all stakeholders who originally committed to the requirement, including developers. Any dissent leaves the old criterion in force.

This gives Product several options for changing their minds:

1. Cancel a criterion and halt all work in that direction.
2. Renegotiate a criterion with engineering.
3. Add new criteria after the current ones are implemented.

Limiting your options to renegotiate creates an incentive to limit the size of an increment of work.

Acceptance Criteria: Entire Scope

One way to help you and your team write better acceptance criteria is to make them the sole authority on the scope of any new work. If there's no acceptance criterion for something, it isn't required. If it isn't required, it shouldn't be done.

For this to work, all ancillary documentation must carry exactly zero weight. The title of a story can't be "a little" meaningful. The description can't enforceable under "special circumstances". They are comments. They mean nothing.

Putting this rule in place has some side benefits I'll delve into later. Right now, I'd like to focus on the fact that this provides an incentive to figure out how to write good acceptance criteria.

If you can only specify a requirement with an acceptance criterion, you'll have to figure out how to make sure each one actually specifies a requirement.

How to Get Better Acceptance Criteria

It's easy to say "rely more heavily on acceptance criteria" but, without some guidelines, the acceptance criteria written end up not being very helpful; they tend to have little to do with acceptance and do a poor job as criteria.

Instituting a few simple rules can boost the effectiveness of shifting toward acceptance criteria as the main vehicle of communication between groups.

1. Acceptance criteria are the sole authority on the scope of a block of work.
2. Acceptance criteria cannot be modified while a block of work is in-flight.
3. An acceptance criterion is met if any of its valid interpretations are met.

I'll drill into each of these pieces of advice separately.

Acceptance Criteria as Vehicles for Specification and Delegation

One way to improve the happiness of both Product and Engineering groups is to change the way they communicate to be more validation-centric. Abandon the question "What do we do, now?" in favor of "How will we know when we've done the next thing?"

Anyone paying attention knows that this is a well-traveled topic, so much so that the question "How will we know when we're done?" is basically a cliché in the software industry. Nevertheless. It's my turn to say it.

Get rid of detailed instructions, descriptions, and design documents. Replace them all with acceptance criteria.

After all, what else matters? How useful is a design document that you can't use to tell whether or not you're done? If you have an instrument that tells you that, what's the value in the other documents?

If you can get good acceptance criteria - really, truly, good ones - you can drive noise out of the communication lines between Product and Engineering. It goes further than that, but fixing the (usually) broken handoff between those two organizations yields enough value on its own to justify the change.

Agreeing with this post is the easy part. Making the change is the hard part. There's a social aspect to it and a skills aspect to it.

In the latter category is a hard question:

How do we get really good acceptance criteria?

More on that to come.