Some updates to my NuGet packages

A while ago, I released a few SpecFlow plugins on NuGet to make it easier to reuse SpecFlow feature files.

I decided that one of the features of one of the plugins really ought to be its own plugin. So I extracted it to this package: HexagonSoftware.SpecFlowPlugins.FilterCategoriesDuringGeneration.

The new plugin is a lot more powerful than the old tag-stripping feature and works in a much more meaningful way.

The old feature literally stripped tags out of a scenario or feature. That means assumptions made about tags present in generated tests could be violated. If you wrote a binding that was coupled to the presence of a tag and then stripped that tag out (because you use it as a control tag), the binding would cease to be found.

The new plugin doesn't have that problem. It works by decoupling test-category generation from the tags in a scenario or feature file.

It has two opposing features: category-suppression and category-injection.

Suppression (right now) works by applying a set of regular expressions to each tag. If any of these regular expressions matches, category-generation is suppressed for that tag.

Alternatively, injection adds one or more tags to every test generated for a particular project.

The exact details can be view in the readme.txt that pops up when you install the package but I'll give a brief example, here. As with all my SpecFlow plugins, it is controlled by a .json file.

Here is an example from one of my test assemblies:

{
  "injected-categories": [ "Unit Tests" ],
  "suppress-categories": {
    "regex-patterns": [ "^BindAs:", "^Dialect:", "^Unbind:" ]
  }
}

The regex patterns for suppressed-categories each select a different prefix. If that's not obvious to you, here's my favorite source for exploring the regular expression syntax. Anyway, this allows me to simply and concisely select who families of tags for which category-generation should be suppressed.

The injected categories are injected directly as categories into generated tests verbatim.

What this does, when paired with the other plugins that allow for (tag-based) selective sharing of scenarios among multiple test assemblies, is allow me to "rewrite" the categories for a generated assembly.

Even though I use tags to select certain dialects into particular scenarios and to control which scenario manifest in which gate, this is what my unit test list looks like when it's grouped by category, like so.

A clean categories list despite my use of dozens of tags for various control purposes

As you can see above (and, in more detail, below), un-rewritten tags still show up as categories. So a single test might end up in two categories but only when that's what I really want.

WillTakeDeathStepQuickly and WillTakeKillShotQuickly both show up in Unit Tests and Slow

As with all things on NuGet.org, this plugin is free. So, if you think it can help you, you don't have much to lose by downloading it and giving it a try.

Here's a little time-saver for managing NuGet packages in your solution

This entry assumes you use Microsoft Visual Studio. If you don't use that, please bookmark this entry so you can revisit it when you've come to your senses.

I like to tend to my NuGet dependencies a lot and my preferred view for doing so is the "Manage NuGet Packages for Solution" window. This can be accessed from the context menu (below), the tools menu (further below), or the search bar (third below).

The first option is pretty quick except that you have to scroll up in your Solution Explorer. So it's kind of disruptive. You also have to fish through a meaty context menu but it's near the top and in a predictable way. So, with the power of habit, this one can be made reasonably cheap.

The tools menu is about as useful as any menu bar in a large application like Visual Studio (Visual Studio desperately needs a ribbon, like Office has). I never use it.

I've been trying to train myself to type Ctrl+Q and then find the option but, for some reason, it just won't take. Even with my trick for retraining muscle memory, I just couldn't make myself do it the new way.

The reason I couldn't make myself adopt the new way is that the new way and the old way both suck. The right way to do this is to have a button I can push without any preceding navigation steps. Like a custom toolbar.

I'm assuming you know how to make a custom toolbar in Visual Studio. Actually, I'm just assuming you are smart enough to figure it out because it's super easy.

What's not easy was finding the damn command because it has a nonsense name in the toolbar editor dialog. That's not really the fault of the dialog - a lot of commands show up correctly. It's something about the way the NuGet commands are configured.

Here is the name: cmdidAddPackagesForSolution

It shows up correctly in the toolbar, just not in that editor dialog.

If you ever need to find a command that is improperly named, the trick is to pretend to edit the menu, first.

Go into the Customize dialog for the toolbar you are trying to edit and then switch to editing the menu group where the command is already located.

When you've done that, you'll be able to see what the command's name is.

Recent improvements to my NuGet packages

I've improved my NuGet packages for sharing feature files and constraining test-generation.

As of 2020.4.2.5, the following changes have been made:

• Feature files are imported to a hidden folder under your base intermediate objects path (usually obj\). This helps make it more clear that the imported feature files are build-artifacts, not maintainable source code.
• Control tags for other test libraries can be stripped out of the test-generation process. This will help prevent miscategorization.
• Shared scenario templates/outlines which were being broken by the incredibly-offensive importance of this class now work properly.

Fun.

The road to pipeline-as-code on Azure DevOps

I've been setting up and maintaining continuous integration builds for a long time. Back when I started, pipeline as code was the option.

Then newer, "better" options that were mostly configured via a GUI were offered.

Now the industry has come full circle and realized that pipelines are automation and automation is code. Ergo, pipelines should be code.

Dwindle was born before that realization had swept its way through the majority of the industry. Most importantly, before the option of a pipeline as code was even a little mature in the Azure DevOps offering.

Now, however, it seems the idea is pretty widely accepted. Again, more importantly, it's accepted - even evangelized - by Azure DevOps. The trip for "that's a good idea" to near feature-parity with the GUI-based pipelines has been dizzyingly fast.

So I had an opportunity to convert my UI-configured pipeline in a code- (YAML-)configured pipeline.

That effort was not without its obstacles. The "get the YAML for this job" button doesn't really work perfectly. Service connections are persnickety. Although: they were kind of difficult to deal with in the UI-configured world, so maybe that's not fair to attribute to switching to YAML.

Most significantly, though, the unification of builds and releases into single pipelines represents a nontrivial (but good) paradigm shift in how Azure DevOps expects developers to shepherd their code out to production.

Previously, I had to introduce some obnoxious kludges into my system that I have replaced with simple, first-class features of the modern pipeline-definition system.

For instance, I used to enter "red alert" work items into a work-tracking system whenever validating a blue environment failed. These red alert work items would prevent any promotions of any environment until they were cleared, which happened automatically at the end of the successful validation of a replacement promotion-candidate. This meant, among other things, that my pipeline was coupled to my work tracking system.

As a result, validations happened in a haphazard way. One promotion could, theoretically, occur if validation along another stream of deployment/promotion failed later.

Likewise, the way I had to marshall build artifacts was a pain. I used to have to download them in one build only to re-upload them so that the next build could access them from its triggering pipeline. That's a lot of wasted time.

Stages, dependencies, and pipeline artifacts changed all that. Pipeline artifacts allow me to upload an artifact one time and download it wherever I need it. Stages and dependencies allow me to ensure all of the following:

• All tests - unit, server, and client -  happen before any deployments to any blue environments.
• All deployments to blue happen before any environment-validation steps.
• All validation happens before any promotion.

The Environments feature makes it easy to track and see at a glance which deployments and promotions have occurred. They also give you a place to introduce manual checks. For instance, because I'm resource-constrained, I only want deployments to blue environments to happen when I approve. Likewise, I only want to promote to any green environments after I've given the okay.

The transition has been largely positive. As a single-person operation, it's made my life easier since I completed it.

As I said, though, it came with challenges.

I will be exploring those challenges and their corresponding rewards in near-future blog entries.

Here are two new NuGet packages for SpecFlow you might enjoy

Hey all. I made my first two packages on nuget.org, today. I must say, it's a lot easier than it was the last time I looked into it. No .nuspec file is required. Uploading at the end of an Azure DevOps pipeline is a snap. The NuGet part of the problem is officially painless.

If you want to cut to the chase, the packages are here:

1. HexagonSoftware.SpecFlowPlugins.ImportSharedFeaturesDuringGeneration
2. HexagonSoftware.SpecFlowPlugins.GenerationFilter

I think it makes sense to explain what they do, what they are for, and how to use them, though.

Import Shared Features During Generation

The former is not really a plugin for SpecFlow so much as it is an extension of the .csproj MsBuild ecosystem. It allows you to designate a reference to a set of feature files. Each such reference points to an external folder (presumably full of feature files) and maps it to a folder inside the project.

This is accomplished by editing the .csproj file and adding a fairly-standard-looking reference to it. Here's an example:

<SpecFlowFeaturesReference
  Include="..\HexagonSoftware.SpecFlowPlugins.ImportSharedFeaturesDuringGeneration.Import"
  ImportedPath="SharedFeatures" />

That will cause all the feature files under ..\HexagonSoftware.SpecFlowPlugins.ImportSharedFeaturesDuringGeneration.Import (relative to the project root, of course) to copied to SharedFeatures (again, project-relative) prior to test-generation.

The internal folder (in this case, SharedFeatures) is completely controlled by the reference. Everything in it is destroyed and rebuilt every build. For my own sanity, I add those folders to my .tfignore (or .gitignore, if I'm feeling masochistic).

Unfortunately, at this time, the best way I was able to get it to work was by making a folder under the project root. In the future and have the files actually be a part of the project while generation occurs. This has a little to do with how difficult it was to access the internals of the SpecFlow generation task from a plugin and a lot to do with how difficult actually getting a reference to the task assembly is.

I'll probably crack it eventually.

I'm sure there are many cases I haven't considered but, of course, the ones I have considered are tested.

Generation Filter

This plugin allows you to control which parts of a feature file are actually generated into tests. You do this by specifying sets of tags that include or exclude scenarios or features from generation.

The tag selection is controlled by a JSON file at the root of your project. The JSON file must be named "specflow-test-filter.json".

Its format looks something like this:

{
  "included-tags": [ "@In" ],
  "excluded-tags": [ "@Stripped" ]
}

As it should be, exclude tags always trump include tags.

Why Both?

These two plugins work together nicely. The first one allows me to reuse feature files. The second allows me to generate a subset of the scenarios within a project. As a result, I can create the SpecFlow equivalent of a "materialized view" within my test suite. Each test assembly can be a subset of all the tests I use for Dwindle.

Before, I relied on environment variables to select tests and set the test mode. Now, the place that a feature file is instantiated sets all the context required.

This worked perfectly in my automated gates. Maybe it was even convenient. At the very least it teetered on the edge of convenience but it was a real pain in the ass for my local development environment.

For one thing, I had to fiddle with environment variables if I wanted to switch between unit, client, or API tests. I was able to kludge my way past that, though: I have three different test runners installed in my Visual Studio instance and each one is configured differently.

Another problem - probably a more important one - is that it made it hard for me to push a button and run all the tests. As I migrate over to using these plugins, I'll be able to run all my tests in a single batch.

Before, the only tests that were convenient to run locally were the unit tests. Those were so fast that I could run them any time I wanted. Everything else was a batch of tests ranging from 30 seconds (just long enough to be infuriating) to ten minutes.

When I'm done migrating to this structure, I'll have a choice. I can right-click and run whatever test I want in whatever context I desire. I can select a ten-minute batch and go get some coffee. I can set up a run that takes an hour and go for a swim or walk.

I'll probably circle back on this with an experience report when the migration is done. My automated gates will all have to change (at little). I'm guessing, in the course of the migration, I'm going to need to add a few more features and refine the workflows a little, too.

Maybe it won't help you the way it (already) helps me but I figured I should build this as NuGet packages for the next person with the same problem.

What if we rethought what a process framework should be in the first place?

Okay. Now I want to make a crazy suggestion.

If you've been tracking what I've been saying, this week, then you've probably read as far as this entry, I which I suggest that the basis for bad technical-decision-making is probably a poor understanding of the relative costs of various decisions. I also made the argument that large, complex corporate budgets and success-metrics tend to confound the problem and make it even harder for some decision-makers to make good long-term decisions.

Now I want to explore process frameworks and how they play into this.

I'm not a deep expert on any of these process frameworks but I know plenty about several of them. In fact, I'm probably as much of an expert as a lot of the people who present themselves as experts. I just don't claim to be.

I'm going to list a few frameworks but I think my argument is true for all of the frameworks of which I am aware:

• Scrum
• SAFe
• Kanban

That's enough to make the point. At least from context, I've heard evangelists for each of these process frameworks make the argument that businesses need to change how they measure.

My first experience with this was when scrum trainers told my team to use Story Points for estimation rather than hours. The idea being that we could empirically determine what a story point means for capacity planning and get better at estimating.

Great.

It didn't work, of course, but great.

It still represents an early attempt to shake up how software projects are measured.

Then came Business Value Points. That oughta do the trick, right? Of course, it didn't, but at least people were trying.

Story Points and Business Value Points both generally failed to have a positive impact (within the scope of my observation) because they were attempts to make the traditional spreadsheets work. They were stand-ins for other numbers that were already being used and empirically backing into a meaningful value was supposed to fix...something...but it couldn't fix the corrupt idea at the heart of it all; the model of software development that is, to this day, essentially descended from manufacturing.

To address those ideas, new processes and mentalities were brought to bear. Plan this way, keep these things in a backlog, manage that kind of work in process, favor this kind of effort over that one, et cetera.

Then there were people arguing for no estimates. There were (are?) proponents within seemingly every framework. There seemed to be a lot of them in Kanban. So much so that I began to associate the two in my mind.

I'm not sure if that was ever an official part of Kanban for software development. Nor do I care. What I do know is that it was another attempt to address the measurement problem in software development and it was brought to bear as a supporting change for some other goal (an "agile" process, usually).

Scrum and SAFe both have explicit mechanisms for their "inspect and adapt" principle - which many believe to be the core of their efficacy. I view this as an attempt to bring qualitative measurement techniques into software development.

The picture I'm painting is that luminaries within the software development industry have been attempting to improve how and what organizations measure for...well...decades, now.

Yet, it always seems like it came as a sideshow, though. It always seemed like there was a shrink-wrapped process that (some people claimed) was going to solve everything and, also, maybe, kinda, you might need to tweak how you measure a little bit.

What if we flipped it on its head? What if we defined a new paradigm for managing software development organizations with a radically different message?

The message, right now, is "You know what's going on, we'll help you do a better job."

I think the message should be "You're going a good job, let us help you know what's really happening."

I think what we need is not process frameworks with a side of metrics. We need metrics frameworks and you can buy a process framework à la carte if you decide you need one.

We could start with "Keep managing how you manage but change how you measure" and see where that gets us.

I think it might get a lot more traction because it's more respectful of the skills that most software development leaders do have. Most of them do know how to manage large groups of people and work with large complicated budgets.

Even if it didn't get widely adopted, though, I think the organizations where a metrics framework landed would benefit a lot more than the ones that are trying to adopt or grow a new software development lifecycle. They'd get more out of the effort because we'd be addressing the real kernel of the problem: Not what we do but what we see.

We need to start working with leadership teams, and maybe even boards, to help them define metrics that set software "projects" (products) up for success. We know these things are going to last a lot longer than a quarter, so quarter-to-quarter measurements aren't going to do the trick. Let's get together as software professionals, define a standard for how success and progress can really be measured in software development, and then start pushing that.

The industry needs to change how it measures and plans and there's no way it will be able to do so without our help.

A deeper exploration of why people might make bad technical decisions

Yesterday, I wrote about budgetary forces impacting technical decision-making.

Everyone just reading that sentence probably is thinking "Well... duh!"

However, I don't mean it in the way a lot of people do.

My argument was that big, padded budgets create a sense of being able to afford bad decisions, allowing technical debt to pile up. I also posited that small, tight budgets might help people see that they need to make good decisions if they want to stay afloat.

Of course, at the end of the blog entry, I noted that this was just one piece of the puzzle. It's not like every big budget results in bad practices. Likewise, not all of the small budgets in the world result in better decision-making.

It should also be noted that this is all anecdotal and based on my own experiences. Really, that should have been noted in yesterday's entry.

Oh well.

As I considered my position, I realized there was a major crack in the argument: Large organizations can't afford to make bad decisions. The large, complicated budgets only make them think they can. As a result, they often defer sustainability-related investments until the technical debt is so bad that they are in crisis mode. Yet, the margin is almost always called, the crisis almost always comes, and the result is almost always bad.

So it is definitely not about being able to afford an unsustainable decision. Actually, just that phrase is inherently self-contradictory. It reduces to either "afford an unaffordable decision" or "sustain unsustainable decisions", both of which are nonsense statements.

Budget-size is probably not what's at work, here. It's just a clue pointing to the real driver: understanding.

Consider this: A small organization on a shoestring budget that makes good decisions about sustainable practices only does so because its membership understands they can't really afford to make bad decisions. If proper software design wasn't perceived as something that was going to pay off, an organization wouldn't do it.

For every one of those small teams with healthy technical practices, there are probably dozens we never hear of because they are crushed under the weight of their own poor choices.

Why did they make those poor choices? Do people intentionally undermine their ability to succeed knowing full well that's what they're doing? Not usually. Again, those small organizations that winked out of existence too fast for anyone to notice were undone by their lack of understanding.

Now let's look at the big organizations. They go months or years before the crisis hits them and then sunk costs often make them drag on for a lot longer after that. Are there really people sitting around in the leadership of those companies, greedily rubbing their hands together and muttering "This project is going to be toast in three to five years! Heh, heh, heh."

Well. Maybe that happens every once in a while.

Most of the time, though, it's probably just that the decision-makers simply didn't understand the long-term ramifications of the decisions they are making.* They don't understand that what they are doing is going to create a debt that cannot possibly be paid when no more forbearances can be obtained.

Furthermore, you will occasionally find very large firms that really do all the things they are supposed to do - keep the test suites comprehensive and meaningful, regularly reconcile design with requirements, properly manage handoffs, continuously improve their processes, et cetera. From what I can tell, it seems like this is often a result of a handful of higher-up leaders who have a deep understanding of what works in software development.

So all four of the main cases seem, to me, to be dependent on understanding that there is a problem. Large budgets just muddy the waters for decision-makers who don't already have a deep enough understanding of how software development works.

To fix the problem, leaders need to know that pressuring developers to defer writing tests or refactoring to a proper design (among other things) will be the seeds of their undoing.

Why don't they know that, already?

I think that question brings us to a real candidate for a root cause.

So many organizations - especially large organizations - claim to be "data-driven". They need numbers to make their decisions. Not only do they need numbers, but they need numbers fast. It seems like a lot of leaders want to see the results of making a change in weeks or months.

Therein lies the problem.

For large organizations, the consequences of poor technical practices take months or years to produce intolerable conditions. Why should damage that took years to manifest be reversible in a matter of weeks or months? It's not possible.

So long as the way we measure progress, success, failure, and improvement in software development is tied to such incredibly short windows in time, those metrics will always undermine our long-term success. Those metrics will always create the impression that the less-expensive decision is actually the more expensive one and that you can squeeze a little more work out of a software development team by demolishing the very foundations of its developers' productivity.

Not all data are created equal. Data-driven is a fine way to live so long as the data doing the driving are meaningful.

We need better metrics in software development or, at the very least, we need to abolish the use of the counterproductive ones.