liveBook · Manning

Experimentation

As explained in Build High Value Software Systems

A deliberately limited activity undertaken to learn about an uncertain idea or approach.

Experimentation makes uncertainty testable before a team commits to an approach. In architecture, short, throwaway spikes compare candidate designs; in day-to-day work, reversible local changes can test process improvements. section 11.1 section 15.1.1

Two practical forms serve different scales of uncertainty:

  • Run parallel, time-boxed spikes to compare options against shared questions before committing to an architecture. section 11.1
  • Try reversible changes in your own workflow, measure the result, and share it without requiring others to adopt it. section 15.1.1
Table 11.1 Notification delivery spike comparison
Question
Spike A (Redis Pub/Sub)
Spike B (Kafka)
Spike C (PostgreSQL Outbox)
Can it deliver 5,000 notifications/sec?
Yes, very fast in-memory
Yes, with partitioning
Moderate, needs polling or NOTIFY
What happens when a consumer is offline?
Messages lost
Messages retained on disk
Messages retained in the outbox table
How complex is the operational story?
Simple but fragile
Cluster management, ZooKeeper/KRaft
Familiar, same DB the team already runs
What does the team already know?
Some experience
No experience
Deep experience
What's the delivery latency at p99?
< 5ms
10-50ms
50-200ms

Compare Redis Pub/Sub, Kafka, and a PostgreSQL outbox across throughput, offline consumers, operations, team experience, and latency. section 11.1

Test a process change

A team tried automated formatting as a pre-commit hook for two weeks. Formatting-related review comments fell to zero, helping reviewers focus on substantive issues. section 15.5.1

Listing 15.1 Eimear's team message after one week of automated deploys
Subject: Staging deploy script: 1 week of data
 
I've been using a script to automate staging deploys this week.
Here's what I measured:
 
  Before (manual):  avg 17 min/deploy, 4 deploys/week = 68 min/week
  After (script):   avg 3 min/deploy,  4 deploys/week = 12 min/week
  Saved:            56 min/week across my deploys alone
 
The script handles image pull, config update, migration,
restart, and Slack notification. Log tailing is still manual.
 
It's in /tools/deploy-staging.sh if anyone wants to try it.
No pressure, just sharing what worked for me.

See how a developer shared measured staging-deploy savings while leaving adoption optional. section 15.1.1

How Build High Value Software Systems covers it

How deep it goes
Explains spike purpose and evaluation criteria, then compares trade-offs; prototype implementation details are not shown.
Examples
Uses notification-system spikes, a staging-deploy script, and a pre-commit formatter trial.
What you can do after
Evaluate competing architecture options through spikes, or test local improvements and share measured results.
How it connects
Connects experimentation to set-based design, architectural decisions, Kaizen, and voluntary adoption.

Also explained in

Compare how 20 books explain experimentation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build High Value Software Systems. The section links point to the parts of the book it draws on.