1 The need for collaborative software design

Great technical teams working closely with well-meaning business stakeholders can still deliver software that fails to meet an organization’s needs. We see it every day. Software teams become too focused on finding solutions without fully appreciating the context of the organization. Business stakeholders can be reluctant to voice their concerns in technical sessions. Tensions escalate, and conflicts go unresolved.

Effective software requires effective collaboration, and just like every other aspect of software development and design, collaboration is a skill you have to learn and practice. Visual collaboration tools can be a key strategy in doing so because they can clarify assumptions and foster improved collaboration between development teams and stakeholders. But tools alone can’t guarantee effective collaboration; someone needs to guide people during collaboration: a facilitator. Fortunately, there are techniques and practices you can use to facilitate effective collaborative design, whether you’re a project leader, a business stakeholder, or anyone in a team that is involved in building software.

In this book, we’ll guide you toward making sustainable design decisions with the goal of developing quality software systems. We’ll also help you improve collaboration with stakeholders through collaborative modeling. Collaborative modeling is a visualization technique to analyze complex and conflict-laden decision-making processes with all relevant stakeholders and decision-makers to create a shared understanding. We consider everyone involved in the software creation process a designer, highlighting that any designer can learn and facilitate collaborative modeling sessions. Even if you don’t envision yourself as a facilitator, this book will demonstrate how collaborative modeling enhances communication. You’ll learn how your involvement can contribute to better dialogue and understanding through collaborative modeling. In figure 1.1, you can see an example of a collaborative modeling session.

Collaborative modeling is applicable outside of the software industry, but in this book, we’re concentrating on its use within it. There’s a growing trend in which teams are becoming more independent in designing, building, and running software systems. If we don’t enable software teams to develop a shared understanding with stakeholders through collaborative modeling, how can these teams be expected to design, build, and run software that is effective, resilient, and ready to evolve with changing organizational needs?

1.1 Design decisions gone wrong at BigScreen

Welcome to BigScreen, an imaginary company grappling with the challenge of redesigning and modernizing its software system. While BigScreen is a fictional organization, its challenges mirror those in real organizations we’ve worked in and advised for. Various companies have sought our assistance over the years to address the same problems BigScreen is encountering, and we’ll share how we enabled them to overcome these challenges through the adoption of collaborative modeling. Although this book recounts our experiences consulting for companies and leading collaborative modeling sessions, it’s important to understand that anyone in any organization can step into the role of a facilitator if they choose to, as well as learn from what we’ve done, regardless of their job title, as long as they have the support to do so.

BigScreen is an international cinema chain with 80 cinemas located in 12 different countries. With streaming services becoming more and more popular, the company was seeing a decline in cinema visitors, which led to a couple of initiatives to counteract this decline. One of those projects was the modernization journey of the software system. BigScreen wanted to create a campaign called “Anytime, Anywhere,” which would allow customers to purchase tickets wherever they were.

There was just one problem—their software architecture wasn’t adaptable enough to implement the vision of this campaign. The user interface (UI) wasn’t mobile-friendly, and it was so intertwined with the backend that it wasn’t possible to make it mobile-friendly. BigScreen also wanted to create a native mobile app for their customers—also not possible because the business logic spanned both frontend (UI) and backend. The company decided to modernize and refactor the current system.

1.1.1 Understanding the landscape

Before we dive into BigScreen’s modernization and refactoring attempt, we’re going to have a look at the software system and the development team in more detail. The development team consisted of 15 developers, a tester, an architect, a product owner, and a team lead. Five years ago, the company made the switch from a waterfall development approach to an Agile way of working. Instead of focusing on projects with long delivery times, the team started focusing on feature delivery because they wanted to be able to get feedback from the users more quickly.

As for the software system, it has been around for 15 years. As you can see in figure 1.2, the codebase became completely interconnected, making it difficult to understand, maintain, or upgrade. This type of system, lacking a clear architecture and boundaries, is often referred to as a big ball of mud (BBoM; http://laputan.org/mud/). The original design of the system is now unclear, especially because the architect who initially designed it is no longer with the company. Over time, the system has grown through numerous patches, quick fixes, and features added in a rush. With no testing framework in place, the team is hesitant to alter any significant functionality within the system, such as the seat allocation code, due to fear of unintended consequences.

The system itself consists of two important parts, which the company called ticketing, where customers can purchase tickets for movies and events, and the back-office, where internal users can schedule movies, create events, and analyze movie performance. The customers have a profile where they can add movies to their watchlist, write reviews for movies and events they attended, and track upcoming movies and events.

1.1.2 BigScreen’s attempt at refactoring

When we as a team of three were hired by BigScreen, their goal was to get help for the development team to redesign the system so it would be able to handle the “Anytime, Anywhere” campaign the company wanted to roll out. We were invited by Meera, the CTO, for a preliminary conversation. In discussing the team’s current efforts to refactor their architecture, we discovered the following problems:

1.2 BigScreen: How collaborative modeling helped to improve design decisions

In our conversation with CTO Meera, it became clear that the problem wasn’t just about modernizing and refactoring the system. Users in the back-office struggled to make informed decisions because data was either inaccessible or not formatted properly. For instance, the system’s handling of movie playtimes was inadequate. Normally, movies are set to show for two months, but this period might be adjusted based on the movie’s popularity, either extending or shortening the available viewing time in cinemas. The back-office feature designed for this task didn’t meet the users’ needs. As a workaround, they had to rely on developers to run database queries, export the data to Excel, and then proceed from there. Decisions to extend a movie’s run had to be manually entered into the back-office as scheduling a new movie, leading to inaccurately stored data and inaccurate performance metrics. Gathering comprehensive performance data across all theaters was cumbersome and rarely undertaken due to the effort required.

We dug a bit deeper into their approach to understand how they collected information on the functionalities required by their back-office users. It turned out that only the product owner, Ralph, was involved in gathering user requirements. After talking with the stakeholders, Ralph would write a user story for the developers with the implementation details instead of the user requirements. This translation often led to incomplete or incorrectly implemented features.

As often happens during our interviews with clients, we discovered that a rigid architecture isn’t the company’s only or biggest challenge. Merely modernizing and refactoring the system is likely to lead to the same problems. The problems with the current software architecture are directly linked to their software development process. A significant challenge is the lack of collaboration between stakeholders and the software development team. Instead of direct communication, there was only a single hand-off at each step of the process, with developers working in isolation. This method meant the team lacked a shared understanding of user needs and the architecture’s design. As a result, developers often make assumptions and resort to quick fixes to meet user demands, which only makes the BBoM bigger. Initially, we aimed to focus on redesigning the architecture, but we soon recognized the necessity for a broader strategy to support the company.

1.2.1 Our approach for BigScreen

Based on our combined experience, we agreed with CTO Meera to help the software development team with the redesign of the system and to teach them how to make sustainable design decisions that aligned better with the business via collaborative modeling. We discussed the following approach:

Collaborative modeling can bring a lot of benefits when it comes to sustainable design decisions:

Now, as with many other techniques, collaborative modeling involves certain tradeoffs. First, achieving the full benefits of collaborative modeling requires buy-in from stakeholders. Without their participation, especially those with domain expertise, it’s difficult to confirm your understanding of the business problem, leading to alignment within the development team(s) but not with stakeholders. Second, higher management often sees collaborative modeling as a significant commitment because it pulls 5 to 20 people away from their regular duties for participation, making the long-term benefits challenging to measure and leading to the following frequently asked question: How much faster will the teams be able to deliver features because we use collaborative modeling? The answer is complex, as delivery speed depends on more than just alignment problems and architectural rigidity. Finally, collaborative modeling isn’t a one-stop solution for all design and decision-making problems, and it won’t solve all the challenges faced during software development immediately.

Throughout this book, we’ll share stories from our collaborative modeling sessions at BigScreen, emphasizing the benefits they offered in meeting BigScreen’s goals. But before we get into those details, we’ll skip ahead and show you the architecture we ended up with.

1.2.2 The new architecture

After the initial interviews, 3 kickoff sessions, and 12 follow-up sessions, we settled on a new architecture that has two major improvements, as shown in figure 1.3. First, this architecture has a clear boundary between the ticketing application and the back-office. We also found a better name for the back-office, planning and scheduling (PaS), because the back-office can mean many things. Separating ticketing and PaS allowed the software team to deploy the two systems semi-independently from each other, making it possible to gather feedback from and adjust the features for the internal users faster.

Second, a clear distinction between client and server was introduced, and all business logic was moved server-side for the ticketing system. This makes it possible to create a dedicated mobile app that can use the server side as well, which previously wasn’t possible. Finally, a separate service for scheduling movies was introduced containing the business logic both systems relied on, avoiding duplication in the two applications.

We collaboratively designed this new architecture, drawing from the diverse ideas of two developers who initially had conflicting views on how to update the system, together with the original architectural concepts from an architect who had left the company. Throughout multiple collaborative modeling sessions at BigScreen, the stakeholders and every team member, including those two developers, voiced their ideas and proposed solutions. Chapters 5 to 11 will provide detailed insights into these sessions.

With a broad array of stakeholders present, including crucial decision-makers, the sessions successfully avoided turning the developers’ disagreement into a personal problem, which had been a challenge previously. The aim was to find the most beneficial strategy for the organization, a task made easier with the decision-makers present in these sessions. The resulting architecture brought together the best suggestions from the developers and the insights of the previous architect, which were further enriched by the team’s collective learning from resolving their conflicts. This collaborative effort led to more streamlined decision-making and less need for later adjustments. In terms of software architecture, it allowed the team to take full ownership, removing the need for an architect outside the team and leading to less refactoring due to wrong assumptions, increased team engagement, aligned mental models, and well-defined boundaries. As consultants, our job was to do the following:

BigScreen kept using collaborative modeling internally because the company experienced the benefits it brought. Collaborative modeling gave BigScreen a better way to understand the organization’s needs, enabling developers to create better solutions for the users. Because the stakeholders were now actively involved in the development process, the quality of the features improved as did the relationships between the software developers and domain experts.

1.2.3 A brief history of software design

Part of our job as consultants is to facilitate collaborative modeling sessions between engineers and stakeholders. During these sessions, there is a lot of individual and group behavior that results from the interaction between people. We refer to that behavior as social dynamics. We’ve noticed that most engineers and developers struggle with these social dynamics when introducing collaborative modeling in their teams or organizations. This makes a lot of sense because most engineering education programs don’t include much social sciences instruction. We believe those struggles are one of the main reasons most engineering teams aren’t successful at starting collaborative modeling.

We think one of the causes has something to do with where we, as an industry, came from. Twenty years ago, the Waterfall model was the de facto standard for most companies developing software. The Waterfall model splits the activities of building software into small sequential parts. It was developed by copying the manufacturing and construction industry’s project management method. Waterfall was successful in that industry because it gave the client certainty of costs and time; however, as a downside, it left little room for changes. This model was introduced in the 1970s when most computer programmers created, edited, and stored their programs line by line on punch cards. Changes after the design were costly and took time, so the Waterfall model made sense.

1.2.4 The Agile theater

Winston W. Royce, in his 1970s paper titled “Managing the Development of Large Software Systems,”¹ was the first to write down a detailed diagram of the process without calling it Waterfall, which you can see in figure 1.4. However, he also felt it had significant flaws stemming from the fact that testing only happened at the end of the process, which he described as “risky and invites failure.” Fast-forward roughly 50 years, and we still base most of the software development approach on the same diagram. Yes, companies did transform to an Agile way of working by splitting up the work into smaller batches and delivering those smaller batches to the users faster so they were able to adapt quickly to the users’ feedback. Still, we observe that it’s usually an Agile theater, implementing SCRUM in the development teams, but keeping the hierarchies and the idea-to-production value chain in place; only the development teams are doing Agile development, whereas the business side of the company maintains a very Waterfall mentality, like the situation BigScreen is in.

We believe this mindset is the result of how software engineering is perceived or misunderstood by people who need software to run their organizations. These organizations want to know how long it will take and what the costs are, similar to the manufacturing and construction industries. They let architects gather the more technical requirements and design the system, and then let those designs be built by software teams. When someone without an understanding of Agile software development with a Waterfall mentality wants software to be built, they will perceive any Agile artifact—such as collaboration—as a waste of time. That person sees it as a waste of time because, in their mind, they just need to give a list of requirements to calculate the costs and estimate the work, and then let it be built.

One of the main differences between designing and building in the manufacturing/constructing industries and the software industry is the rate at which requirements change. For example, if we want to construct an office building, we can gather requirements and design the building up front. During construction, hardly any changes are needed in the design. There might be adjustments, but the main purpose and functions of the building scarcely change after design. This is different from most software systems we develop. As soon as a user interacts with a new feature of the software system, that feature might be exactly what they want, might have potential but need improvement, or might not be what they want at all. Usually, it’s the latter two because we’re building a solution for the needs of our users, which hasn’t been built before. Unlike constructing office buildings, a practice refined over many years with established good and best practices, in the software industry, we’re mostly designing and building software that hasn’t been built before. If it had been built before, we could just copy and paste it or just buy it off the shelf. So, you can expect a lot of new insights once users interact with our software. Most of the time, when building software, we can only see a clear solution in retrospect.

Building software requires a different approach to design than we use in manufacturing and constructing. Designing software requires taking an Agile approach to architecture, which involves collaborating on design, experimenting, following emergent design principles, prototyping a solution, and enabling the engineers building the software to understand the problem.

When organizations start to work in an Agile way, most of them don’t know what to do with the software architects, and, usually, the software architects keep designing and gathering requirements in a Waterfall way, which can create a bottleneck for the development teams. The teams either are forced to wait on the software architect or start building the software without designing the software architecture because it’s not considered Agile. This is what happened in BigScreen—the software architect left, leaving the company without any software architecture knowledge in the team. However, a key aspect of Agile also means that the team has the knowledge to take ownership of their software architecture.

1.2.5 Enabling teams to do collaborative software design

When teams take control of their software architecture, they bypass the potential blockers in the flow of development that come from depending on an external architect. However, eliminating architects outright could leave a substantial gap in architectural knowledge within the team. It’s important, then, to provide support that enables teams to make architectural decisions independently. Collaborative modeling stands out as an effective method for this purpose because it’s straightforward, it’s easy to engage with, and it promotes working together on the software architecture.

Engaging in collaborative modeling brings to the surface the social dynamics we’ve mentioned before. We’ve personally faced challenges where discussions stalled, decisions cycled without advancement, or a supposed consensus led to delays because, in reality, there was no broad agreement. For instance, at BigScreen, we saw disagreements between two people over the architecture bring progress to a standstill. Managing these social dynamics during collaborative modeling is vital for reaching well-supported design decisions. These are decisions that harness the collective knowledge of the group, maximizing the decision’s effect and earning the backing of the entire team. This critical aspect of dealing with social dynamics in collaborative modeling isn’t discussed or taught enough in software architecture.

The software architect, acting as a facilitator, can significantly change these social dynamics. As individuals already engaged in the organization’s software design, they are well placed to serve as enablers. They can catalyze teams and the broader organization to embrace Agile methods in designing software. Software architects can facilitate teams in making design decisions, provide advice, and avoid imposing designs on them. This is accomplished by initiating collaborative modeling sessions with teams, thus allowing them to collaboratively design software with stakeholders. This approach shortens the feedback loop from requirements to deployment, enabling quick, timely, and reliable value delivery to users. The software architect acting as a facilitator isn’t enough. The architect should also teach the teams about collaborative modeling so that the team isn’t reliant on the architect anymore and the team can run these collaborative modeling sessions on their own. When teams are capable of designing software independently, it removes the bottleneck of software changes and frees the software architect to focus on maintaining alignment within the team and with the organization’s vision and goals, thereby enhancing the agility and efficiency of the software development process.

We won’t need to enable teams to start collaborative modeling; starting within the team itself can often be more beneficial because it allows team members to make mistakes and learn in a less risky setting. Successes achieved internally can motivate other teams to adopt similar practices and eventually encourage stakeholder participation.

As we write this book, we’re at a pivotal moment in the software industry as we transition from software architects designing the architecture to teams taking ownership and doing software architecture without an external software architect. Now you might think, I’m not a software architect, so will this book help me? Rest assured, it will. The concepts discussed are applicable within any software team context. We’ve successfully introduced these practices to a wide range of roles, not just to software architects, including software engineers, testers, product owners, product managers, CTOs, engineering managers, user researchers, and business analysts. Anyone involved in the software development process is seen as a designer, capable of both facilitating and participating in collaborative modeling. However, as long as someone in the session has experience in software design and architecture, anyone can lead collaborative modeling sessions. Thus, everyone is capable of enabling their teams and others to undertake collaborative modeling and manage the associated social dynamics. The goal is to enable software teams to collaborate with their stakeholders, understand their needs, and allow this understanding to shape their software architecture, which is the essence of teams engaging in collaborative software design!

1.3 Collaborative software design as a catalyst for better design decisions

This book is about looking at software design in a new (or different) way. Collaborative software design isn’t just from a technical perspective but also from a business perspective, all while taking into account the social dynamics that affect both collaborative modeling sessions and design decisions. This is what makes this book different; we didn’t find many books on software design and architecture that talk about the effect and opportunity that social dynamics have when we need to adapt when a company strategy or business strategy changes. From experience, this is valuable and relevant information, which is why we ended up writing this book. In this section, we’ll give you an overview of what to expect in the coming chapters.

1.3.1 Collaborative modeling, design, and architecture

As mentioned earlier, we see it as vital for teams to take ownership of their collaborative design decisions—a lesson learned from less-than-ideal experiences. We experienced frustration when provided with an unrealistic solution by an architect from above (in the famous ivory tower meme). Our valuable suggestions, unknown to the architect, were overlooked. This led to software that failed to satisfy us or our stakeholders, a situation similar to what BigScreen faces today. Our goal was to shift this dynamic; we aimed for teams to reach their highest potential and to develop software that they and their stakeholders could be proud of. Collaborative modeling became the key to initiating this change. We witnessed the method’s capability to unlock a team’s full potential and guide us toward better models and solutions.

Figure 1.5 offers a visual representation of a collaborative modeling session, similar to the session photo shown earlier in figure 1.1. Chapter 2 dives deeper into what collaborative modeling entails, discussing the various tools used within the modeling space. Throughout this book, we share stories from our collaborative modeling sessions at BigScreen, using a range of tools designed for different needs. We provide insights into our firsthand experiences with collaborative modeling tools, including the Business Model Canvas, several types of EventStorming, Example Mapping, domain message flow modeling, Wardley Mapping, Domain Storytelling, and others.

Figure 1.5 also features the facilitator as the catalyst, enabling the group to make sustainable design decisions. Chapter 5 dives into the facilitator’s role and the essential competencies and skills needed for collaborative modeling. Continuing in figure 1.6, we see the same facilitator next to a table with the heuristics toolkit on top, which chapter 3 discusses in detail. Throughout the book, we’ll share some of our heuristics, hoping to inspire you when creating your own. Also shown on the table are various collaboration styles that a facilitator might employ to guide the group’s efforts, as discussed in chapter 4: Together Alone, Split and Merge, Small Group Diverge and Converse, Liberating Structures 1-2-4-All, Ensemble, Fish Bowl, Anarchy!, Guerilla, and The Secret Modeler.

Together with all the principles and practices mentioned throughout this book, the heuristics toolkit and collaboration styles act as the chemicals the facilitator uses to manage the social dynamics of collaborative modeling. Mixing these chemicals appropriately is crucial to align with the group’s needs and to structure the sessions effectively. Facilitating collaborative modeling involves numerous decisions: choosing the appropriate chemicals for the group, determining the most suitable style, using the right heuristics, and so on. This requires many skills and figuring out the best way to combine them. All of these considerations require careful preparation, foresight, and flexibility. A misstep in mixing these chemicals can disrupt the group dynamic, akin to a chemical reaction causing an unintended explosion, potentially leading to significant group tension. Collaborative modeling sessions can be intense, and the incorrect mix of tools and styles can profoundly affect participants in a personal way.

This is the reason we’re writing this book—to guide you in effectively selecting and blending these chemicals yourself. We’ll provide you with the fundamental chemicals, but it’s essential for you to explore which chemicals work for you and experiment with using and mixing new principles and practices, heuristics, tools, and styles. Getting comfortable with these tools and trying them out in a safe environment allows for learning from mistakes without serious consequences.

1.3.2 Collaborative modeling ingredients and potential benefits of facilitation

Collaborative modeling sessions are designed to be fluid, letting us tweak the approach by introducing different chemicals as needed. Although the sessions are dynamic, collaborative modeling has several ingredients and stages to guarantee the effectiveness of each session and its follow-up activities. In chapters 4 and 11, we’ll explore these ingredients in more detail. We try to avoid calling collaborative modeling a process because the dynamic of collaborative modeling is nonlinear, so we see it more as a framework of stages. We’ll cover the stages for preparing a session, conducting a check-in, leading the session, performing a check-out, holding a retrospective, and communicating and recording the outcomes and results in various ways. This approach is depicted on the poster shown earlier in the top-left corner of figure 1.6, similar to a periodic table poster in chemistry labs. Essential chemicals and ingredients are integrated into the stages, ensuring the collaborative modeling group can successfully develop their shared understanding.

1.3.3 The effect of social dynamics on collaborative modeling sessions

Facilitators mix specific chemicals to handle the complex social dynamics within a group, such as social rank, cognitive biases, conflicts, and polarities. These dynamics can disrupt the progress of a session, blocking the group from moving forward and making well-informed design decisions. Observing these hidden dynamics is compared to shining a flashlight on the group that shows the shadows, as depicted in the model in figure 1.7. If not properly addressed, these shadows can prevent the group from using its collective potential, knowledge, and wisdom to create a shared understanding and come up with sustainable decisions. Addressing these shadows can be challenging for a group, which is where the facilitator comes in, making the group aware of these shadows and assisting them where needed in tackling them. Ignoring these underlying shadows allows them to intensify and become demons that emerge and form significant barriers within the group, significantly affecting the outcomes.

Chapters 6 through 10 are dedicated to observing these shadows and providing strategies for facilitating the group to manage them. Topics such as conflict, ranking, cognitive bias, resistance, and polarities are examined, with techniques offered for addressing these social dynamics and the unseen aspects within the group. Facilitators must understand that groups may not always be ready to confront their shadows. Although you might observe what’s holding the team back, the decision to address these problems rests with the team itself.

These shadows can also influence us as facilitators if we aren’t careful. For those who naturally lean toward solving problems, it can be frustrating to see a group avoid confronting their problems. It’s important for us to manage our own shadows to stay as neutral and provide the high-quality facilitation the group needs. Losing too much neutrality as a facilitator can inadvertently strengthen these shadows, turning them into demons and preventing us from effectively assisting the group. We’ll delve deeply into observing the various shadows present within ourselves and in the group in chapter 5. Addressing these shadows successfully hinges on our ability to create a psychologically safe environment where they can be tackled. This book will extensively cover this topic, and you’ll find discussions on it throughout most chapters, highlighting its critical role.

1.3.4 Collaborative decision-making

The shared model that emerges from collaborative modeling sessions becomes a key tool for teams and stakeholders to engage in collaborative design decisions. These decisions can involve changes to the business model or strategies for constructing solutions. Chapter 9 will go into greater depth on decision-making theory and how to enable collaborative design decisions to ensure that decisions are supported across the entire group. This doesn’t imply that all group members must be present for each decision or have decision-making authority, but it’s critical to involve participants in the process to ensure they are on board with the decisions made. While the ideal scenario involves everyone in decision-making, practical time constraints might not always allow for this.

Addressing the follow-up of making collaborative decisions, it’s also essential to formalize these decisions for future reference and clarity. Chapter 11 discusses how architectural decision records (ADRs) and diagramming serve as a documentation tool for capturing all aspects of architectural decisions, including the context, the decision itself, and its consequences, to avert future confusion and facilitate sustained decision-making. Yet, it’s important not to focus too much on documentation. The real outcome of a collaborative modeling session isn’t just the visual model produced but the shared understanding and collaborative design effort it generates. This collaborative approach is what leads to the design and development of software that is both sustainable and capable of adapting to the organization’s evolving needs.

This introduction, along with the combined mental model from figures 1.5 to 1.7, which we detail in the appendix, provides an overview of the discussions to be found in this book. Note that this book isn’t the definitive resource on collaborative software design but rather an initial step for digging into this topic.

Summary

¹   www.praxisframework.org/files/royce1970.pdf.