Introduction

Domain-driven design (E. Evans, Domain-Driven Design, 2004), extreme programming (XP) (K. Beck, Extreme Programming Explained, 2004), Scrum (K. Schwaber, Agile Project Management with Scrum, 2004) and service-oriented architecture (SOA) (T. Erl, SOA: Principles of Service Design, 2007) are most widely accepted agile development practices. Some may think SOA isn’t one of agile development practices. But no one probably can deny that SOA has been helpful to increase agility of enterprises in business-IT alignment, and has become the standard architecture style. So it belongs to a broad sense of agile development.  Today these agile development practices are combined with design thinking (T. Brown, Change by Design, 2009), Lean Startup (E. Ries, The Lean Startup, 2011), DevOps (J. Kim, et al., The Phoenix Project, 2013), microservice architecture (MSA) (S. Newman, Building Microservices, 2015), BizDevOps practices, etc.  All these practices have a commonality. They all try to create the maximum value for the users of the software with minimal time and cost, and do so by releasing working software increments frequently to users, learn from the users’ feedback and adapt the software to their needs. So these all can also be regarded agile development practices.

The essential characteristics of agile development include value proposition, continuous and Lean delivery, validated learning, and software engineering competency.  If your version of agile development lacks any of these essential components, you have misunderstood what agile development is about.  A successful adoption of agile development requires a mechanism that allows you to nail down the specific value proposition to be delivered by your software to the users, a mechanism of Lean production and continuous delivery of zero-bug software increments in a fast release cycle, a mechanism to learn from customer responses, and strong competencies in software engineering capabilities that make these three mechanisms not just possible, but globally competitive.

In this page I will discuss the essence and art of agile development.  The content of this page is a subset of my book "Essence and Art of Agile Development" available in the Amazon marketplace both in print and as a Kindle eBook.  (Click this to see the book: https://www.amazon.com/dp/1792855389https://www.amazon.com/Essence-Art-Agile-Development-Enterprises-ebook/dp/B07MKT6K4F/ref=sr_1_1?ie=UTF8&qid=1546320143&sr=8-1&keywords=Essence+and+Art+of+Agile+Development)

 

Reading this page will make you rethink about agile development as a business strategy and as a business process that help you survive and succeed through digital transformations.  It will let you understand that software talent management—an HR management process—is fundamentally important for agile development, and so are the product management process in software product businesses, and the service engineering and service engagement processes in professional IT service businesses.  Agile development happens in the context of a business, and the business processes surrounding the software development should have been properly reengineered to be supportive of agile development.  It will be shown in this page that many practices recommended in agile development are indeed very natural consequences of ordinary process reengineering efforts that have been observed in manufacturing and service industries since 1980s.  Agile development practices adopted common business process reengineering (BPR) patterns such as concurrent engineering, empowerment, job enrichment, flattened organization, Lean production, upstream shift, self-service, etc., as will be explained later in this page.

Comprehensive understanding of agile development will help you set up comprehensive initiatives to adopt and improve your agile development practices successfully.  One of the most important things to learn from this page is to understand what knowledge, skills, practices, techniques and tools software project team members need to be trained in and become capable of, in order to be successful in performing an agile development project.  Depending on the current skill maturity level of your software development organization, you might better develop a long-term road map plan and a strategic organizational transformation plan to adopt state-of-the-art agile development step-by-step.  It should be noted that no one can skip the maturity level. 

Please visit this page from time to time to see the progress in writing, and send me your comments.

Agile Manifesto Revisited

The Manifesto for Agile Software Development (K. Beck, et al., Manifesto for Agile Software Development, 2001. http://agilemanifesto.org/) valued
 
individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan.
 
No one can deny that Scrum has been the most widely adopted practice of agile development. The State of Agile survey by CollabNet/VersionOne in 2017 (https://explore.versionone.com/state-of-agile/versionone-12th-annual-state-of-agile-report) reported that 56% of respondents used Scrum as their agile practice (with the second most used being a hybrid of multiple practices at 14%). It also reported that 88%, 67%, 46% and 35% of respondents did sprint planning, release planning, product roadmapping, and agile portfolio planning, respectively; and that the percentage of respondents who used such tools as agile project management tool, bug tracker, automated build tool, unit test tool, continuous integration tool, requirement management tool, release/deployment automation tool ranged between 74% and 44%.
 
We can see that many people who actually have done agile development followed a very rigorous process such as Scrum. They employed many tools for automating some human activities in software development. They made many plans from long-term plans such as project portfolio plans and product roadmaps, mid-term plans such as release plans to short-term plans such as sprint plans. Process, tools and planning are so common ways how people work when they try to collaborate to achieve certain goals in an efficient manner. Therefore, people developing software in teams would also seek the benefit of well-honed processes, adaptable plans that all team members share, and advanced software tools that can improve productivity and quality.
 
Documentation isn’t always bad either. It’s basically converting implicit knowledge to explicit knowledge so that the knowledge can be stored, searched, classified, shared, reused, continuously improved, taught, and so on. Application programming interface (API) is a sort of documentation, for which most use the language of SOAP or REST web service today. SOAP and REST API documents made service-oriented architecture (SOA) the prevailing architecture style since the mid-2000s, which in turn led us into cloud computing, microservices, DevOps and further into the current era of digital businesses.
Agile manifesto really does not address the essence of agile development that all agile projects should be characterized by. It suggests a direction of change of common practices of software development that were becoming outdated in 2001. Jim Highsmith, for example, said in 2001 on behalf of Agile Alliance (http://agilemanifesto.org/history.html):
 
The Agile movement is not anti-methodology, in fact, many of us want to restore credibility to the word methodology. We want to restore a balance. We embrace modeling, but not in order to file some diagram in a dusty corporate repository. We embrace documentation, but not hundreds of pages of never-maintained and rarely-used tomes. We plan, but recognize the limits of planning in a turbulent environment.
 
Agile development has become a buzzword for modern, effective way of software development. When I talk about the essence of agile development, it really means the essential things that software developers should know and do today. Now, if we look at the Principles behind the Agile Manifesto (http://agilemanifesto.org/), there are more clues about the essence of agile development.
 
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. … Working software is the primary measure of progress. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
 
A keyword I see here is value. We should develop software that provides value to the user. In other words, software should have a value proposition; viz., relieve pains associated with incumbent solutions, and/or create unseen gains to the user (http://www.businessmodelgeneration.com/canvas/vpc). The software that provides discriminant value proposition to a customer will help the customer gain competitive advantage over its competitors who don’t use the software or use other vendors’ software.
 

How do software developers know the software they plan to build has distinctive value to the target customer? This uncertainty requires the developers to pay for the information they want, i.e. the true information about the set of features that have distinctive values to the customer. This information is best obtained when the customer is given a working software that the developers built based on their hypothesis about the valuable features for the customer. The customer’s feedback from actually using the working software is the best source of information to learn valuable features.  This is what Eric Ries called validated learning in his book on Lean Startup in 2011. The Lean Startup method of creating new software goes through a Build-Measure-Learn cycle—a fast feedback loop where a product increment is delivered to the customer frequently to get feedback to learn desirable features.

Design Thinking is widely adopted by software development organizations these days because it helps them run the Build-Measure-Learn cycle effectively. Design Thinking suggests a creative design process that goes through Empathize, Define, Ideate, Prototype and Test (Tim Brown, Change by Design, 2009). Empathize with customers to learn what they really want consciously and unconsciously; make a hypothetical definition of the customer problem to solve; ideate the software solution, perhaps producing user stories, innovative business processes, intuitive user experiences, etc.; then build and release the minimum viable product (MVP) (Eric Ries, The Lean Startup,  2011) many times a day, in order to test the hypotheses and learn from the customer feedback.

Allowing the user to actually experience the use of a solution is the best way to learn real (sometimes hidden) needs of the user. Therefore, prototyping is used in many fields of engineering. In software engineering, prototyping is relatively more productive than in hardware engineering because software can be changed more easily and therefore prototypes can be salvaged into the shippable product. In software engineering, prototyping has been encouraged since as early as mid-1970s. (F. Brooks, The Mythical Man-Month, 1975)  It’s thus not a surprise at all that Agile Manifesto promoted continuous delivery, which is to produce prototypes incrementally based on user feedback in a very fast cycle, maintaining a constant pace indefinitely.

Another important keyword from the Principles behind the Agile Manifesto is, therefore, continuous delivery. In Wikipedia, continuous delivery (CD) is defined as a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software with greater speed and frequency.

Continuous delivery is one of the hardest parts of agile development. Do your developers have the competencies to release zero-bug software increments several tens of times a day, or at least once a month? Do you know what competencies are required for your developers to be able to do that? If your answers are ‘no’ to these questions, then stop and first check your readiness for adopting agile development! This page helps you know how to get ready for agile development, and succeed in your journey of improving software development productivity and quality by becoming better agile. Now let’s review the rest of the Principles behind the Agile Manifesto quoted in the following: 

Business people and developers must work together daily throughout the project. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. … Continuous attention to technical excellence and good design enhances agility. Simplicity--the art of maximizing the amount of work not done--is essential. The best architectures, requirements, and designs emerge from self-organizing teams. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. 

Why must business people and developers work together daily throughout the project? Continuous delivery is possible if developers can build software fast without errors. What makes continuous delivery impossible is the software bug. If bugs are not cleaned up continuously, they tend to accumulate and you can’t release the software knowing that there are an unknown number of bugs in the software. Among all kinds of errors, requirement errors are the most costly to fix. Leffingwell and Widrig estimated that requirement errors contributed the majority—often 70 percent or more—of the rework costs (D. Leffingwell and D. Widrig, Managing Software Requirements, Addison-Wesley, 1999). Since rework typically consumes 30–50% of a typical project cost (B. Boehm and P. Papaccio, Understanding and Controlling Software Costs, IEEE Transactions on Software Engineering, 14:10, Oct. 1988, pp. 1462 - 1477), it follows that requirement errors can easily consume 25%–40% of the total project cost! The best way to avoid requirement errors is to have business people and business analysts (or product managers) work together to specify requirements and validate them—first as designed in models and then as implemented in shippable product increments.

Often the requirement engineering role and the solution design and implementation role are separated in a project team. If business analysts or product managers don’t implement the solution for themselves, but have a separate team of developers do it (as is promoted in the Scrum practice), business people, i.e. customers and users, business analysts or product managers, and developers (in its narrow sense) should work together daily throughout the project.

But should they work together through face-to-face conversation? How many agile projects are run that way? Is it desirable to do so? Many development teams often use cloud services for distributed software development, distributed source code management, distributed project management, real-time communication, social collaboration, and video conferencing instead of meeting face-to-face. More and more components of a software solution are not home-grown, but are sourced from external services or third-party software systems, and assembled together via APIs. You can’t invite all those developers of external sources for face-to-face meeting. You need to read documents, exchange emails and interact with them using social media, web conferencing, open or closed communities, etc. It almost sounds nonsense today to say that software project team members should work in the same location and collaborate face-to-face.

What would have been an underlying principle that required face-to-face collaboration? I think that the underlying fundamental principle is the Lean principle. The Lean process tries to maximize the customer value with a minimum waste in production (Lean Enterprise Institute, What is Lean? 2018. https://www.lean.org/WhatsLean/). If there is a gap between what business people want and what developers think they want, it will cause waste of development efforts. The gap is often introduced by errors in written requirement specifications (structured or unstructured). It will help reduce that waste if business people and developers (in the broad sense) meet frequently (online or offline), work together on requirements specification, and validate them together using frequently released working systems.

One of the principles behind Agile Manifesto, working software is the primary measure of progress, means that the team should minimize producing non-value- adding, work-in-process (WIP) which may include market requirements documentation (MRD), product requirements documentation (PRD), analysis models, design models, and so forth. But the Principles behind the Agile Manifesto also include: Continuous attention to technical excellence and good design enhances agility. ... The best architectures, requirements, and designs emerge from self-organizing teams. It encourages producing work-in-process such as architecture, requirements and design.

Yes, it is all about the balance, not black or white. We don’t want to produce non-value-adding documentation, but want to produce just-enough documentation, just in time, pulled by the working system to be released next. The Lean principle is also expressed as simplicity—the art of maximizing the amount of work not done—in the Principles behind the Agile Manifesto.

Lastly, another important keyword in the Principles behind the Agile Manifesto is technical excellence and good design. A closely related keyword is self-organizing teams. Effective agile development is possible only if the team members collectively have excellent software engineering capabilities including domain knowledge, analysis and design modeling skills, and mastery of state-of-the-art implementation technologies relevant for the project. The more multitasking team members are capable of, the faster the agile development can be, because there are less hand-offs between the members which tend to introduce delays and rework. This is why Scrum insists on having only three roles in a project—product manager, developer and Scrum Master. Developers are responsible for understanding and elaborating requirement specifications provided by the product owner, designing and refactoring architecture and detailed design, coding, testing, build and deployment—pretty much all kinds of activities required in software development.

To summarize, we have extracted a few essential elements, hence recommendable patterns, of agile development in this section from reviewing the Manifesto for Agile Software Development. They are:

Value Proposition: The whole purpose of agile development is to create distinctive value for the user.

Validated Learning: The best way to overcome the uncertainty about what functionalities are valuable to the user is to build a working software and have the user experience the software.

Continuous, Lean Delivery: We want to shorten the feedback cycle (i.e., release cycle) because the shorter the cycle, the less the waste according to the Lean principle.

Software Engineering Competency: People with certified competencies in knowledge, skills and tools necessary for the project must form the development team.

Value proposition is the goal, while validated learning and continuous, lean delivery are the methods to achieve the goal, and software engineering competency is the essential resource for carrying out the methods successfully.

 

More to come ...