The Manifesto for Agile Software Development (K. Beck, et al., Manifesto for Agile Software Development, 2001. http://agilemanifesto.org/) valued
individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan.
No one can deny that Scrum has been the most widely adopted practice of agile development. The State of Agile survey by CollabNet/VersionOne in 2017 (https://explore.versionone.com/state-of-agile/versionone-12th-annual-state-of-agile-report) reported that 56% of respondents used Scrum as their agile practice (with the second most used being a hybrid of multiple practices at 14%). It also reported that 88%, 67%, 46% and 35% of respondents did sprint planning, release planning, product roadmapping, and agile portfolio planning, respectively; and that the percentage of respondents who used such tools as agile project management tool, bug tracker, automated build tool, unit test tool, continuous integration tool, requirement management tool, release/deployment automation tool ranged between 74% and 44%.
We can see that many people who actually have done agile development followed a very rigorous process such as Scrum. They employed many tools for automating some human activities in software development. They made many plans from long-term plans such as project portfolio plans and product roadmaps, mid-term plans such as release plans to short-term plans such as sprint plans. Process, tools and planning are so common ways how people work when they try to collaborate to achieve certain goals in an efficient manner. Therefore, people developing software in teams would also seek the benefit of well-honed processes, adaptable plans that all team members share, and advanced software tools that can improve productivity and quality.
Documentation isn’t always bad either. It’s basically converting implicit knowledge to explicit knowledge so that the knowledge can be stored, searched, classified, shared, reused, continuously improved, taught, and so on. Application programming interface (API) is a sort of documentation, for which most use the language of SOAP or REST web service today. SOAP and REST API documents made service-oriented architecture (SOA) the prevailing architecture style since the mid-2000s, which in turn led us into cloud computing, microservices, DevOps and further into the current era of digital businesses.
Agile manifesto really does not address the essence of agile development that all agile projects should be characterized by. It suggests a direction of change of common practices of software development that were becoming outdated in 2001. Jim Highsmith, for example, said in 2001 on behalf of Agile Alliance (http://agilemanifesto.org/history.html
The Agile movement is not anti-methodology, in fact, many of us want to restore credibility to the word methodology. We want to restore a balance. We embrace modeling, but not in order to file some diagram in a dusty corporate repository. We embrace documentation, but not hundreds of pages of never-maintained and rarely-used tomes. We plan, but recognize the limits of planning in a turbulent environment.
Agile development has become a buzzword for modern, effective way of software development. When I talk about the essence of agile development, it really means the essential things that software developers should know and do today. Now, if we look at the Principles behind the Agile Manifesto (http://agilemanifesto.org/), there are more clues about the essence of agile development.
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. … Working software is the primary measure of progress. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
A keyword I see here is value. We should develop software that provides value to the user. In other words, software should have a value proposition; viz., relieve pains associated with incumbent solutions, and/or create unseen gains to the user (http://www.businessmodelgeneration.com/canvas/vpc
). The software that provides discriminant value proposition to a customer will help the customer gain competitive advantage over its competitors who don’t use the software or use other vendors’ software.
How do software developers know the software they plan to build has distinctive value to the target customer? This uncertainty requires the developers to pay for the information they want, i.e. the true information about the set of features that have distinctive values to the customer. This information is best obtained when the customer is given a working software that the developers built based on their hypothesis about the valuable features for the customer. The customer’s feedback from actually using the working software is the best source of information to learn valuable features. This is what Eric Ries called validated learning in his book on Lean Startup in 2011. The Lean Startup method of creating new software goes through a Build-Measure-Learn cycle—a fast feedback loop where a product increment is delivered to the customer frequently to get feedback to learn desirable features.
Design Thinking is widely adopted by software development organizations these days because it helps them run the Build-Measure-Learn cycle effectively. Design Thinking suggests a creative design process that goes through Empathize, Define, Ideate, Prototype and Test (Tim Brown, Change by Design, 2009). Empathize with customers to learn what they really want consciously and unconsciously; make a hypothetical definition of the customer problem to solve; ideate the software solution, perhaps producing user stories, innovative business processes, intuitive user experiences, etc.; then build and release the minimum viable product (MVP) (Eric Ries, The Lean Startup, 2011) many times a day, in order to test the hypotheses and learn from the customer feedback.
Allowing the user to actually experience the use of a solution is the best way to learn real (sometimes hidden) needs of the user. Therefore, prototyping is used in many fields of engineering. In software engineering, prototyping is relatively more productive than in hardware engineering because software can be changed more easily and therefore prototypes can be salvaged into the shippable product. In software engineering, prototyping has been encouraged since as early as mid-1970s. (F. Brooks, The Mythical Man-Month, 1975) It’s thus not a surprise at all that Agile Manifesto promoted continuous delivery, which is to produce prototypes incrementally based on user feedback in a very fast cycle, maintaining a constant pace indefinitely.
Another important keyword from the Principles behind the Agile Manifesto is, therefore, continuous delivery. In Wikipedia, continuous delivery (CD) is defined as a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software with greater speed and frequency.
Continuous delivery is one of the hardest parts of agile development. Do your developers have the competencies to release zero-bug software increments several tens of times a day, or at least once a month? Do you know what competencies are required for your developers to be able to do that? If your answers are ‘no’ to these questions, then stop and first check your readiness for adopting agile development! This page helps you know how to get ready for agile development, and succeed in your journey of improving software development productivity and quality by becoming better agile. Now let’s review the rest of the Principles behind the Agile Manifesto quoted in the following:
Business people and developers must work together daily throughout the project. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. … Continuous attention to technical excellence and good design enhances agility. Simplicity--the art of maximizing the amount of work not done--is essential. The best architectures, requirements, and designs emerge from self-organizing teams. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Why must business people and developers work together daily throughout the project? Continuous delivery is possible if developers can build software fast without errors. What makes continuous delivery impossible is the software bug. If bugs are not cleaned up continuously, they tend to accumulate and you can’t release the software knowing that there are an unknown number of bugs in the software. Among all kinds of errors, requirement errors are the most costly to fix. Leffingwell and Widrig estimated that requirement errors contributed the majority—often 70 percent or more—of the rework costs (D. Leffingwell and D. Widrig, Managing Software Requirements, Addison-Wesley, 1999). Since rework typically consumes 30–50% of a typical project cost (B. Boehm and P. Papaccio, Understanding and Controlling Software Costs, IEEE Transactions on Software Engineering, 14:10, Oct. 1988, pp. 1462 - 1477), it follows that requirement errors can easily consume 25%–40% of the total project cost! The best way to avoid requirement errors is to have business people and business analysts (or product managers) work together to specify requirements and validate them—first as designed in models and then as implemented in shippable product increments.
Often the requirement engineering role and the solution design and implementation role are separated in a project team. If business analysts or product managers don’t implement the solution for themselves, but have a separate team of developers do it (as is promoted in the Scrum practice), business people, i.e. customers and users, business analysts or product managers, and developers (in its narrow sense) should work together daily throughout the project.
But should they work together through face-to-face conversation? How many agile projects are run that way? Is it desirable to do so? Many development teams often use cloud services for distributed software development, distributed source code management, distributed project management, real-time communication, social collaboration, and video conferencing instead of meeting face-to-face. More and more components of a software solution are not home-grown, but are sourced from external services or third-party software systems, and assembled together via APIs. You can’t invite all those developers of external sources for face-to-face meeting. You need to read documents, exchange emails and interact with them using social media, web conferencing, open or closed communities, etc. It almost sounds nonsense today to say that software project team members should work in the same location and collaborate face-to-face.
What would have been an underlying principle that required face-to-face collaboration? I think that the underlying fundamental principle is the Lean principle. The Lean process tries to maximize the customer value with a minimum waste in production (Lean Enterprise Institute, What is Lean? 2018. https://www.lean.org/WhatsLean/). If there is a gap between what business people want and what developers think they want, it will cause waste of development efforts. The gap is often introduced by errors in written requirement specifications (structured or unstructured). It will help reduce that waste if business people and developers (in the broad sense) meet frequently (online or offline), work together on requirements specification, and validate them together using frequently released working systems.
One of the principles behind Agile Manifesto, working software is the primary measure of progress, means that the team should minimize producing non-value- adding, work-in-process (WIP) which may include market requirements documentation (MRD), product requirements documentation (PRD), analysis models, design models, and so forth. But the Principles behind the Agile Manifesto also include: Continuous attention to technical excellence and good design enhances agility. ... The best architectures, requirements, and designs emerge from self-organizing teams. It encourages producing work-in-process such as architecture, requirements and design.
Yes, it is all about the balance, not black or white. We don’t want to produce non-value-adding documentation, but want to produce just-enough documentation, just in time, pulled by the working system to be released next. The Lean principle is also expressed as simplicity—the art of maximizing the amount of work not done—in the Principles behind the Agile Manifesto.
Lastly, another important keyword in the Principles behind the Agile Manifesto is technical excellence and good design. A closely related keyword is self-organizing teams. Effective agile development is possible only if the team members collectively have excellent software engineering capabilities including domain knowledge, analysis and design modeling skills, and mastery of state-of-the-art implementation technologies relevant for the project. The more multitasking team members are capable of, the faster the agile development can be, because there are less hand-offs between the members which tend to introduce delays and rework. This is why Scrum insists on having only three roles in a project—product manager, developer and Scrum Master. Developers are responsible for understanding and elaborating requirement specifications provided by the product owner, designing and refactoring architecture and detailed design, coding, testing, build and deployment—pretty much all kinds of activities required in software development.
To summarize, we have extracted a few essential elements, hence recommendable patterns, of agile development in this section from reviewing the Manifesto for Agile Software Development. They are:
Value Proposition: The whole purpose of agile development is to create distinctive value for the user.
Validated Learning: The best way to overcome the uncertainty about what functionalities are valuable to the user is to build a working software and have the user experience the software.
Continuous, Lean Delivery: We want to shorten the feedback cycle (i.e., release cycle) because the shorter the cycle, the less the waste according to the Lean principle.
Software Engineering Competency: People with certified competencies in knowledge, skills and tools necessary for the project must form the development team.
Value proposition is the goal, while validated learning and continuous, lean delivery are the methods to achieve the goal, and software engineering competency is the essential resource for carrying out the methods successfully.
More to come ...