Data-Informed Retrospectives

TL; DR: Data-Informed Retrospectives

In their book Agile Retrospectives, Esther Derby and Diana Larsen popularized the idea that a Sprint Retrospect comprises five stages. The second stage refers to gathering data so that the Scrum Team can have data-informed Retrospectives.

As I have observed in practice, many Scrum Teams either limit the data gathering part of the Retrospective, thus lacking vital information. Or they invest too much time doing so, leaving little capacity to analyze the data and come to conclusions on how to best improve as a team.

Read on and learn how you can avoid falling victim to both scenarios by gathering data continuously and asynchronously.

🗞 Shall I notify you about articles like this one? Awesome! You can sign up here for the ‘Food for Agile Thought’ newsletter and join 31,000-plus other subscribers.

🎓 Join Stefan in one of his upcoming Professional Scrum training classes!

Join more than 150 peers from May 27-29, 2021, for the Virtual Agile Camp Berlin 2021, a live virtual Barcamp using open space technology principles and practices.

Purpose of Sprint Retrospectives

Let us start our excursion by revisiting the purpose of the Sprint Retrospective according to the Scrum Guide:

The purpose of the Sprint Retrospective is to plan ways to increase quality and effectiveness in the process.

The Scrum Team inspects how the last Sprint went with regards to individuals, interactions, processes, tools, and their Definition of Done. Inspected elements often vary with the domain of work. Assumptions that led them astray are identified and their origins explored. The Scrum Team discusses what went well during the Sprint, what problems it encountered, and how those problems were (or were not) solved.

The Scrum Team identifies the most helpful changes to improve its effectiveness. The most impactful improvements are addressed as soon as possible. They may even be added to the Sprint Backlog for the next Sprint.

Source: Scrum Guide 2020.

For our purpose, we need to understand that there are two primary sources of data that the Scrum Team can analyze:

1. There are hard, quantitative facts measurable during the Sprint, for example, lead time and cycle time, the number of Product Backlog items turned in Increments, the ratio of fixing work to feature work, or the number of defects escaping to the production system. An additional metric might be the amount of anticipated work at the beginning of the Sprint vs. done work delivered at the end of the Sprint. (Another good source for actionable agile metrics is the State of DevOps Report.)
2. And there are notions and opinions from team members how they perceived the previous Sprint personally. These opinions have a qualitative nature. In my experience, they are best accessible by anonymous surveys run in advance of the Sprint Retrospective.

Cannot see the form?
Please click here

The Usefulness of Team Metrics

Generally speaking, metrics help to understand the current situation better and allow us to gain insight on change over time. Without metrics, assessing any effort or development will be open to gut feeling and bias-based interpretation.

A metric should, therefore, be a leading indicator for a pattern change, providing an opportunity to analyze the cause in time. The following three general rules for agile metrics have proven to be useful:

1. The first rule of meaningful metrics is only to track those that apply to the team. Ignore those that measure the individual.
2. The second rule of meaningful metrics is not to measure parameters just because they are easy to track. This practice often is a consequence of using various agile project management tools that offer out-of-the-box reports.
3. The third rule of meaningful metrics is to record context as well. Data without context, for example, the number of the available team member, or the intensity of incidents during a Sprint, maybe turn out to be nothing more than noise.

For example, if the (average) sentiment on the technical debt metric (see below) is slowly but steadily decreasing, it may indicate that:

• A team may have started sacrificing code quality to meet (arbitrary) deadlines, or
• A team may have deliberately built some temporary solutions to speed up experimentation.

While the latter probably is a good thing, the first interpretation is worrying.

Helpful Practices for Data-Informed Retrospectives

While you probably have quantitative data readily available somewhere in your project management system, or at least you could configure it so that the application delivers the desired information, you will need to gather the qualitative data differently. Let’s have a look at a few proven techniques:

Keeping a Team Diary

What occurrences influenced the Scrum Team’s progress towards achieving the Sprint Goal? Keeping a team diary effectively collects information on events or meetings, communication threads, incidents, problems, or wins and fails. Based on this information, creating a timeline of a Sprint becomes simple.

Timelines are very helpful when you want to visualize the flow of events and how those have influenced each other. For example, using a timeline allows you to connect management pressure on the Scrum team to meet a delivery date with a subsequent decrease in the perceived quality of the tech stack as the Developers had to cut corners to meet the deadline, increasing technical debt.

Keeping a team diary also allows voting on the importance of occurrences, as individual team members may have a different take on what happened. Also, timelines provide the groundwork for overall retrospectives with stakeholders and customers, which a Scrum Team should consider running in regular intervals.

Collecting Qualitative Data with Anonymous Surveys

My tool of choice to collect qualitative data is anonymous surveys. There are plenty of tools available for this purpose, from Typeform to Google Forms to SurveyMonkey, to name a few. Please ensure before running them, though, that anonymous surveys are in line with the governance rules of your organization. (I once worked for a large automotive supplier in Germany where every survey among workers needed to be approved by the work council—four weeks in advance—which rendered the tool practically obsolete for Sprint-based surveys.)

My preferred set of questions for every Sprint is:

1. Do you think that we accomplished the Sprint Goal this Sprint?
2. On a scale from 0 (= worthless) to X (= very valuable), how would you rate the current Product Increments’ value to our customers?
3. On a scale from 0 (= no debt) to X (= we should rewrite everything), how would you rate the level of technical debt of our platform?
4. How likely would you recommend an open position in our organization to a good friend with an agile mindset? (0 = never; X = immediately.)
5. How would you rate your situation on a scale of 0 to X? (0 = I am already looking for a new job; X = I am having the best time of my professional life.)

Of course, I visualize the results over several Sprints to identify patterns as early as possible. Again, the survey results complement the data from the team diary.

The anonymous survey is also well-suited to gather data on the Scrum Team’s fluency in Scrum Values. In this application, you use the five Scrum Values of commitment, courage, focus, openness, and respect as questions:

On a scale from 0 (= not at all) to X (= always), how would you rate the team’s appreciation of the Scrum Value “Commitment?”

Action Item Accounting

Often, Scrum Teams excel at identifying problems only to drop the baton once it comes to apply a remedy to the challenges at hand. Here, two simple practices help the team to move forward:

1. Identify a directly responsible individual for an individual improvement task.
2. Keep track of all improvement tasks—a simple Kanban board will do—and start every Retrospective with a short follow-up on the state of the team based on this information.

The data from this exercise, for example, the lead and cycle time of action items, or the ratio of open to delivered improvement items, also proves to be helpful for a data-informed Retrospective.

A Questionable Technique

Should you provide a Sprint mailbox for grievances? The idea is that the mailbox is permanently accessible by comparison to the Sprint survey, which is typically answered at the end of the Sprint. Please note that having to provide a Sprint mailbox points to severe communication and collaboration issues at the team level. Team members do not seem to live Scrum Values like openness, respect, and courage, and there appears to be a lack of psychological safety. Hence I would consider providing a Sprint mailbox only as a temporary means until the communication issues within the team have been solved.

Conclusion

Without data, there is no information; without information, there is no insight; without insight, there is no way to understand where you are heading as a Scrum Team. If you take ‘continuous improvement’ as a concept seriously, you will enjoy identifying the right set of metrics for your Scrum Team, embracing the idea of data-informed Retrospectives.

What data are you tracking to understand the progress of your Scrum Teams better? Please share with us your lessons learned and tips & tricks.

✋ Do Not Miss Out and Learn about Data-Informed Retrospectives: Join the 9,000-plus Strong ‘Hands-on Agile’ Slack Team

I invite you to join the “Hands-on Agile” Slack team and enjoy the benefits of a fast-growing, vibrant community of agile practitioners from around the world.

If you like to join now all you have to do now is provide your credentials via this Google form, and I will sign you up. By the way, it’s free.

📖 Data-Informed Retrospectives — Related Posts

Agile Metrics — The Good, the Bad, and the Ugly

21 Sprint Retrospective Anti-Patterns Impeding Scrum Teams

The Overall Retrospective for Team and Stakeholders

Faking Agile Metrics or Cooking the Agile Books

Download the Scrum Anti-Patterns Guide for free.

📅 Scrum Training Classes, Workshops, and Events

You can secure your seat for Scrum training classes, workshops, and meetups directly by following the corresponding link in the table below:

See all upcoming classes here.

You can book your seat for the training directly by following the corresponding links to the ticket shop. If the procurement process of your organization requires a different purchasing process, please contact Berlin Product People GmbH directly.