Scrum: A Review After Two Years

If you’d told me a few years ago that I’d be adopting scrum, I wouldn’t have believed you. Isn’t this what they do at boring corporates? Don’t scrum teams end up with tons of technical debt? How much does a certification you earn in a two-day workshop really mean? Shouldn’t we be focusing on people instead? I had so many concerns and preconceptions about what scrum was and what it encouraged.

Now, after two years of use, my opinions about the framework have shifted. While I think there are valid reasons for the kind of suspicions I had, I don’t think there is anything wrong with the framework itself. I’ll review some of my original concerns below, but first let’s look at the context in which we adopted Scrum.

A Quick History

As a small team at a small start-up, we initially focused on having good technical practices and following good management principles. We were a three-person team at the start, and I focused heavily on finding the right balance between building new features and managing tech debt. Right at the beginning, we only did stand-up meetings and 1-on-1s. Pretty soon, we also started doing retrospectives every two weeks.

Then, the company started growing quickly, so we decided that it could be a good idea to adopt a framework. This would come with the benefit that people start with some shared knowledge. Additionally, we’d been growing our processes incrementally. By adopting a framework we could potentially end up quicker where we were headed anyway.

A colleague suggested reading the “Scrum” book by Jeff Sutherland. I kept an open mind, read it, was pleasantly surprised, and was convinced enough to give it a shot. The week after we decided to try it out, but with the mindset that we wouldn’t do anything without understanding it. Without understanding the thinking behind a process, it loses much of its value and becomes a recipe for bureaucracy.

Overall, we were happy with the results and kept using it indefinitely. Now, after two years, I can review a few of my original concerns.

Concern 1: Doesn’t promote technical excellence

I don’t remember exactly where I first got this idea form, but the “Flaccid Scrum” blog post by Martin Fowler summarizes it quite well. It seems like many teams that adopt Scrum end up with a lot of technical debt. For some reason they don’t pay enough attention to the internal software quality.

As Fowler mentions, Scrum itself doesn’t promote any specific technical practices, so it’s up to the team to make sure those are there. In our case we were already pair programming, doing test-driven development, and deploying via a CI pipeline. Those practices, and continued discussion about these topics, helped us keep technical debt under control.

In my opinion, this is why it’s important that whoever picks up the scrum master role not only has good people skills but also has a strong enough technical background, like a tech lead or an engineering manager. They must have a good understanding of the impact on tech debt and must give the engineers ample opportunity to voice their concerns. The sprints and their sprint goals encourage people to be really focused on meeting those goals, but you must make sure that this doesn’t erode the internal software quality over time.

Concern 2: Certifications earned in two-day workshops

If you are thinking about adopting Scrum, and you then think that sending someone off to a two-day workshop is enough to make your organization “agile”, you’re going to have a bad time. These certifications create a wrong expectation about competence.

I think for these kinds of projects to be successful, you need people who have a good understanding of processes, communication, and technical practices. You can’t learn all of those in just one workshop. The workshop might be useful, but you’re going to need a lot more than a two-day workshop to make a significant improvement. In fact, you’ll probably be improving your processes for a long time, or even indefinitely.

Read more books. Set clear expectations about the fact that this will be “imperfect” for a long time, but you’ll continually improve. Don’t use “it’s not agile” or “because they said so in the workshop” as arguments. Instead, explain more clearly why you think something isn’t a good idea. Make sure you have at least one clear action item coming out of the retrospective, and follow-up on them in the next one.

I did go onto a Scrum workshop eventually, and exchanging ideas with the agile coach leading the workshop was useful, but it’s important to have the right expectations.

Concern 3: Isn’t the competence of the team members more important anyway?

I think Tom DeMarco and Tim Lister had it right in Peopleware when they said: “The final outcome of any effort is more a function of who does the work than of how the work is done.”

If you aren’t able to hire the right people, to figure out their strengths, to unlock their potential, to give them enough autonomy so they can develop themselves, then whatever framework you adopt is going to have a negligible effect. People aren’t like cogs in a machine that you can fit into a bunch of teams and procedures. This is why I think the 1-on-1 meeting is the most important one there is.

However, this is not an either-or matter. Process matters too. Hiring the best and wasting their talent on ineffective processes would be most unfortunate. Without good processes, we’re less likely to collaborate effectively, build the right thing, or reach our potential. A good process allows the team to become larger than the sum of its parts. Scrum can be a way to help you achieve those better processes.

Just remember, teams can succeed with good people and without Scrum. The reverse is less likely to be true. Mind you that Scrum and valuing people are not opposed. It’s just that, too often, teams adopt Scrum and mostly focus on the practices and procedures, rather than the principles behind them.

Conclusion

In the end, I don’t think the Scrum framework itself has any major issues like I originally suspected. If you are currently struggling and you adopt it, you won’t be any worse off by using it. It just won’t make you automatically better unless you have a deeper understanding of the principles behind it.

I also don’t think there’s one framework that’s the best either. During my last job I regularly had meetings with people who lead teams using Kanban, Shape Up, or something custom. The ideas and discussions we had were immensely valuable, and I’ve borrowed ideas from all of them. In the end, it’s all about having a happy high-functioning team that’s able to handle the inherent complexities of software and its unintended consequences. Scrum is just one of the tools you could use.

The Why and How of Giving Room for Failure

Giving up control and creating room for failure is one of the most valuable skills I had to learn as a manager. It helped our team flourish and created many educational opportunities for individual contributors. However, a fear of disastrous mistakes and a misunderstanding about how people learn prevented me from taking this approach earlier. I hope that my story can help others in reaching this understanding more quickly.

As an individual contributor, I had been reasonably successful at maximizing my own performance. In the process, I learned to anticipate and prevent potential issues. Now, I was eager to help others acquire similar skills. My initial approach was to give lots and lots of "proactive" feedback and to be constantly on the lookout for potential mistakes, for which I then provided advice about what I would do.

What I expected is that I could prevent any mistakes from happening, while people would somehow learn from my past experiences through osmosis. In reality, this approach was neither effective nor scalable. As any good teacher or coach will tell you, we learn best when we get to practice on our own, not by someone prescribing us solutions. The people that did improve significantly, during this time, did so despite my approach, not because of it.

Looking back at my own experiences, I realized that I learned most from situations where I felt a high level of control and felt responsible for the outcome. The most educational moments were the ones where I was also slightly out of my comfort zone. Going through such experiences helped me truly internalize new knowledge, rather than just knowing about it.

After some reflection and reading up on the topic, I realized that I had failed to promote the same experiences in others. I was preventing team members from making any of the mistakes that had taught me so much, inadvertently robbing them of valuable learning opportunities.

Take deployments, for example. In the beginning, I checked every pull request myself, before it got deployed, to make sure it couldn’t bring down production. You don’t learn much about the various risks associated with deploys if somebody else is always guarding the process for you like that. It’s only once it’s you that’s deploying those changes to production, and you feel responsible for it, that you actively learn about those matters.

After that realization, I started asking myself the following question: How can I create room for failure and learning, while maintaining alignment and preventing disastrous mistakes? The latter part of that question is why I was so overly careful in the first place.

Eventually, I found that the best way to solve that question was to define clear goals and expectations. This was important for two reasons. First, if there isn’t enough clarity about what’s expected, then there’s a high chance of misalignment. This would lead to suboptimal decisions at best or chaos at worst. Secondly, by setting goals, we give people some freedom on how to achieve them. That allows them to own up to it and find the best way for themselves to achieve those goals.

Changing my habits to create clarity, instead of providing solutions wasn't always easy. When done well, however, it allowed people to take responsibility and reach those goals without a lot of guidance. Gradually they took on more and more. Pretty soon everyone was routinely doing the things I had initially been so worried about delegating. Looking back at it now, I feel ridiculous that I once worried so much about delegating those matters in the first place.

Often, I still worried about mistakes but I started handling those worries differently. Rather than preventing any from happening, I focused on making sure we learned from the mistakes that did. Whenever we did make mistakes, we did post-mortems or retrospectives and made sure we had at least one action item to make things better the next time.

This consistently led to incremental improvements to our tech stack and our processes, which over time added up to significant benefits. In the example of the deployment process, we went from me being the only one merging all pull requests, to everyone being able to do that in a Continuous Deployment setup, which not only ran our test suite but also our automated blackbox QA tests and sanity checks, like assessing whether our assets were correctly uploaded to the CDN.

As for those "disastrous" mistakes I was originally worried about: Most of them could be prevented by the improvements I just talked about. For the ones that weren't, and where I thought we were going to make a mistake, I did step in, but that became increasingly rare.

Additionally, I worked on instilling this mindset in everyone in our team. That way anyone could coach interns, or each other, in a similar fashion. This helped us share knowledge more systematically, and made us more effective in onboarding new team members. An example of this that I liked particularly is how we changed our pair and mob programming sessions. Too often, these practices involve a senior member doing the majority of the thinking and driving. Instead, we made sure that everyone got a chance to code, and that there were always moments where we asked each other to think through why we were doing things in a particular way.

In the end, giving people room for failure felt a lot more like giving them room for success. By creating an environment that promoted autonomy and learning from mistakes, we became a more resilient team.

---

Thanks to Alexander Claes, Charlotte Dekempeneer, and Filip Schouwenaars for reviewing my drafts.

Encouraging a Culture of Written Communication

More and more people are being exposed to working remotely. One of the key factors for success in a remote workplace is a culture of written communication. It’s not always obvious how to create such a culture, and it takes at least some level of discipline from the people involved to make it a habit.

I’ve worked with mostly remote teams over the past three years. Here are a few of my observations on what helped cultivate such a culture.

Drafts and High Standards

In High Output Management, Andrew Grove said that writing things down is a medium for self-discipline. It forces us to be more precise than in verbal communication and thus leads to clearer thinking. Similarly, you may have heard of Amazon’s culture of six-page memos, which Jeff Bezos talked about in one of their shareholder letters.

Based on these stories, you might be determined to demand high writing standards everywhere, right off the bat. I believe this is the wrong approach though. Context matters. In a team of individual contributors, it’s crucial to share ideas early, to iterate and collaborate on them. Not everyone will have the same level of writing skills. Improving it can take a long time. Lower the barrier, encourage more writing, and people will get more practice.

Where you should enforce high-quality writing is broad or upwards communication, because there clarity is more important than speed and ideation. Here too, it’s important to keep in mind that good writing takes time. In the shareholder letter, Bezos mentioned that the great memos usually took a week to write. They incorporated input from co-workers, and were edited or even re-written multiple times.

Don’t require that amount of effort everywhere. Imperfections, even typos, should be allowed for communication within the team. This will ultimately lead to more people collaborating, more people writing, and better solutions because of the combination of both.

Easy to Search. Single Source of Truth.

The usefulness of lowering barriers extends to the area of knowledge management too. I believe it has to be easy for people to write a document without having to worry too much about where to put it. Therefore, I believe it's better to have a tool with good search functionality rather than investing in having a perfect information architecture. In the latter case, we're putting up extra roadblocks for the writer, because then they'll need to spend extra time figuring out where to put their writing in the first place.

Similarly, you should try to have a single source of truth as much as possible. If you have many places where information can be found, it will affect both people writing the documentation and those trying to find it. The writers will have yet another blocker to just getting stuff written down. The readers won’t know where to look and will have to interrupt people in Slack, as opposed to referring to the knowledge base themselves.

Balancing Async and Synchronous Communication

Sometimes written discussions can become quite long. While this may seem like a good sign initially, you’ll notice that discussing in this way can become tiresome because of the extended back-and-forth. So when a comment thread becomes a bit too long, it’s often better to jump on a video call to discuss things and then summarize it afterward. You don’t want to have these video calls every time, but finding the right balance between synchronous and asynchronous communication is good. This is usually more appropriate in the early stages of a draft, when there are still a lot of questions to ask and context to fill in.

The rule of thumb is to prefer asynchronous communication as much as possible, but don’t become too pedantic about it when you notice it starts to break down. It’s better to get on a few video calls and push for more async if you feel it’s appropriate than thinking black-and-white and sketching video calls as completely evil.

Thinking Out Loud

Finally, good teams communicate well, whether it’s a professional basketball team, a crew on a nuclear submarine, or a product team in a startup. As much as possible, encourage people to post anything of interest to the public channels. Notice that I’m saying public channels, and not private messages, because we don’t ping anyone unless it’s necessary. We just want to make the information available.

If you are changing a configuration option on production, considering to do a database migration, or just noticed something interesting in the performance graphs… post it to the chat. When you are making critical changes, this has the benefit of making the action deliberate. Just like the Japanese train conductors who deliberately point at whatever they are checking, you’re much less likely to make mistake that way.

If you’re uncertain about something, or investigated something of interest, but didn’t quite get to the bottom of it… share it in the chat. It gives others a chance to chime in and it builds trust. Posting in public channels can be daunting, if you have to mention you don’t know, or that something might be going wrong, or you think you may have caused an issue but aren’t sure. Encouraging everyone to do this regularly builds up both the team’s level of psychological safety and people’s courage to be vulnerable.

Once a crisis happens, and one always does at some point, you’ll be happy to have a team that’s ingrained with the habit of continuously and informally sharing information like this. It’ll be much easier to get a grasp of what’s going on and to get to the bottom of it quickly.

And, because of the trust you’ve built and the information that’s being pushed into public channels, there’s less of a need to speak via private messages or video calls. This means fewer interruptions and thus more pleasant and productive workdays in general.

What to Consider Before Embarking on a Rewrite

The itch to rewrite a piece of software from scratch can be one that’s hard to ignore. However, as many developers will learn throughout their career, rewrites can be tricky to pull off successfully because of the sheer amount of complexity and effort involved. If not executed correctly or done for the wrong reasons, then it will result in a loss or little to no gain if you're lucky. Thus, before embarking on that big rewrite it’s important to consider why it’s required and how we can increase the likelihood of success.

Let's take a look at some of the steps we can take throughout the decision making process. Typically, software engineers will be the first ones to raise alarm because they're close to the source. After that happens it’s important to investigate why. Intuition is a good indicator that something is up, but it doesn't necessarily point us to the root cause or the appropriate solution.

When we feel like a piece of software is becoming hard to deal with, it can be tempting to jump to conclusions and say that it has to be rewritten from scratch in a shiny new technology. This is like the example from "Thinking, Fast and Slow", where an investor picked a Ford stocks because he was impressed by their car show. The reason is that they had a tough question in front of them: "Should I invest in Ford stuck?", but subconsciously answered the easier question: "Do I like Ford cars?". We all have that tendency, so we have to make sure we're asking the right questions, and not the easy one about which technology we like.

Therefore, before we start prescribing a solution, we must make sure we have a good understanding of the problem. What is it that we’re trying to fix here?

The proper reasons for requiring a change are roughly going to fall into one of two categories. Either it’s something needed in service of our users and that helps us with company objectives, or it has a big enough impact on developer productivity and happiness to make it worth our while.

If it's something related to users or product, then it's likely a bit more obvious when there's an issue. When we look at metrics like, for example, the performance, weekly bug counts, incidents, and mean time to restore, and we can't find any other explanation than application related problems, then it's pretty obvious we should fix something in the product. This doesn't necessarily mean a rewrite is in order — we'll investigate that question later — but it does mean something has to improve.

Now, if we’re talking about issues related to developer productivity, then humans are involved so the topic becomes quite a bit more complex. We can still look at metrics here, but it’s important to realize that when they relate to human behavior, the picture the metrics alone can paint will not be black-and-white. There’s going to be some gray area, and we'll need to interpret everything in context.
For example, if the cycle time is trending upwards, we should rule out other causes, like how the team breaks up stories, before concluding that it's caused by to technical debt. If the team is healthy and has been stable for a while, then determining whether it's a tech or process issue should be pretty easy.

Once we've identified the category that the problem falls in, have gotten a more objective view of the issue, and determined that it is indeed a technological problem, then we can look at the harder part of the decision, which is figuring out whether a rewrite is appropriate and whether the cost is acceptable.

You may have heard that Google rewrites most of its software every few years, but you're unlikely to be a Google-scale company with more than 20.000 engineers and billions of dollars in profits.

The reality is that rewrites are both costly and risky. The original code has been battle-tested for what may have been years. When we decide to rewrite, all that complexity will resurface and will have to be re-implemented. Knowing that, we can assume there will be surprises during development, which will cause us to take longer to get to parity, which means it will take longer before we can do any new product improvements. On top of that, things could happen during that period that cause the company to shift focus and if we're caught up in the middle of a rewrite, then we may need to shelve it. That could mean losing months of work.

For those reasons, it's preferable to go for an incremental approach where possible. With this approach we try identify the biggest pain points and incrementally start chipping away at them. That way, we'll start providing value much faster than a rewrite ever would. Additionally, it's less risky, because we can change direction in between the increments and won't lose a huge amount of work if we have to drop something.

In many cases it's also difficult to answer the question: Is the cost worth the gain? There's no easy answer for this. The best we can do is make a reasonable estimate based on a cost-benefit analysis. Doing this in some quantifiable way is good, as long as we are aware that it can never be completely accurate and only serves as a rough estimate. For any significantly complex piece of software, there will always be unintended consequences and we should be prepared to deal with those.

We've already seen that we tend to underestimate the cost, and while doing our analysis we should keep in mind that we often overestimate the benefits of switching technologies too. "Hype is the plague on the house of software" as Robert Glass put it in Facts and Fallacies of Software Engineering. Changes in tools and technologies typically only result in a 5 to 30 percent increase in productivity and quality, and that's after the initial learning curve during which productivity is lower. So while there are definitely cases where technology changes are warranted, we should be aware of biases to estimate the costs and benefits incorrectly. Rewriting a React app to Vue probably won't benefit you much, but if you are still hiring COBOL developers, it might be worth thinking about incrementally replacing that system with something more modern to improve scalability and maintenance costs.

Now, once we've truly exhausted all other options, and are certain that the benefits will outweigh the costs, we can conclude that a rewrite is in order and be reasonably confident that it's the right solution.

Time to Reflect

Eight years ago, I started my first job at a web agency. The fact that I got to do programming for a living was very exciting to me. Only a few years earlier, I didn’t really know what to do and wasn’t sure if I could find a job that I would like. Now, I got to work on a lot of projects with a bunch of creative people and was super excited going to work every day. However, after a bit over a year there, I found that I preferred doing more long-term product work rather than continuously jumping from one project to the next.

That’s when I discovered a startup was building access control solutions for festivals, so it seemed like a cool job for someone who only recently graduated. I joined them and spent most of my time on building out a cashless payment system which would allow festival goers to pay for drinks via the NFC chip in their wristbands. We went through a lot of stressful periods there, especially the first season we deployed that software, but I liked the adrenaline rush. While we were at it, I learned a lot about building and maintaining software over a longer period of time.

After three years of that, I got a message from one of DataCamp’s recruiters. By that time I wasn’t necessarily planning on leaving my current job, but I did build up some frustrations about their processes, which in my opinion weren't good enough and didn't allow us to tackle recurring problems in a better way. At the same time, e-learning was something that excited me, because it had helped me in the past and I felt like it leveled the playing field, so I decided to make the switch and joined them by the end of 2016.

At DataCamp I initially spent a lot of time applying what I had learned previously and getting some technical debt related to the CI, payment and learning management systems under control. We started growing fast, and as we were hiring more and more engineers, the opportunity presented itself for me to become a team lead. I wasn’t really planning on that, but because of my frustrations at my previous job, I had spent a fair amount of time reading management books and was curious about what would happen if I could apply some of that knowledge, so I agreed. The job was quite a bit different than I expected in a good way, and I enjoyed it a lot. I loved helping people in 1on1s, collaborating with the team to improve processes, figuring out how to set clear expectations, recruiting new software engineers, and everything that came with building a highly-functioning team. I worked with some of the most talented people I’ve met here, and tackling challenge after challenge with such a team was very fulfilling.

Nevertheless, over the last half year or so, I’ve felt my excitement fade and start to be replaced by various frustrations for which I couldn’t pinpoint the root cause. I tried working through this with the help of various people, but I feel like I wasn’t making progress fast enough and decided the only way for me to get unstuck is to take a break.

Throughout the past 7-8 years I was learning continuously and never gave much thought about what’s next. There always seemed to be some clear, shiny problem ahead of me and I tried to tackle them all one after the other. This was great while it lasted and I’m grateful for it, but never having taken the time to reflect is what got me into this situation, so now I’m giving myself some time to catch up. Last week was my last at DataCamp. Today, I’m starting my sabbatical which will last for at least the summer, potentially longer. This will help me clear up my thinking, re-explore what interests me, and allow me to make better decisions about what’s next.