Agile Principles

Introduction to Scrum and Management (Part 6 of 6)

This is the part I wrote first. All the other parts were written to justify this coldhearted analysis on what should be the role of management in Scrum. I was convinced that there had to be something more for management to do than “support the team and get out of the way.”

Over the years, managers of all stripes, engineering managers, product managers, project managers, manager managers have complained to me, usually as a stage-whispered aside, that “agile is dead” or “scrum is not agile.” Their frustration seemed to come from several places: the lack of promised accelerated productivity, the lack of visibility (other than the sphinxlike story point’s slow burndown), and complicated answers to simple Waterfall milestone status questions.

We managers, of all flavors, have layered on a whole superstructure of improvements on top of Scrum in our quest for certainty in an uncertain world. But let’s look ourselves in the selfie: Have these improvements worked? Have we improved Scrum? Have we delivered more certainty than what Scrum originally promised? No.

Working through the Computer Science foundations of Scrum, the data structures and algorithms, I realized that all these improvements to Scrum brought about by managers like me haven’t improved Scrum but obscured a scientific model of work under a fog of superstition, old husband tales, and best practices.

So, now, after all this, what really is the role of Management in Scrum?

Scrum is system and humans are its parts

Scrum System Design

First, a quick summary of parts 1, 2, 3, 4, and 5

  • I read a book on Scrum by the inventor and co-creator of Scrum and his son
  • I read this book because while I’ve been supporting Scrum for more than a decade, I kept hearing about how Agile is dead and Scrum is not Agile.
  • I realized two insights from a close reading of the book: managers have no formal role in Scrum (autonomous teams don’t need managers) and there is a hardcore computational basis for the many of the processes that people follow in Scrum.
  • I further realized that if you don’t treat these data structures and algorithms for what they are, you don’t get the productivity and team happiness benefits of Scrum.

I bet, as an experienced scrum master, you already knew all this. But most of the management folks I run with don’t think of Scrum as a computational system. We managers tend to see Scrum as a set of new best practices for project management. This is a little like seeing Astronomy as a new a better way to cast horoscopes for Astrology.

Scrum, at its heart, is a computational system that creates a human-based machine. Scrum uses this human-based machineto accelerate productivity by removing waste from the work process. The secret of Scrum is in the constraints it puts around inefficiencies but not around creativity. The beauty of Scrum is in its economy of design. This design enables Scrum to apply to a wide range of work problems (not just software development). A side effect of Scrum is that the human-machine manages itself and its moving parts (team members) are happier than they are in with a traditional manager managed process.

If Jeff Sutherland, like Jeff Bezos, had built a private platform out of Scrum instead of a public framework, he would be rocketing people to Mars and tooling around on his billion-dollar yacht.

Treat people like machines

OK, fellow managers, here is my advice (caveat emptor)

First, leave Scrum alone. Don’t fix it. Don’t do pre-work outside of the Sprint. Don’t tell the Sprint team or the Scrum master what to do or how to do it. Let the Scrum process fix itself over time.

Second, fix the problems outside of Scrum with formal computation systems (human machines) for those folks left out of the Scrum process. Translate your work into data structures and algorithms and eliminate waste. Don’t worry about whether the computation will be performed by silicon or carbon.

Scrum does an excellent job of work-as-computation at high efficiency. It does this by creating formal roles for the people who Sprint and ensuring that all work is filtered for priority and done with in a predictable, repeatable, time-boxed process.

BTW, this process of treating people like machines is nothing new!

The first computers were not made of silicon and software. They were people. For thousands of years people were doing the computing that enabled empires to trade, businesses to serve customers, and NASA to send rockets to the moon. Only within my lifetime have we delegated computation to non-humans.

I sense your eyebrows rising sharply! Managers who treat people like machines are inhumane.

And you are right. If we don’t follow Scrum’s model of how to compute well with people, then we managers are the living incarnation of Dilbert’s pointy-haired boss. We are micromanagers who make buzzwords out of useful tools like Agile, Scrum and DevOps. But if we don’t treat our people like machines what are we treating them like? Resources? Head counts? Soft capital?

So, if you think about it, as a manager, you pretty much treat your people like machines at some level. You give them tasks, expect them to ask relevant questions, and then to do the task to your specifications by the due date. You expect high-functioning employees to work well with vague input and all the rest to require SMART input. You don’t expect the employee’s feelings to impact the work. You are not a monster, but you have a business to run.

It is interesting to note that the people-treated-like-machines who follow a Scrum practice are far happier than their beleaguered and belabored non-Scrum counter parts Why is that?

Formal (systems) beats casual (anything)

I know we live in an age of the casual work environment. Dress codes are relaxed, hours are flexible, and hierarchies, while still in use, have been hidden away like ugly relics of a less enlightened age. But only the outside of the workplace is casual. On the inside our workplaces are just as formal as they have always been. I believe the patina of unscripted, casual interaction has made the workplace hard to navigate and an unhappier place.

Let’s contrast the formalism of Scrum with the casualism of the rest of the office:

WorkloadPrioritized backlog (sorted queue) locked during the Sprint.
Lee just sent a high priority email. Scrum master will take care of it for me!
Multiple uncoordinated sources that can change at any time. 
Lee just sent a high priority email. Should I drop everything to work on it?
WorkdayDefined by the sprint as a loop of predictable duration, where the team commits to a specific number of story points and a daily check-in meeting.
I can completely focus on my stories and if I get blocked the scrum master will unblock meI only have one meeting a day, so I don’t have to rudely work on my laptop during that meeting.
Multiple uncoordinated open ended workstreams with soft deadlines that demand multitasking.
I can’t focus completely on Lee’s request so it’s going to take days instead of an hour or two. I have so many meetings that I have to work on my laptop during each! I should also work during lunch and stay late but I’m feeling low energy and the kids need help with their home work.
Work unitStory point: a well described task with a set business priority and expected labor value such that worker knows if they are spending too much or too little time.
I tested, documented, and committed my code. My teams are doing a code review and will get back to me with feedback shortly. I know for myself that my work is on track, so I’ll start on my next story.
An email, a document, a presentation, a spread sheet, a list with no definition of done or labor value.
I sent Lee a deck, but I had to bump my other work to complete it. Is it finished? Should we meet to review it? Will my boss get a call from an angry department head because of all the bumping?
Work teamProduct owner, scrum master, and a specific set of developers. Nobody else is on the team.
I know exactly who is working with me on this project. Lee is the EVP of XYZ but I don’t have to worry about that. The Scrum master will take care of it.
Probably the people on the email you just got. 
Is Lee working on this project of is Lee a stakeholder?  Even Lee isn’t sure so to be safe just CC Lee on everything! The RACI is always out of date!

We can easily see why the members of a Scrum are happier than the members of a Non-Scrum. Formalism brings clear boundaries so that employees know what they are doing, how well they are doing, and when they are finished. Non-Scrum team members might work all night on a project and find they failed because they didn’t work with the right info, or the right people, or the right priority. This kind of work-tragedy brings tears of frustration to the most experienced and valuable employees and leads to cynicism and other productivity busters that we managers are supposed to be managing out of the organization!

Because Scrum embraces and thrives on change the RACI is never out of date! Inside the sprint the priorities, the work to do, the due dates, the team members, and the estimated labor values do not change! Outside the sprint management brings everything the team has to do up to date. As a manager who prides himself on closing and finishing, I love the elegant efficiency of Scrum. I don’t know how other managers in other departments cope without Scrum.

We managers need not to fix Scrum but to fix ourselves. The dev team has become super effective. We, engineering management, product management, project management, and all the other managements need to catch up. We need formal systems of our own, similar to Scrum in the sense that they use data structures and algorithms to eliminate waste and accelerate work. 

Agile Principles

Introduction to Scrum and Management (Part 5 of 6) presents, the penultimate episode of ITSAM! Starring the algorithms of Scrum. The computational thinking that makes it possible to do “twice the work in half the time.”

Last episode, part 4, starred the story point as a data structure of enumerated values and its function as a signal of complexity. Story points are expressed as Fibonacci numbers, ratios of intuitively accelerating magnitude. The humble but nuanced story point is like the pitch of the teeth in the gear that runs your sprint iteration: The finer the pitch (smaller the story point values) the faster your productivity flywheel turns.

In this episode we turn away from story points and take a step back to discuss four unambiguously defined recipes that precisely describe a sequence of operations that drive the Scrum process. Scrum is often visualized as a set of nested loops and we’re going to do the same. These loops take an input state, the backlog, and transform it by iterations, into an output state, working software.

Ah, but there is a catch! People are not machines. We tend to mess with the sequence and order of Scrum operations and derail the efficiency of its algorithms and then wonder why “Agile is dead.”

The algorithms of Scrum

What an algorithm is and is not is critical to understanding how to Scrum. Get it right and the Scrum fly wheel spins faster and faster. Get it wrong and the Scrum fly wheel wobbles and shakes, eventually flying off of its axis.

At the surface, almost any well-defined and repeatable process is an algorithm. Counting on your fingers, singing Baby Shark, and the spelling rule i before e except after c are more or less algorithms. To be a true computational algorithm all variation has to be nailed down. If human judgement is required in implementing an algorithm, as in knowing the random exceptions to the i before c rule, the algorithm isn’t reliable or provable. 

Jeff and JJ Sutherland, in their book Scrum: The Art of Twice the Work in Half the Time, don’t mention algorithms. Probably because what I’m calling algorithms don’t strictly fit the Wikipedia definition. But I believe if we refine these processes as close to true computation as we can get, Scrum works well. I believe it because I’ve seen it! So, let’s take a quick survey of each core algorithm in turn–we’re looping already.

The sprint (outer loop)

The outer loop of Scrum is the sprint. It’s a relatively simple Algorithm.

// pseudo-code implementation of a sprint loop
while value of epic count doesn't yet equal 0 {
  play planning poker() with highest-priority epic
  for each work day in sprint duration {
  	standup() with sprint backlog for 15 mins
  if demo is not accepted {
    throw sprint broken error()

I like the idea of the sprint as algorithm because there isn’t a lot of room for human creativity. But there are a few hidden constraints!

  • Scrum doesn’t want you to rest or waste time between sprints. Start the next sprint on the next working day.
  • Scrum wants the whole team participating in the sprint.
  • Scrum doesn’t want you to start new a sprint before the last one has completed.
  • Most importantly: Scrum wants all development activities to take place inside the sprint. This constraint creates a huge headache for product management, UX design, and QA as they are commonly practiced.

One reason Agile is dead and Scrum’s hair is on fire is that anything that happens outside the sprint is not Scrum, does not go fast, and creates terrible stories. 

For example, designing all your screens upfront with focus groups is not Scrum. Manually testing all your code after the demo is not Scrum. Skipping the demo, adding more engineers during the sprint, or asking engineers to work harder is not Scrum. The sprint loop with its constraints works really well if you don’t do any work outside the sprint!

Planning poker (pre-condition)

The first thing a Scrum team does on the first working day of a sprint is to plan. The core of that meeting is the planning poker algorithm. It takes patience and practice to get right.

// pseudo-code implementation of planning poker
while consensus is not true {
  product ower explains story
  team asks clarifying questions
  for each developer in sprint team {
    compare story to previously developed story
    estimate work using story point value
    present estimate to team
  if story points match {
    set consensus to true // breaks the loop

The goal is to transform an epic into a prioritized backlog for the sprint. That means to break a vague unworkable narrative into a specific, measurable, achievable, realistic, and time-bound (SMART) story-and discovering new stories in the process. The result of planning poker is pre-condition, a state to which the backlog needs to conform, to enable a successful sprint.

In many Agile processes an epic is sometimes groomed or broken into stories before the sprint. It’s an honest attempt to get ahead of the game. But, honestly, breaking down an epic without the team playing planning poker means you get all the bad qualities of Waterfall–the qualities that Scrum was created to avoid.

Daily standup (inner loop)

Have you ever been stuck in a status meeting with no ending in sight and most of the participants in the room paying attention to their phones and not the person speaking? The daily standup algorithm was created to banish the status meeting from the realms of humankind.

// pseudo-code implementation of daily standup
accomplishments = List()
today's work = List()
impediments = List()
timer() start for 15 minutes
  for each developer in sprint team {
    announce() accomplishments, append to team accomplishments list
    announce() today's work, append to team today's work list
    announce() impediments, append to team impediment list
  if timer() rings {
    throw standup duration error()
timer() stop

I personally think this algorithm works for all types of work, not just development. Without a strict, formal model to follow, status meetings become planning meetings, brainstorming meetings, complaint sessions, political battle grounds, ad in finitum.

High performing Scrum teams hardly ever drift from the classic daily status formula as described by Jeff Sutherland. Unfortunately, I’ve seen struggling teams given into temptation and turn a good daily standup into a bad trouble shooting meeting. Don’t do it! Go around the room, check the boxes, and follow up with a cool head after all the accomplishments, today’s work, and impediments have been collected (so you know to start with the most urgent issues).

Retrospective (post-condition)

I have to admit that the retrospective is my favorite part of the sprint process. If you do it well and stick to the algorithm a poor performing Scrum process naturally evolves into a high performing Scrum process.

// pseudo-code implementation of daily standup
keep doing = List()
stop doing = List()
change = List()
for each member in sprint team {
  // includes product owner, devs, any other core team members
  announce() what went well, append to the keep doing list
  announce() what didn't go, append to the stop doing list
  announce() what needs to change, append to change list

Like the daily stand up it takes a surprising amount of resolve to stick to the plan and not turn the retrospective into a war crimes trial or a cheerleading exercise. Oddly, the other major problem with the retrospective is lack of follow-up! We get these great lists of things to repeat, to stop repeating, and to change but many times they go nowhere.

It’s important to drive the items on each list into SMART territory so that a manager can do something about them. Noting that “the backlog was not well groomed” or “the stories needed more refinement” just isn’t enough signal to result in a meaningful change. And, of course, there are issues that can’t or won’t change. They have to be worked around.

While the retrospective is very much like a computational algorithm your response to its findings has to be creative and bold. After every retrospective I expect a scrum master to barge into my office, interrupt whatever I’m doing, and hand me a list of what must change. It’s the one output of the Scrum process that an engineering manager can participate in and it doesn’t happen often enough!

As our heroes, the algorithms of Scrum, walk arm-in-arm into the sunset let’s review the basic tenet of we what learned: The more you treat the sprint, planning poker, the daily standup, and the retrospective like the gears in a clockwork engine, the faster that engine runs. There is plenty of room outside of these algorithms but resist the temptation to add value. You’ll probably be surprised at how Scrum works when you respect it and don’t try to fix it.

In our next and final installment of ITSAM I’m going to actually talk about management: If a manager’s job doesn’t involve giving orders, taking temperatures, and holding people accountable–what is her job? Why do we even need managers if we have Scrum?

Agile Principles

Introduction to Scrum and Management (Part 4 of 5 or 6)

Our story so far: in part 3 I described the Scrum team as a data structure-an undirected graph. I tried to show how the properties of an undirected graph predict how a Scrum team behaves and how it can be optimized for productive behavior. Part of that optimization is keeping teams small, eliminating hubs, and breaking the sprint if anything doesn’t go as planned. Undirected graphs are harsh but if we respect them, they will reward us.

Today we’re looking at the third major data structure of Scrum: the story point. OMG! Let me just say that story points are the most powerful and most misunderstood idea in Scrum. Because story points are expressed as integers, its hard for even for experience engineering managers like me not to mistake them for integers.

The story point

This series of blog posts has become for me, my own A Song of Ice and Fire. Author George R.R. Martin originally estimated that he was writing a trilogy. But as Martin started writing, the series became six and now seven books. Honestly, I don’t trust Martin’s estimate of seven books. Given how popular “Game of Thrones” has become, if Martin lives forever, I expect he will be writing ASOIAF books forever.

When I started out writing an Introduction to Scrum and Management, I took my detailed notes from reading Jeff and JJ Sutherland’s book Scrum: The Art of Twice the Work in Half the Time and estimated I could express myself in three blog posts, maybe four just to be on the safe side. I need to time box my projects as spending too much time on any one project steals valuable time from others. As you can see from the subtitle of this post (Part 4 of 5 or 6) my estimate of the number of parts continues to increase. My project is over budget and the final post is delayed!

Jeff Sutherland, a good engineering manager, knows that people are terrible at estimating effort. Sutherland knows that less than one third of all projects are completed on time and on budget. He also knows that there are many reasons for this (poor work habits, under- or over-resourced teams, impediments that never get addressed) but the root cause is our inability to estimate timing (unless we have done the task before and have transformed it into a repeatable process).

The problem with writing fantasy novels and software is that they are not repeatable processes.

This is why Sutherland invented story points and George RR Martin still write his novels with WordStar running on a DOS PC. Since Sutherland and Martin cannot control the creative process, they put constraints around it.

The story point was invented by Jeff Sutherland because human beings really can’t distinguish between a 4 and 5. Jeff was looking for a sequence of numbers where the difference between each value was intuitive. Jeff realized that the Fibonacci numbers, a series of numbers that are known as the Golden Ratio, were the perfect candidates to do the job of estimating work. Art lovers, architects, mathematicians, and scientists, all agree that the world around us is built upon a foundation of Fibonacci numbers.

I could muse for endless paragraphs on how Fibonacci numbers are so elegant that they enable artists and artichokes alike to create beautiful compositions. But let’s just take Fibonacci numbers for granted and see how they are used to implement story points.

Here are the first eight Fibonacci numbers. It is easy to see that as the numbers increase in value the difference between each number increases. This acceleration in difference is in harmony with our ability to detect fine differences at a small scale but not a large scale.

1, 1, 2, 3, 5, 8, 13, 21

Each number in the sequence is the sum of the pair of numbers that immediately proceed it. You can do the math if you don’t want to take my word for it!

A diagram of Fibonacci squares shows the magnitude of Fibonacci progression nicely.

But let’s back up a bit. Why do we need Fibonacci numbers? We’re developing software not paintings or artichokes!

In Scrum a story is a simple description of a chunk of work to do. A sprint is a repeating and limited duration of time in which to do work. Since the work to be done is creative, it can’t fully be understood until the worker is doing it. Thus Scrum constrains process of doing the work but not the work itself.

In summary

  • Stories constrain the definition of work
  • Sprints constrain the time allotted to work
  • Story points constrain the amount of work based on a story that is planned to be executed during a sprint.

If you have done something before, and absolutely nothing has changed, then you don’t need story points. But almost all software development projects involve new requirements, new technologies, and new techniques. When planning a software development project, the big problem is where to start. It’s hard to know how to break down a big project into nicely workable chunks.

Story points get the developers and product owner talking about where to start and how to break the problem down. In discussion during the sprint planning meeting, 13-point stories are broken into several 8-point stories. 8-point stories are broken down into many 5-pointers. And so on until all that is left are dozens if not hundreds of 1-point stories (which are, by their nature, very well understood stories).

Scrum masters and engineering managers know that a 13-point story isn’t dividable into one 5-pointer and one 8-pointer! A backlog of story points is not communicative, associative, or distributive like the ordinary numbers we grew up with. Story points can’t be added, subtracted, multiplied or divided together.

We also know that one team’s 13-point story is another team’s 21-point story. Story points are relative to the team, they change in value as the team gets better (or worse), and are not comparable unless the same people have worked together on the same project for hundreds of sprints.

As a data structure the enumerated values of story points are a wonderful set of flags, where the different between each flag is intuitive. Story points are signals not units.

Alright, this blog post was a bit long but in my defense story points are a nuanced concept. I think we’re just about at the end–which should be a relief to all of us. The good news is that my ability to estimate has significantly improved by doing the work. In the next blog post I’m going to talk about the Algorithms of Scrum.

Next, the penultimate episode, part 5

Agile Principles

Introduction to Scrum and Management (Part 3 of 5 or 6)

Ah, I can see from those weary, sleepy eyes, that like me, you are obsessed with improving your team’s WIP (work in progress). Stick with me and we’ll get to the bottom of the productivity conundrum with the power of our computational thinking!

In part two, I listed the three data structures and four algorithms of Scrum as described in Jeff and JJ Sutherland’s book Scrum: The Art of Twice the Work in Half the Time. I also dug deeply into the first data structure, the prioritized backlog, which from a computer science POV looks a lot like a sorted queue. I explained that if you don’t treat the backlog exactly like a queue you break your sprint and have to throw your sprint planning away and start over. Accessing an end of a queue enables O(1) efficiency. Accessing some random element in a queue… well let’s just say there be dragons of O(unknown).

In today’s blog post we’re going to look at the second data structure of Scrum, the team (an undirected graph). Like a queue, I’ll show that if you don’t treat this data structure with respect your Scrum process will fail, your sprints will leave story points on the table, and your stakeholders will demand status reports and commitments to dates!

The team

Every Scrum team is a communications network where the nodes are the people and their communication patterns are an undirected graph. Undirected here means there is no direction to the edges between nodes. Undirected communication is what you want in a Scrum team.

In the Waterfall days a manager would get all the requirements, analyze the work, and dole it out to the team members. If a team member had a question, she had to ask the boss for clarification. That kind of communications network is known as a directed graph and in particularly bad organizational patterns it becomes hub and spoke where bosses talk to bosses and team members talk to bosses and all communications require one or more hops before an answer arrives. This creates latency (delays in responses) and error (as each hop adds the opportunity for misunderstanding). 

Scrum avoids the hub and spoke model by eliminating the manager role. Any team member can talk to any other team member. Manager approval is not needed or even available. There are no hops and questions can be answered in real-time.

There is, however, a downside to a communications network based on an undirected graph model: limited scale.

Growth of Nodes and Edges in an Undirected Graph

If the team has 1 person, she only has to communicate with herself-which I assume is a low latency, high bandwidth connection. If the team contains 2 people, there is 1 bi-directional communication connection, or edge, between person 1 and person 2. 

So far so good! But as you add people to the team the number of potential connections between them increases with an accelerating growth rate of n * (n-1)/2. If we drop all the constants, we get O(n^2)-quadratic complexity. This means with each additional team member, communications become more and more difficult-if not impossible.

A team of 10 people creates an almost intolerable communications situation! There are 45 possible edges in an undirected graph with 10 nodes. This means a great deal of potential chatter, as many as 45 conversations happening simultaneously, with each person having to juggle threads with up to 9 other people. This also means treaded conversations in a large chat room become unreadable.

Large Team Communications Scale

Jeff Sutherland knows all this. He’s a CTO. Scrum, as Jeff created it, requires you to keep the team small. As small as possible. 

This is also why you can often speed up a project by reducing the number of people involved. If a team of 10 is reduced to 8, then there are roughly 38% less possible conversations and each team member only has to ask up to 7 people (in the worst case) a question before she finds someone who can give her the answer. Theoretically, 8 people will accomplish less story points per sprint than 10 people. In practice communication efficiency gives a real-world advantage to the smaller team.

Small Teams Communications Scale

I want to empathize that not having a hub (a boss) and keeping the team small (less than 10) are hard requirements of Scrum. If you need a bunch of managers (engineering manager, project manager, scrum master, product owner) supervising the team you’re adding latency, error, and hops to your undirected graph. This is also why the ideas that the “product owner is the CEO of the team” or that the “Scrum master is the Engineering Manager” are bad ideas.

Now wait a minute, Mr. Pavley! We need all these bosses! What if something goes wrong during the sprint? What if a story is wrong? What if new work comes in? What if an engineer needs help that her team members can’t supply?

Break the sprint. Redo the plan. Start over. Design a new team and a new backlog. Before and after the sprint bring in all the bosses you want! Just leave the team alone during the sprint.

If you are still as excited as I am about getting Scrum to actually work as advertised, the next installment of Introduction to Scrum and Management will explore story points and Fibonacci numbers, the best numbers in the whole world!

And here is part 4!

Agile Principles

Introduction to Scrum and Management (Part 2 of 4 or 5)

Welcome back! In part one, I expressed my dismay that Scrum was conceived with no formal role for management, especially not Engineering Management. I also claimed that Scrum is Agile, that Scrum is not dead, and that Scrum was created long before the hyperconnected Internet we now inhabit came into being. I found that Jeff and JJ Sutherland’s book, Scrum: The Art of Doing Twice the Work in Half the Time, helped me, after years of supporting Scrum, actually understand Scrum.

In this blog post I want to dig a little deeper into how Scrum works from an engineering perspective. When I told my team (in our internet chat room) that I was reading the Sutherlands’ book, one of the comments I got went something like this: “2x the work in 1/2 the time… that sounds too good to be true!”

And you know something? The that engineering manager was right: “Twice the work in half the time” is what a gonzo diet supplement promises. Any engineer with good critical thinking skills is going be skeptical of a process that promises to break the laws of Physics.

But Scrum is not snake oil. I’ve seen it work time and time again. I’ve also seen it fail. What separates a successful Scrum process from an unsuccessful one? And by what mechanism does Scrum accelerate work? Let’s find out!

As an engineer I think of any system, even a human run system like Scrum, in terms of data structures and algorithms. These are the building blocks that determine how a system will scale deal with bottlenecks. We can even apply Big O analysis to Scrum and see if we can predict where it will be efficient, O(1), and inefficient, O(n!).

Scrum at its heart is a computational model for work. In this model Scrum has three primary data structures and four primary algorithms.

Data StructuresAlgorithms
The prioritized backlog (a sorted queue)Sprint (outer loop)
The team (an undirected graph)Planning Poker (pre-condition)
Story points (a set of Fibonacci values)Daily Standup (inner loop)
Retrospective (post-condition)

The well-known properties of these data structures and algorithms help Scrum operate efficiently but also point to why Scrum is hard to scale. 

The prioritized backlog

A sorted queue is very fast to access: no searching needed. You dequeue one end to get the current element and enqueue the other end to add a new element. As long as you don’t randomly access elements in the middle of the queue you are always assured to get the element with the highest priority–If the elements are added to the queue in order of priority during the sprint planning process. Access is O(1), aka constant time, whether there is one or one million stories.

This is why Scrum requires the backlog to be locked during the sprint: any addition or subtraction means that all your planning efforts were for naught. Changes to the backlog during the sprint is like randomly accessing the elements of a queue. It means that your story points are no longer relative, the team’s rhythm is broken, and predictability is compromised. This is why, if the backlog must change during the sprint, the sprint is “broken” and must be started over with a new queue of work to do. Range out of bounds error

I don’t know about you, but it’s hard to get the team, including the Scrum Master and Product Owner, to break a sprint. We all just sheepishly adjust the backlog. This is especially true when a high priority story becomes blocked and an engineer is sidelined. Instead of breaking the sprint and re-planning, the engineer is usually told to just grab the next story in the backlog–which ruins the whole queue. Access out of bounds error

Right here we can see a hint of a deep computer science basis to Scrum as it was originally conceived by Jeff Sutherland and his co-creators. We can also see why it’s important to stick to the original conception to get the benefits. If you’re not breaking sprints and if you’re access stories randomly during a sprint the efficiency of this part of the process jumps from 0(1), which is a good as it gets, to some other O(n) like O(n^2) or the dreaded O(n!). Take it from an Engineering Manager: you don’t want to go there!

In the next post we’ll take a hard computational look at the scrum team and prove that less team members is always better than more. If you want to speed up, remove members from the team!

Onwards to part 3!

Agile Principles

The Three Laws of Agile Process

As early as 2007 Agile practitioners, or at least people who blog about Agile, began to observe that we live in a post-agile world. I’m not sure what means but Agile is a conversation about the best way to manage the software development process that has been going on for a long time. Has it been too long? What comes after Agile?

Eventually all wisdom becomes dogma.  Many engineers feel strongly that we’re already there with the process for process’ sake when the scrum master moves into the cubicle next door. Fair or unfair it’s a criticism needs to be explored. However nobody has put forth a process to succeed Agile–one that builds upon it’s  strengths  and shores up it’s weaknesses. At this point in the history of software  development  were we to  abandon  Agile we might as well go to ouija boards and seances to plan our projects.

So in truth, we make it up as we go along. That’s what most experienced scrum masters mean when they say they are flexible about adapting the processes of Agile to the culture and business needs of an organization. Poor Agile implementations are only a few second-guesses away from good ones.

Perhaps it’s time to take what we’ve learned in the 10 years since the framers of the Agile Manifesto met in Snowbird and create something that fits even better with the technological and business climate of the Mobile-Cloud-Long Tail-Social-Search Driven-Virtual world we log into every morning.

I’d like to start with the very first value of the Agile Manifesto:

Individuals and interactions over processes and tools

Do we still value people over process? Is it that simple? From this value the principles of customer satisfaction, working with business people, building projects around motivated  individuals, face-to-face conversation, sustainable development, and team reflection are derived. Change this value and we change the equation for all of Agile. Do we weaken it? strengthen  it? or replace it with something else?

Even with the unique  challenges  of developing software for servers you can’t touch and cell phones with fingerprints all over them we know what true process-for-process’ sake is like. We call that Waterfall but it was many more manifestations. It’s the paperwork you fill out in the emergency room and the electronic EULA you click “agree to” without ever reading.

So we have to keep it. In fact I would argue we have to make it much more  imperative: A rule that is never broken which says it’s OK to break the rules!

I don’t mean we should give up our process at the drop of a hat (or the crash of a market). But we must give it up when our people will be rendered ineffectual. Issac Asimov’s Three Laws of Robotics can inspire us with a new rendering of what is, perhaps, the most important value of Agile:

  1. A process may not injure a team or, through inaction, allow a team to come to harm.
  2. A process  must obey any orders given to it by a team, except where such orders would conflict with the First Law.
  3. A process  must protect its own existence as long as such protection does not conflict with the First or Second Law.

If it will keep our robot overlords from killing us it will work for keeping an overzealous  Agile process in check!

Agile Principles

Communication Flavors

Found a Bug

It’s not even funny how human networks and computer networks exhibit the same behaviors. I know we’re all patiently waiting for the singularity, for the event where computer networks become sentient and life as we know it changes forever. But given how the current example of networked sentient beings behave I fear that switching platforms (from organic to silicon) won’t improve anything. Collaborators will still compete, confuse, and confound each other, just at faster bit rates.

Dynamic network communication behavior is important to help us understand the 6th principle of the Agile Manifesto:

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

At first glance the idea that F2F (face-to-face a type of peer-to-peer) communication is the most efficient and effective method of communication is non-obvious. After all, if I want to tell something to many people a broadcast is much quicker than a pointcast. Implicit in the idea of F2F communication is talking. (I doubt the framers of the Agile Principles want us to pass notes to each other.)

It’s not immediately clear that talking (realtime streaming)  it is much more effective than detailed written instructions (store and forward). F2F communication means lots of clarifications, and repetitions when words and whole sentences can be lost due to network noise or an inattentive client.

The problem is that we are comparing F2F and written instructions in terms of abstract efficiency outside of the time-bound world we live in. This is a common error. We tend to think of our priorities and tasks as critters that live in independent time-space bubbles. Thus we don’t take into account logistics, dependencies, and elapsed time when estimating how effective or efficient a flavor of communication is.

There are not 24 hours in a work day. Really there are 4. The other 20 are taken up with sleeping, eating, commuting, and status meetings. That leaves 4 uninterrupted hours for work. Imagine if you baked this idea into your communications habit. You don’t have the time waste on writing or reading long, detailed emails and wiki posts. You need a communication strategy where you can cut to the important stuff. That means finding the right co-worker (peer) and talking to them directly.

A true co-worker is someone who knows you’re strengths and weaknesses, knows when you are joking, respects you, and doesn’t want to you see fail. Unfortunately, even if you have super social skills, you can only manage 5-7 relationships like those at a time, and most of these are with your family members. Most of the people you are working with either just don’t know you very well to be able to parse your text get the message without errors. You can’t simply write a detailed enough specification or requirements document to get though the noise.

Given the shortage of time and deep relationships you have to talk to co-workers F2F. You have to interrupt and be interrupted. What seems like a slow process, like a four hour F2F planning meeting, is actually much more efficient than a  100 page document.

Agile Principles Management & Leadership

Volunteer Scrum Master Handbook

Scrum Masters Handbook

I have to disclose upfront that I am more of an Agile guy than a Scrum guy. Which is to say I feel more at home discussing Agile in general than Scrum in particular. (I’m not even sure how to capitalize it–SCRUM or scrum?)

Over the years I’ve made my peace with Scrum and as I dig into it’s theory and history I find I have few real arguments with the brainchild of Schwaber and Sutherland. My arguments are not with Scrum but with Professional Scrum Masters.

Over past 19 years that Agile/Scrum have reshaped the process of software development many of the traditional roles of software engineering have drastically changed or simply gone away. Technical Writers, Business Analysts, and non-coding Development Managers aren’t needed much by self-organizing teams that talk directly to Product Owners and write up their backlogs on sticky notes. But the one big role that Agile/Scrum has made obsolete is the hard nosed, battle tested, two fisted, lock jawed, steely eyed, organization shaking, cat herding Project Manager.

Back in the day it was the Project Manager who got things done. The good engineers were taught to be sheep and good senior managers were turned into putty by the power of the Project Manager. Microsoft’s variety of Project Manager, the TPM, is still a legend in the industry. It was Windows Vista epic failure and not Agile/Scrum that finally dethroned the TPM.

So when an organization finally buys in to Agile/Scrum where do the people who filled the roles of the waterfall process go? Product Managers became Product Owners. Development Managers who could code became team members or even team leads. Spec writers and Business Analysts moved on. Often Project Manager became Scrum Masters–If the non-coding Dev Managers didn’t beat them to it first.

It is where I got my first bad taste of Scrum: Project Managers turned Scrum Masters who were still doing Project Management. These pseudo-Scrum Masters were the ones with the Microsoft Project plans detailing every sprint from here to release. They demanded status explained in non-technical terms during the daily standup. They stuffed the sprint backlog too tightly (or sandbagged it) and argued with Product Owners over priority. What a mess! It was all Bagile and Fragile.

At least with the non-coding Dev Managers who become Scrum Masters the social graces were respected and they generally stayed out of the way (lest a technical question come their way).

To combat the pseudo-Scrum Master I like to use the role of the Volunteer Scrum Master (VSM). This is just one of the team members (either a coder or a tester) who plays the role of part time Scrum Master. It’s a volunteer and rotating position which can be filled by anybody but the Product Owner. Most of the time the VSM writes code or tests. When the need arises the VSM laces up his boots and does Scrum Master things: Calls for help to scare away any chickens, reminds everyone to follow the process, and let’s me or the full time Scrum Master know that those servers need to be updated by Ops. The VMS removes impediments to the success of the sprint by communication not coercion.

It’s not a glamourous job. There’s no extra pay. There is no authority. You don’t get to wear a uniform or the keys to the executive washroom.

While the job of a VSM is simple and straight forward it can be hard for a new recruit to know where the Scrum Master job starts and ends. So I ask each VSM to think of his part time job as a journalistic one. Observe what is going in the scrum. Take notes. Report findings to the full time Scrum Master or one of the Engineering Directors.

Note: Team members should be responsible for managing themselves. A Scrum Master is not Mom or Dad or a Police Officer. The best way to remove impediments is to communicate them to someone who can address them.

Let’s review how the VCM fits into the role of the Scrum Master…

Responsibility Pro Scrum Master Volunteer Scrum Master
The Scrum Master is not the leader of the team In Theory Yes
The Scrum Master teaches the process Yes Has to learn it first
The Scrum Master has no authority and may not make commitments That’s idea but… Definitely yes
The Scrum Master surrenders control to the Product Owner Hopefully 100% Yes
The Scrum Master is an enforcer of rules Sure Nope
The Scrum Master sets up the meetings If she has time Nah, set up your own meetings!
The Scrum Master maintains the backlog and updates the Scrum board Busy work All team members should pitch in
The Scrum Master explains to management what the team is doing Bad idea (It’s the Product Owner’s Job!) Wouldn’t bother
The Scrum Master acts as a buffer When there is no other option Dials 911

As you can see the Pro Scrum Master has to be careful. As the Scrum canon defines her job there is some inherent contradiction with the job of Product Owner and even some chicken-like (i.e., management) responsibilities. The VMS has no such ambiguities. He’s just passing through–his sense of self-worth isn’t caught up with the role.

Using VSM enables an organization to limit the number of Pro Scrum Masters, minimize political struggle, and help the team members learn the game of Scrum (the best way to learn is to volunteer).

Here are some sources that help define a Scrum Master’s Role:
Agile Principles

Agile Fables


I love a good fable. The kind that Aesop used to write with animals acting out human morality tales. Of course, if you do the research, you quickly find out that Aesop didn’t write most (if any of the fables) we ascribe to him and like Shakespeare he probably never existed. I don’t let reality like that spoil my appreciation for Aesop’s work.

For Scrum Masters, Project Managers, and fans of Getting Things Done there is nothing like the fable of the Tortoise and the Hare. The slower, but diligent tortoise wins the race over the quicker, but arrogant hare. The only problem Agile process lovers have with this fable is that it’s not true! The tortoise (Waterfall) always loses to the Hare (Agile).

But we have a paradox: Most chickens (managers, stakeholders, investors) and even some pigs (engineers) agree with Aesop! The tortoise always beats the hare because the tortoise makes a plan and executes it well, even if he is a bit slow. (If the tortoise fails it’s because a chicken yells at her to hurry up and so she speeds up and screws up. Call it the “Fable of the Spineless Tortoise and Impatient Chicken.”)

To resolve this paradox I’d like to tell the “Fable of the Tortoise, Hare, and Monkey.”

A Hare one day ridiculed the Microsoft Project plans and change control documents of the Tortoise. The latter, laughing, said: “Though you be swift as Google Go, I will beat you in a race to deliver working software.” The Hare, deeming her assertion to be simply impossible, assented to the proposal; and they agreed that the Fox (VP of Product Management) should choose the requirements, and define the release. On the day appointed for the race they started together. The Tortoise never for a moment stopped, but went on with a slow but steady build cycle straight to the end of the course. The Hare, trusting to his native widgets, cared little about unit testing and user feedback, and lying down under his desk after long night of coding, fell fast asleep. At last waking up, and logging on as fast as he could, he saw the Tortoise had reached the goal, and was demoing to the Fox.

But the story doesn’t end yet…

By the time the Hare caught up with the Tortoise and the Fox he saw the Tortoise slam her laptop shut and slump away. The Hare asked for an explanation. “Why didn’t he Tortoise win?” The Fox explained that what the Tortoise delivered was very late to market and the requirements were were out of date. “But,” the Fox said, “Happily for all, the Monkey has delivered working working software that meets most of our customer’s high priority needs!”

“Monkey? Where the hell did the Monkey come from?” Exclaimed Hare.

“Ah yes,” sighed the Fox, “While you were crashed out under your desk, the Monkey came and delivered a working prototype. I told her that it wasn’t quite right so the Monkey went away and came back a little while later with a revised prototype. It took a few tries but eventually the Monkey got it right and won the race. In fact the Monkey’s working prototypes helped me understand what I was asking for. You see when I gave the requirements to you and the Tortoise, there were a few stories I missed and during the race a new competitor entered the market. Even if you hadn’t fallen asleep you would not have won as I had to keep moving the goal. Our customer, Mr. Zeno, is a hard man to please!

Just in case you’re not into reading between the lines here’s my analysis of this fable (since I wrote it my analysis should hopefully be accurate).

The Hare lost the race because he had to fight the Law of the Handicap of a Head Start. This principle states that pioneers and innovators lack a clear signal. They are out in front, they have no one to follow, and so their advantage becomes a handicap. Often the first movers are stuck building their software on immature technology that becomes obsolete before it is finished. The United States has this problem with its Internet and telecommunications infrastructure. We got there first but now we are lagging in connectivity.  The Hare development process, starts out fast and light but never stops to get feedback–its one big long sprint. A lot of code is written. Little of it is ever used.

The Tortoise lost the race because we no longer live in the world of the 4th century BC. Slow and steady hasn’t been a winning strategy for a long time. The goal keeps moving. Actually the ancient Greek philosopher Zeno wrote his own fable where he counters Aesop. In Achilles and the Tortoise, the Hare-like Achilles can’t catch up to the Tortoise (who has been given a head start) because every time Achilles reaches the spot where the Tortoise was, the Tortoise has moved on. This is the problem of accomplishing an infinite number of tasks in a finite amount of time. Somehow we do it every day (but if you really think about it your brain hurts).  The Tortoise development process, creates dichotomies every step of the way: Design, Development, Test, Release, Post Release, et etc. Dichotomies are full of infinite regressions and thus are never finished.

The Monkey won the race because he used an Agile development process. First the Monkey took a prioritized subset of from the list of infinite tasks (and every project has a never ending number of bugs and feature requests) and delivered them to the Fox (Product owner) as working software. It was was rough but the Fox had the results in plenty of time to give feedback and course corrections. Thus the Monkey was not subject to the Law of the Handicap of a Head Start. Second the Monkey kept repeating these short sprints of implementing the high priority tasks and getting feedback on a real working system so that as the goal moved she could quickly change course. The Monkey development process, done right, treats every sprint as a whole, where no dichotomies are created. The software is released to the user, it does what it does, feedback is gathered, and the process is repeated until a winner is declared.

Agile Principles Management & Leadership

Managing the De-Motivated


It still amazes me how a process created by engineers for engineers can make so many engineers so unhappy.

I’ve seen all kinds of responses to Agile from engineers. Some are immediately enthusiastic. Others are cautiously optimistic. Many are amused and cynical. And some are down right hostile. Over time the responses polarize and the body of engineers undergoing an Agile process split into two camps: Those for whom Agile is working and those for whom it isn’t.

I’d like to point out that I’m not talking about Fragile (Waterfall Agile) or Badgile (just a plain bad Agile implementation). Fragile and Badgile don’t work for anyone, not the Agile fanboys nor the management types that mistakenly create these processes. (If your boss says, “Agile yes, but we must have controls!” then you’re headed down the road to sadness.)

Unfortunately in my experience there are really good engineers out there for whom Agile in all its manifestations, from Scrum to XP, just doesn’t work. I’ve done a fair amount of work to figure out why and what to do about it. I’m going to use Scrum in my examples but this isn’t a problem of flavor. However you enact the principles the problems are the same for this segment of the engineering population. Also note this isn’t a generational thing. Agile-challenged engineers come in all shapes, sizes, sexes, IQs, and ages.

The top four reasons why Agile doesn’t work for some engineers:

  1. You have to like to talk to people. Engineering as a profession self-selects for people who find human-machine interactions more interesting then human-human or human-animal interactions. People who like to talk to people become salesmen or lawyers not programmers. But for some engineers human-human interaction is down right painful. If you are not into face-to-face communication Agile is not for you.
  2. You have to need and accept help. I love to remind people that intelligence is a side effect of evolution not its goal. (The gold of evolution, if there is one, to enable a species to adapt to a changing environment. Intelligence is just one experiment in adaptation and probably not the most successful one since it’s only worked for one species.) I’ve found that some really intelligent engineers are so smart that it’s difficult for them to accept help (like a prioritized backlog or a bug report) from people they view as obviously less intelligent. You can call this arrogance but then would Mozart have accepted help on his symphonies?
  3. You have to embrace change. Many people from all walks of life have difficulty handling change-rapid or slow. They just don’t like the idea that the world they have finally gotten used to and mastered no longer exists. Some of these change-adverse people happen to be engineers. You can spot them easily. They are programming in languages and paradigms that are no longer on the leading edge. I often have to say to them: Yes, you can still write desktop applications but nobody under 40 is going to care.
  4. You have to like focusing and working on a schedule. Whether your sprint in two weeks or two months isn’t important. What is important is that you stick to it-even it means, and it usually does, modifying your work to fit the time window. For the Agile-challenged engineer this is a blasphemy akin to chopping eight inches off the left side of the Mona Lisa. They see their software as a whole that isn’t finished until it is finished. Scrum’s timing boxing technique is admittedly arbitrary. The creative process cannot be bounded by deadlines without losing its integrity. At least that’s their argument. I see it differently, and can give lots of contrary arguments, but rational argument has little impact on personal preferences. These engineers find sprinting to be stressful where the Agile-friendly engineer finds sprinting to be liberating.

At the end of the day it’s a waste of time to force people to do something they simply don’t want to do. Unenthusiastic people need to find something they are truly enthusiastic about. There is nothing wrong with the Agile-challenged. They are master craftsmen, they are very smart, they are mavens, and they are artists. So what is an Agile organization to do with them?

The first thing to do is get them out of Agile. These engineers are wasted on Scrum and will simply quit and make a lot of other miserable on their way out. Smart unhappy people can make a lot of mischief!

The second thing to do is to create a lab, give them a mission, and turn them lose. It’s interesting to note that many of famous labs in the history of computation have been notoriously bad at implementing and productizing their creations. Xerox Parc invented most of what we use when we use personal computers yet it is in decline. It’s too bad that Xerox Parc didn’t merge with Apple or Microsoft, organizations that are good at creating markets and feeding them incremental improvements. Instead Apple and Microsoft developed R&D labs of their own so that they have a never-ending fountain of new ideas and home for their dreamers.

You can’t and shouldn’t manage the lab with Agile techniques and you must not staff it with Agile-friendly engineers. The engineers who thrive under Agile get lost in lab, their ideas can’t compete with the Leonardos and Michelangelos.

In the end the secret to managing the de-motivated is finding a place for them in the organization where they are motivated to be. Agile is great but it cannot be applied to all problems.