North Star

Successful companies usually have a secret sauce. It could be an algorithm or an insight. But whatever that secret sauce is, it is used to create or disrupt a market.

Apple created the PC market when Steve and Steve figured out that affordable pre-built personal computers would be really useful for consumers. IBM disrupted the PC market that Apple built with the insight that a standard, expandable, business-oriented PC would be especially valuable to businesses. After a while Microsoft disrupted the disrupter with the key insight that PC resources, CPU speed, RAM size, and disk space, were essentially infinite according to Moore’s Law.

Yet secret sauce alone is not enough create or disrupt a market for very long. You might have a brilliant algorithm or insight but if you can’t focus on it and deliver it to your audience then you got nothing.

Secret sauces are a common and cheap. The ability to focus and deliver is rare and expensive!

Let’s take the case of Google. Larry and Sergey started Google with the idea of Page Rank. They turned that idea into a set of algorithms and code and turned it loose on the web. Suddenly Larry and Sergey had the best search engine on the market.

But Page Rank on its own didn’t create Google. This might be hard to believe today but when started Google it was an underdog. Google was the epitome of a scrappy startup that hardly anyone paid any attention to.

Luckily Larry and Sergey had something else: A north star.

I don’t know if they called it a “north star”. That’s what we call it now. They probably didn’t call it anything. Looking back, I think Larry and Sergey, Like Steve and Steve, and all successful market creators/disrupters had an intuitive sense of focus and delivery that was superhuman. They got everyone around them, investors, employees, and partners, to focus on search and to think hard about the best way to deliver search to the consumer. They followed their north start to the detriment of everything else including sleep, civility, and revenue.

Obviously it paid off. Once the nascent search market was disrupted Google attained all the things they had sacrificed. They made money. They decided to be really nice. They got a good night’s sleep.

I see this pattern repeating though out the boom and bust cycle of business. When a company is following it’s north star it eventually becomes successful. When a company is distracted or tries to follow too many stars it eventually fails.

When I worked at Apple in the 90s our north start was summed up in the question, “will it sell more Macintoshes?” If you could answer “yes” then you had tacit approval to do it. Don’t ask. Just do it. HyperCard, QuickTime, TrueType, Unicode, these are all examples of technologies that “sold more Macintoshes.”

At the time I was working on ClarisWorks for Kids. It was a bit like Microsoft Office for the K-12 market. Our theory was that productivity software tools for kids would sell more Macintoshes (to parents and schools) and so I was asked to go and do it. I didn’t fill out a product plan or forecast revenue. I just convinced a group of engineers that ClarisWorks of Kids was cool and off we went. I hired as many people as I needed. I figured out features and even helped design the box art. Since I had a north star, I didn’t have to be managed. My boss was more like my personal coach. I went to him for advice and not orders.

Since I had never shipped a product before I made a few mistakes. I didn’t get fired. As long as I was following Apple’s north star everyone had trust and confidence in what I was doing. And I wasn’t special. I was one of hundreds of Apple engineering managers leading projects in partnership with hundreds of engineers all following a single north star.

ClarisWorks for Kids turned out to be a big hit. We won some awards. More importantly we sold a lot of Macintoshes. ClarisWorks for Kids was part of an  educational bundle that filled up computer classrooms across the world with Power PC-based Power Macs.

But then we turned away from our north star.

In the late 1990’s Apple’s marketshare continued to slip. In spite of all our focus and smart insights we were not sell enough Macintoshes. Risc chips, CD-ROMs, and built-in digital signal processors were not cutting it with the consumer. Most people bought IBM compatible PCs that ran Windows.

Instead of doubling down on our north star or discovering a new north star we at Apple decided to pursue many different strategies. Sometime we would follow multiple strategies at the same time but usually it was a new strategy every month. Some of these new “stars” included “Mac is the best PC” and “Let’s find more ways to make money from our existing users” and “Apple is really a software company!” Ouch. None of these stars become north stars. They were more like fly-by-night comets that burnout by dawn.

Without a strong north star, I no longer manage myself. I had to be told what to do. Once day I was told to “port Claris Works for Kids to Windows.” I asked how this project would “sell more Macintoshes?” Apparently Apple wasn’t concerned about that old idea any more and frankly I had not been asked for an opinion.

So we gritted our teeth and cracked open the Windows 3.1 disks and started porting. It was kinda of fun and a huge technical challenge as the Mac programming model was very different from Windows. So we dug into it. As an engineering manager there wasn’t as much for me to do so I got into project plans and status reports. I don’t think anyone read them. At some point we were done. ClarisWorks for Kids could now run under Windows on IBM PCs.

This is the point where we were all laid off. Nothing personal. Business was bad, new management was in town (Steve was back), and Windows software was not needed. It didn’t “sell more Macintoshes” because it didn’t run on a Macintosh.

After we were gone Apple got back in the business of following it’s original and true north star. Mac computers become exciting again with bold design and a new UNIX-based operating system. (OK an old UNIX-based OS but it brought the goodness of UNIX to a mass market.)

ClarisWorks and ClarisWorks for Kids were gone but Apple replaced them with a suite of productivity tools. Pages, Keynote, and Numbers are the great-grandchildren of ClarisWorks. I don’t know if they “sell more Macintoshes” but they have some cool features. Besides, Apple’s north star now is probably “Does it sell more iPhones?” or something like that.

These days I work really hard to provide a north star to my teams and to advocate for a north star in my organization. A good north star is easy to understand and easy to remember. A great north star enables employees to mange themselves and renders budgets and project plans obsolete. An awesome north star fuels growth and turns businesses around.


Telling Time as an Engineer

Time is the most precious resource. It’s in limited supply, once spent we can’t get it back, and you can’t trade it directly. This might sound a little radical but most global, national, business, and personal problems, seem to me, to boil down to problem of time and who’s time is more important than yours.

Before we can decide how we should spend the time given us, we have to put some thought into the process of analysis of tasks which take time. In software development a considerable amount of thinking has been applied to just this analysis. We usually call it “process management” and “time management”. Many methodologies have been created to solve the problem of time and yet when the rubber hits the road the management of time, which includes task prioritization and effort estimation, is full of errors and random results.

A great example is the Agile Development Processes, which has become the standard as well as declared dead by many of its original creators. Why is this?

Here is a simple example…

A high priority story is pulled from a general backlog and estimated, along with other stories, by an experienced engineering team. A product owner then weighs the cost of the stories based on the effort estimations and value to the business and feed them into a sprint backlog. The engineering team then works on each story, in order of priority, and completes the required stories by the end of the sprint. The work completed is demoed to the stakeholders and everyone is happy as everyone’s time has been well spent.

Well, except this happy plan almost never happens.

Something like 80% of all feature and products are delivered late or not at all. And often when a feature or product is delivered its buggy enough that we regret delivering it on time if at all. I’m sure Samsung engineers are less concerned about deadlines these days and more concerned about taking the time to do their tasks with more quality. Blizzard has made a billion dollar business of never giving dates for games and missing them when they do. Facebook and Spotify just spring new feature on their users without any warning and kill bad ones before they spread beyond a small segment.

It my opinion successful tech companies don’t bother with time management and leave schedules and task estimates to unsuccessful tech companies. I’m not saying successful tech companies don’t do agile or create project plans. I’m saying these are more like historical accounts and data gathered for analysis than pseudo-predictive planning.

Why is task estimation so non-predictive?

The problem is that it’s impossible to know how long a task will take unless you have done exactly that task before. When I worked at Apple Computer (before it was just Apple) we said that in order to to understand how long a project would take you had to build it and then write the schedule.

This is why experienced engineering team is so important in effort estimation. If you get a group of engineers who are a bit long in tooth they can work together to pool estimates on work they have performed previously.

But much of the work of an experience engineering team is work they have never done before. Experience engineers tend to see everything though the lens of previous experience. The result is that effort estimates are inaccurate as they have mistaken a novel task for an nostalgic task. I can’t count the number times I have said, “I thought the problem was X and it would take Y story points, but the problem is really Z and I’m still doing the research so your guess is as good as mine.”

The fact that for novel work your guess is as good is mine is why startups of inexperience engineers succeed with problems that mature companies fail at. The boss says “This problem will take too long to get to market. Let’s just not do it. It’s a waste of time.” The boss also says, “Hey brilliant engineers, you didn’t deliver this product on time! You suck! You’re fired.” Both judgements are typical of mature companies where value has to be delivered every quarter and experimental failure damage previous reputations.

In a typical tech startup, or any kind of new business, if you did the estimates you would have never started down the path. But startups don’t care! They are labors of vision and love usually staffed by people without experience who don’t know better. A good startup certainly doesn’t worry about effort estimates or punish engineers for not being able to tell time.

My advice to any engineering team that needs to worry about time is as follows:


You need a mix of experienced and inexperienced engineers on the team. This doesn’t mean old and young as much as people who have done it before and people who have not. Mix your insiders with your outsiders. For example if you’re building a web app bring in a a few mobile devs to the sprint planning. And some interns. And listen to the outsiders. Engage in a real discussion.


If someone in charge wants to know how long a novel task will take from just the title of the task, without any true discussion, walk away. You’re going to give them a wrong answer. By the way good estimates are rarely rewarded–they are expected! But bad estimates are almost always punished. An honest “I don’t know” is always better than “2-3 weeks” or “2-3 story points”.


Remember there is no value in hitting the deadline without quality, performance, or value to the user. In fact I’m always a little suspicious of teams that never miss a deadline. Apps that crash or cutting scope to the point of no visible progress is a hallmark of teams that hit their deadlines. I’m not saying don’t work hard or try to hit your deadline just be tough about the result at the end of the schedule: Give it more time if needed!


The problem with my advice is that everyone wants to know the schedule. So many other functions depend on the schedule. Releasing a product on time is critical to the business. So my final piece of advice, and you’re not going to like it, is let the business set the deadline. Instead of wasting everyone’s time upfront with a broken planning process to arrive at a deadline, get the deadline first and work backward with effort estimates. While time is limited the amount of time we spend on on a task is flexible. We work differently if we have 3 months, 6 months, or 12 months to accomplish a task. Ask any college kid how much time they put into their studies at the end beginning of the semester when time seems unlimited vs the end of the semester when time is in short supply.

Time is always in short supply.

Volunteer Scrum Master Handbook

Scrum Masters Handbook

I have to disclose upfront that I am more of an Agile guy than a Scrum guy. Which is to say I feel more at home discussing Agile in general than Scrum in particular. (I’m not even sure how to capitalize it–SCRUM or scrum?)

Over the years I’ve made my peace with Scrum and as I dig into it’s theory and history I find I have few real arguments with the brainchild of Schwaber and Sutherland. My arguments are not with Scrum but with Professional Scrum Masters.

Over past 19 years that Agile/Scrum have reshaped the process of software development many of the traditional roles of software engineering have drastically changed or simply gone away. Technical Writers, Business Analysts, and non-coding Development Managers aren’t needed much by self-organizing teams that talk directly to Product Owners and write up their backlogs on sticky notes. But the one big role that Agile/Scrum has made obsolete is the hard nosed, battle tested, two fisted, lock jawed, steely eyed, organization shaking, cat herding Project Manager.

Back in the day it was the Project Manager who got things done. The good engineers were taught to be sheep and good senior managers were turned into putty by the power of the Project Manager. Microsoft’s variety of Project Manager, the TPM, is still a legend in the industry. It was Windows Vista epic failure and not Agile/Scrum that finally dethroned the TPM.

So when an organization finally buys in to Agile/Scrum where do the people who filled the roles of the waterfall process go? Product Managers became Product Owners. Development Managers who could code became team members or even team leads. Spec writers and Business Analysts moved on. Often Project Manager became Scrum Masters–If the non-coding Dev Managers didn’t beat them to it first.

It is where I got my first bad taste of Scrum: Project Managers turned Scrum Masters who were still doing Project Management. These pseudo-Scrum Masters were the ones with the Microsoft Project plans detailing every sprint from here to release. They demanded status explained in non-technical terms during the daily standup. They stuffed the sprint backlog too tightly (or sandbagged it) and argued with Product Owners over priority. What a mess! It was all Bagile and Fragile.

At least with the non-coding Dev Managers who become Scrum Masters the social graces were respected and they generally stayed out of the way (lest a technical question come their way).

To combat the pseudo-Scrum Master I like to use the role of the Volunteer Scrum Master (VSM). This is just one of the team members (either a coder or a tester) who plays the role of part time Scrum Master. It’s a volunteer and rotating position which can be filled by anybody but the Product Owner. Most of the time the VSM writes code or tests. When the need arises the VSM laces up his boots and does Scrum Master things: Calls for help to scare away any chickens, reminds everyone to follow the process, and let’s me or the full time Scrum Master know that those servers need to be updated by Ops. The VMS removes impediments to the success of the sprint by communication not coercion.

It’s not a glamourous job. There’s no extra pay. There is no authority. You don’t get to wear a uniform or the keys to the executive washroom.

While the job of a VSM is simple and straight forward it can be hard for a new recruit to know where the Scrum Master job starts and ends. So I ask each VSM to think of his part time job as a journalistic one. Observe what is going in the scrum. Take notes. Report findings to the full time Scrum Master or one of the Engineering Directors.

Note: Team members should be responsible for managing themselves. A Scrum Master is not Mom or Dad or a Police Officer. The best way to remove impediments is to communicate them to someone who can address them.

Let’s review how the VCM fits into the role of the Scrum Master…

Responsibility Pro Scrum Master Volunteer Scrum Master
The Scrum Master is not the leader of the team In Theory Yes
The Scrum Master teaches the process Yes Has to learn it first
The Scrum Master has no authority and may not make commitments That’s idea but… Definitely yes
The Scrum Master surrenders control to the Product Owner Hopefully 100% Yes
The Scrum Master is an enforcer of rules Sure Nope
The Scrum Master sets up the meetings If she has time Nah, set up your own meetings!
The Scrum Master maintains the backlog and updates the Scrum board Busy work All team members should pitch in
The Scrum Master explains to management what the team is doing Bad idea (It’s the Product Owner’s Job!) Wouldn’t bother
The Scrum Master acts as a buffer When there is no other option Dials 911

As you can see the Pro Scrum Master has to be careful. As the Scrum canon defines her job there is some inherent contradiction with the job of Product Owner and even some chicken-like (i.e., management) responsibilities. The VMS has no such ambiguities. He’s just passing through–his sense of self-worth isn’t caught up with the role.

Using VSM enables an organization to limit the number of Pro Scrum Masters, minimize political struggle, and help the team members learn the game of Scrum (the best way to learn is to volunteer).

Here are some sources that help define a Scrum Master’s Role:

Managing the De-Motivated


It still amazes me how a process created by engineers for engineers can make so many engineers so unhappy.

I’ve seen all kinds of responses to Agile from engineers. Some are immediately enthusiastic. Others are cautiously optimistic. Many are amused and cynical. And some are down right hostile. Over time the responses polarize and the body of engineers undergoing an Agile process split into two camps: Those for whom Agile is working and those for whom it isn’t.

I’d like to point out that I’m not talking about Fragile (Waterfall Agile) or Badgile (just a plain bad Agile implementation). Fragile and Badgile don’t work for anyone, not the Agile fanboys nor the management types that mistakenly create these processes. (If your boss says, “Agile yes, but we must have controls!” then you’re headed down the road to sadness.)

Unfortunately in my experience there are really good engineers out there for whom Agile in all its manifestations, from Scrum to XP, just doesn’t work. I’ve done a fair amount of work to figure out why and what to do about it. I’m going to use Scrum in my examples but this isn’t a problem of flavor. However you enact the principles the problems are the same for this segment of the engineering population. Also note this isn’t a generational thing. Agile-challenged engineers come in all shapes, sizes, sexes, IQs, and ages.

The top four reasons why Agile doesn’t work for some engineers:

  1. You have to like to talk to people. Engineering as a profession self-selects for people who find human-machine interactions more interesting then human-human or human-animal interactions. People who like to talk to people become salesmen or lawyers not programmers. But for some engineers human-human interaction is down right painful. If you are not into face-to-face communication Agile is not for you.
  2. You have to need and accept help. I love to remind people that intelligence is a side effect of evolution not its goal. (The gold of evolution, if there is one, to enable a species to adapt to a changing environment. Intelligence is just one experiment in adaptation and probably not the most successful one since it’s only worked for one species.) I’ve found that some really intelligent engineers are so smart that it’s difficult for them to accept help (like a prioritized backlog or a bug report) from people they view as obviously less intelligent. You can call this arrogance but then would Mozart have accepted help on his symphonies?
  3. You have to embrace change. Many people from all walks of life have difficulty handling change—rapid or slow. They just don’t like the idea that the world they have finally gotten used to and mastered no longer exists. Some of these change-adverse people happen to be engineers. You can spot them easily. They are programming in languages and paradigms that are no longer on the leading edge. I often have to say to them: Yes, you can still write desktop applications but nobody under 40 is going to care.
  4. You have to like focusing and working on a schedule. Whether your sprint in two weeks or two months isn’t important. What is important is that you stick to it—even it means, and it usually does, modifying your work to fit the time window. For the Agile-challenged engineer this is a blasphemy akin to chopping eight inches off the left side of the Mona Lisa. They see their software as a whole that isn’t finished until it is finished. Scrum’s timing boxing technique is admittedly arbitrary. The creative process cannot be bounded by deadlines without losing its integrity. At least that’s their argument. I see it differently, and can give lots of contrary arguments, but rational argument has little impact on personal preferences. These engineers find sprinting to be stressful where the Agile-friendly engineer finds sprinting to be liberating.

At the end of the day it’s a waste of time to force people to do something they simply don’t want to do. Unenthusiastic people need to find something they are truly enthusiastic about. There is nothing wrong with the Agile-challenged. They are master craftsmen, they are very smart, they are mavens, and they are artists. So what is an Agile organization to do with them?

The first thing to do is get them out of Agile. These engineers are wasted on Scrum and will simply quit and make a lot of other miserable on their way out. Smart unhappy people can make a lot of mischief!

The second thing to do is to create a lab, give them a mission, and turn them lose. It’s interesting to note that many of famous labs in the history of computation have been notoriously bad at implementing and productizing their creations. Xerox Parc invented most of what we use when we use personal computers yet it is in decline. It’s too bad that Xerox Parc didn’t merge with Apple or Microsoft, organizations that are good at creating markets and feeding them incremental improvements. Instead Apple and Microsoft developed R&D labs of their own so that they have a never-ending fountain of new ideas and home for their dreamers.

You can’t and shouldn’t manage the lab with Agile techniques and you must not staff it with Agile-friendly engineers. The engineers who thrive under Agile get lost in lab, their ideas can’t compete with the Leonardos and Michelangelos.

In the end the secret to managing the de-motivated is finding a place for them in the organization where they are motivated to be. Agile is great but it cannot be applied to all problems.

In Search of the Motivated


I’m doing a lot of hiring right now. So much that I’ve had to form a hiring scrum and treat it like an engineering project. The scrum is doing a great job. We now have a systematic way to write job descriptions, assign interviewers, evaluate resumes, perform phone screens, and conduct interviews. All this to find the needle in the haystack: The motivated individual.

There are other requirements: The ability to code with grace and manage with charm is appreciated. Good manners and excellent communication skills a plus. Experience with open source, P2P networks, and super-scalable n-tiered web stacks don’t hurt your chances.

But all that means nothing without the man on a mission to drive it (or woman on a mission. I hear there are female programmers out there. There should be a lot more. Something is wrong with the world when less than 25% of programmers are women. We need to fix that).

Why is motivation so important? Let’s turn to the Agile Manifesto to find out. Principle 5 states:

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

This principle is in the form of a imperative. No reasons are given and a lot is asked of management. Shouldn’t the framers of the manifesto give us a few hints as to why we chickens should obey this command? It’s almost as if they expect us to know the basis for this principle a priori. Anyone with even a few months of experience in software development knows why the motivated individual is so important that you give him (or her) the keys to the executive bathroom, a company car, and the employee-of-the-month parking spot:

  • Unmotivated people, no matter how good they are at coding, managing, manners, communication, open source, p2p, or n-tier architectures get nothing of consequence done.
  • Motivated people, work wonders no matter how poorly they code, manage, or communicate! And if you find highly motivated l33t hax0rs then the world is your osyter iPad.

I don’t know of any organizations that don’t strive to hire the motivated. They put up all sorts of road blocks to filter out the underachievers. 12 hour interviews, brain teasers, reference checks all designed to make sure you really want the job. One startup I worked at would call applicants late on a Friday and ask them to prepare a presentation over the weekend and present it early the Monday morning as part of the interview process for salespeople. This was a brilliant filter. Only those  hunter/farmers who really wanted to work for the company would put up with this kind of hazing.

But here’s the rub: That same startup did not follow the rest of the Agile plan: They did not give the newly hired highly motivated salespeople the environment and support they needed nor did they trust them to get the job done. Nothing is more frustrating to the motivated than to be treated like slackers and held on a short leash. There was high turnover and poor sales. Management had to result to clever motivational hacks: Contests, awards, and threats to keep sales on track.

But every good manager knows the motivated out performs the unmotivated 10 to 1. The motivated need only a license to kill. For the motivated the journey is the reward.

Human sacrifice, dogs and cats living together… mass hysteria!

A product owner and his developer

What can we learn from Dr. Peter Venkman about the Agile development process? It’s true that Venkman was the less technical member of his team. Clearly Dr. Egon Spengler was the blue-sky researcher while Dr. Ray Stantz was the practical engineer.  But Venkman brought a lot to the table: Charm, business acumen, and the ability to spot an opportunity. In short, Venkman was the product manager.

Egon and Ray focused on arcane reference books and ecoplasmic gadgets. It was Venkman first used their technology to get results in the marketplace—or at least meet girls. Venkman was usually critical of Egon’s and Ray’s “out of the box” ideas but once won over Venkman stood behind his team and rallied them when the chips were down (as when Gozer the Destroyer took the form of the Stay-Puft Marshmallow Man).

Perhaps Ghost Busters is not a textbook example of product management in action but it’s a funny movie and I’ve worked with far less amusing product managers than Bill Murray’s lovable but obnoxious Venkman.

For all his faults Venkman obeyed the 4th principle in the Agile Manifesto:

Business people and developers must work 
together daily throughout the project.

Venkman did not separate himself from his team:

  • He lived in the Ghost Busters HQ (Hook and Ladder 8 ) with Egon and Ray.
  • He took his turn driving the ambulance that served as the Ectomobile (Ecto-1).
  • He even wore the overalls and shouldered a Proton Pack.

Clearly Venkman, Egon, and Ray worked together daily as equals on each project, whether it was ridding a fancy hotel of a noisy poltergeist or cleaning ectoplasm off a hapless victim.

Too often in semi-Agile development processes product management (the business people) function as the boss. They make the decisions. They are the “decider guys” stepping into the problem space for a moment, asking for the summary, and making a quick decision without context or technical expertise (like Walter Peck from Environmental Protection who demands that the Ghost Busters shutdown their “storage facility” and inadvertently releases hundreds of ghosts into Manhattan).

The problem here is that software development is a complex, subtle, and sophisticated problem. It doesn’t sit well with summaries and snap decisions. To really make a good software product decision you have to get into the details. You don’t have to be a programmer but you do have to live with one. Any complex, subtle, sophisticated process benefits tremendously from people who work so closely together they know each other’s strengths and compensate for each other’s weaknesses.

To ensure the cats and dogs are living together I recommend the following four “Make Sures” (as usual based on the bitter pill of experience):

  1. Make sure the product owners (product managers, business people, what ever you call them) sit with the engineers. It’s good if they eat lunch with them and go out for beers once a week.
  2. Make sure the product owners and engineers understand that they are all pigs. The product owners are not “management” and the engineers are not “labor”. This means product owners must participate in the development process in real time.
  3. Make sure the roles and responsibilities of the product owners and development leads are clear. PO’s own the backlog and priority. DLs own the sprint backlog and technical strategy.
  4. Make sure the team makes all decisions though rigorous debate. There is always a business trade off and a technical trade off that has to be resolved—If not you’re not innovating. Software development is not a democracy and everyone does not have to agree. But the best path, not the most obvious one, has be discovered.

Improving Your Whuffie with Agile

Whuffie Burn Up Story Point Burn Down Chart

Virtual currency like Cory Doctorow’s concept of Whuffie will probably replace real money in the next 100 years. Maybe sooner with the way our current economic crisis is going. Once all the hard currency in the world is spent it’s Gresham’s law FTW!

You can get a head start by using Agile development principles to min-max your Whuffie earning potential.

Cory shows us how in his seminal book Down and Out in the Magic Kingdom. (Released under a creative commons license.)

I’m not sure if Cory meant to illustrate the principles or Agile development or he just stumbled into them. Since principles are laws of nature that exist independently of people I like to think that Cory was just observing how collaborative projects tend to take longer than expected or turn into epic failures. Some truths are self-evident as the founding fathers liked to say.

In Down and Out the protagonists are trying to quickly refurbish Disney’s Haunted Mansion with cool new robotic effects. They ask the head Imagineer (Still the coolest job title in the world!) how long it will take:

“Okay, so tell me, if we came to you with this plan and asked you to pull together a production schedule — one that didn’t have any review, just take the idea and run with it — and then pull it off, how long would it take you to execute it?”

Lil smiled primly. She’d dealt with Imagineering before.

“About five years,” he said, almost instantly.

Our heroes don’t have five years. A rival team is updating the Hall of Presidents with leet virtual reality spam in just a matter of weeks. So our plucky protagonists demand an explanation:

“Five years?” I squawked. “Why five years? Debra’s people overhauled the Hall in a month!”

“Oh, wait,” he said. “No review at all?”

“No review. Just come up with the best way you can to do this, and do it. And we can provide you with unlimited, skilled labor, three shifts around the clock.”

He rolled his eyes back and ticked off days on his fingers while muttering under his breath. He was a tall, thin man with a shock of curly dark hair that he smoothed unconsciously with surprisingly stubby fingers while he thought.

“About eight weeks,” he said. “Barring accidents, assuming off-the-shelf parts, unlimited labor, capable management, material availability. . .” He trailed off again, and his short fingers waggled as he pulled up a HUD and started making a list.

Five years? Eight weeks? Our heroes are perplexed. It seems outside the realm of sanity that a project could take either Five years or eight weeks. (But not if you think about Windows Vista or Apple’s System 7.)

“Wait,” Lil said, alarmed. “How do you get from five years to eight weeks?”

Now it was my turn to smirk. I’d seen how Imagineering worked when they were on their own, building prototypes and conceptual mockups — I knew that the real bottleneck was the constant review and revisions, the ever-fluctuating groupmind consensus of the ad-hoc that commissioned their work.

Suneep looked sheepish. “Well, if all I have to do is satisfy myself that my plans are good and my buildings won’t fall down, I can make it happen very fast. Of course, my plans aren’t perfect. Sometimes, I’ll be halfway through a project when someone suggests a new flourish or approach that makes the whole thing immeasurably better. Then it’s back to the drawing board. . . So I stay at the drawing board for a long time at the start, get feedback from other Imagineers, from the ad-hocs, from focus groups and the Net. Then we do reviews at every stage of construction, check to see if anyone has had a great idea we haven’t thought of and incorporate it, sometimes rolling back the work.

It’s interruptions that add months and even years to a schedule. Corry is not making this stuff up.

An Agile process like SCRUM recognizes this fact and uses the concept of the sprint to create a bubble of focus in an ocean of distractions. During a sprint nothing should change—if something does change (new business reality, a framework that doesn’t work as advertised) the sprint is broken. Sprints are kept short so that change can be accommodated in efficient chunks. Corry is simply pointing out why sprints lead to faster development cycles than non-Agile processes.

A few things are interesting to note:

  • The 12 principles of the Agile Manifesto don’t talk about sprints. The principle of time-boxing a problem seems to be missing. SCRUM and other Agile methods inspired by automotive process management married the idea of sprints or locked iterations to Agile.
  • In Corry’s book the Imagineers fall weeks behind schedule because they can’t handle their newfound freedom from stakeholder feedback. I’m pretty sure it’s because our heroes are too busy struggling through their own character driven plot to act as product owners and keep the Imagineers focused.
  • There is a big difference between stakeholder feedback (changing requirements) and product owner feedback (understanding requirements). SCRUM groks this. The Agile manifesto hints at it.

Using an Agile/SCRUM process to develop your projects enables you to deliver on time while incorporating changes. The net-effect is a dramatic increase in your Whuffie. You might as well start banking it now 🙂

The Shorter Timescale

Eternity (Sprints Not Drawn To Scale)

I’m reading a very scary book right now: Heidegger and a Hippo Walk Through Those Pearly Gates. It’s a funny and informative look at how philosophers and religious thinkers deal with death. I don’t want to be a spoiler but the basic message of the book is that most people live in denial of their own mortality and most philosophers are trying to wake them up. Them being us.

This is especially true in software development. Any experienced engineer looking at a 6-12 month project plan will tell you there is a lot of denial going on. I don’t want to sound overly melodramatic but the 3rd Principle from the Agile Manifesto

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

…is all about waking up and getting out of denial.

I covered the idea of working software in a previous blog post but “preference to the shorter timescale” idea deserves some focused attention all on it’s own.

If you live life to a Tim McGraw sound track then you don’t waste a lot of time on planning or doing things that are simply not going to happen. The idea of a preference for shorter timescales come from the principle that risk grows with distance into the future.

This principle doesn’t mean you shouldn’t dream big! I’ve met many engineers who tell me “Agile is only for minor tweaks, not big initiatives.” (No I won’t name names but I dare you guys to a blog war!) Your backlog should be as big as forever. It’s just that you plan and sprint for what you can achieve as a human mortal. Zeus and Hera can execute a 12 month plan with a waterfall process–Agile is not for immortals.

I like 2 week sprints (with one week of planning in between) for three reasons:

  1. It’s been my experience that (most) human mortals only have vague capacities for planning and managing large chunks of future time. Today, tomorrow, Next week are  cool. We live in a bubble of now that trails 1-2 weeks behind and runs 1-2 weeks ahead.
  2. Engineers (and artists) are mostly monochrons. (Almost everyone else tend to be polychrons.) It’s part of the self selection process that leads people down the road of life to end up either alone in front of a computer (or piano or canvas) or in a group running around the planet making noise. Maybe it’s all due to Asperger’s Syndrome but the creative process seems to require that we focus on doing one thing at a time for a fixed period of time in a repeatable cycle.
  3. Engineers (and artists) get lost in their work (the music of the spheres as it were) and lose track of time. Short sprints mean they can’t go too far astray before they check in with home base! Once an brilliant engineer told me it took a month to fix an ad server bug because he first had to rewrite Math.c. Good thing I checked in when I did or he might have rewritten all of UNIX!

So if we end up working together I will ask you to think in short increments that give us plenty of time to capitalize on the future 🙂

True Working-ness

A Bird in the Hand

Last week I presented to my team at Lime Wire Agile and Scrum in all it’s most excellent glory. Jason Herskowitz, Lime Wire’s VP of Product Management, and I ran though all the major Agile elements and vocabulary words. The best part of the presentation was questions asked by developers, testers, system admins, product managers, and even a couple of chickens. Many of these guys and gals have been the victims of bad Agile programs. It seems easy to play the Agile game but it isn’t. The most critical path to successful Agile isn’t to follow all the rules–it’s to follow all the principles.

The Agile Manifesto’s 3rd principle, Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale, seems to be often ignored. You can not really ignore a principle. A true principle is a law of nature, like gravity or the axioms of geometry. If you ignore the principle of gravity you’re going to get hurt. If you ignore the principles of Agile your project is going to fail.

I don’t know why the delivery of working software is often left out of the plan. Perhaps it’s that to recognize the truth of this principle requires more than just a thin patina of Agile paint on a waterfall process. Most Bagile (Bad Agile) processes do not understand that a milestone without working software delivered is a milestone not met. In Bagile working software is mutated into “ready for QA” or “all P1 bugs closed” or “works on my machine”.

I’m not taking an extreme view of working software! It’s just that “done” should mean “nothing more to do!” I’ve lived through the bitter experience of discovering that done doesn’t mean working software. Nothing is more discouraging than having to wait weeks for a so-called completed project to be ready for release.

If you think about it, delivering working software in this complex world of clouds, grids, restful APIs, and P2P networks, is daunting. It may even feel wasteful to go though all the effort to get the components integrated and tested every couple of weeks.

But true working-ness means you can get feedback before every iteration from actual flesh and blood users. Feedback like this is invaluable to avoid wasted effort.

True working-ness means you can focus on the future and get rational priorities for your backlog with each sprint.

True working-ness means if you want to go on vacation, you leave things in a good state and others can pick it up while you are gone.

True working-ness means true achievement. It’s pretty hard to make lasting, solid achievements in general, but delivering working software frequently means you have made a concrete difference in the life of your users today.

There is a 14th century English saying that taught me the same the principle as a kid: “A bird in the hand is worth two in the bush.”

The Name of the Game is Change


The second principle from the Agile Manifesto is “Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.”

I call this the Product Manager’s dream principle: “I get to change the requirements, even after development has started! Woot!”

First let’s break this principle down and see what it really means and then let’s talk about how to deal with it.

There are 4 ideas embedded in the PM’s dream:

  1. Welcoming changing requirements in general
  2. Welcoming changing requirements even late in development
  3. Harnessing change
  4. Customer’s competitive advantage

These ideas come from the grim realization that requirements are going to change, at any stage of the development process, and there isn’t a damn thing you or any process can do about it. We can pretend that we have control over change. We can ask business owners and product managers to sign an iron-clad contract but Mother Nature and Murphy’s Law are not going to stop working on our plans.

Given that change is inevitable we can either hate it or love it or just deal. If this principle was written from the hating change POV it would read “Abhor changing requirements, especially late in development…” On the other hand if the authors of the Agile Manifesto loved change the principle would have read “Encourage changing requirements, the later the better…”

“Welcoming changing requirements” is a positive way to deal with a problematic issue in the same way that many cultures have a tradition of adopting strangers. It’s a smart coping mechanism and recognition that shit happens.

Why do Agilists welcome change and not run the other way and lock the door? Because they are confident that Agile process and policies are setup to withstand change. They have accounted for change in the Agile process and thus change is not going to do much damage to the overall big picture.

Agilists also recognize, according to this principle, that they can “harness change”. Agilists can guide change instead of being driven by it. That’s a pretty tall order but with the right process, one that assumes the inevitable, Agilists take advantage of the opportunities that change creates. Change is where true competitive advantage comes from! Not from a carefully designed, well executed plan, but from a process that gives customers the chance to quickly capitalize on changing market conditions. Wiggle room beats precision any day of the week.

Stephen Jay Gould once wrote about how his interest in fossilized snails, from which he would derive many insights about evolution, came about completely by accident. Gould spent his life discovering and teaching how random events lead to advantages that life is able to exploit. That’s agility right there.

Working with Niles Eldredge, Gould popularized the theory that evolution works in cataclysms called punctuated equilibrium: Short periods of sudden change separated by long periods of relative calm. Agilists see a lot of punctuated equilibrium in software development. Everything in the project is going along as planned and then WAMMO! The market changes, a bug appears, or a forgotten requirement emerges! (This is in well run organizations. In poorly run organizations the change is constant and internally generated.)

OK, now that I have accepted change how do I deal with it? Each school of Agile development from XP to SCRUM has different foci for dealing with change. I’ve seen five patterns emerge that successfully welcome change (but I’d love to hear about others):

  1. Time boxing: Limit the amount of time you spend on any one task so that the cost of change isn’t devastatingly high.
  2. Locked iterations: Restrict change to outside development cycles. Once a dev iteration is started ignore change until victory can be declared. If you really must change requirements in the middle of an iteration stop and re-plan.
  3. JIT planning: Plan as little as possible and as late into the project as possible.
  4. Light design and documentation: Waste as little time as possible on design and documentation as it’s all going to change anyway.
  5. Measure and analyze: Don’t act on assumptions or intuition. Do what the data tells you to avoid surprises.

These solutions in isolation create as many problems as they solve. But when used together in a well thought out and integrated process they create a flexible fabric that can capture the energy of change and use it to gain an advantage. This is hard to do, not because it’s complex, but because many engineers (and normal people too) are uncomfortable with change. One strategy that helps people surf the chaos is to keep a central vision in place with a general roadmap of how to get there. As long as the vision is stable then change is the process of getting there.