The spirit of Agile - what's the big idea?

I am writing this article as a kind of conceptual, multi disciplinary, cross industry exploration of “agility” in its largest and most encompassing sense. I do this because I think this kind of understanding is important, maybe even essential.

Contents:

  1. The three big problems

  2. Priority inverting ideas and the cost of not getting it

  3. Inventory: the problem with wanting to have more stuff rather then less stuff

  4. The three parents of Agile and what is Full Agile?

  5. Decentralization: how far can it go? How far should it go?

  6. Direct and indirect control

  7. Control spikes

  8. The wetware

  9. The good and bad of process: question primed processes and exclusion processes

  10. Horses and electric cars

  11. Agile adoption: all at once?

Just to set the record straight, I am not an agile zealot, primarily because I dislike the incremental thinking it usually leads to, and I also dislike being a zealot in general, about anything, so I’m not here to tell you that agile is the best thing since sliced bread. However, I do think some kind of incremental lean-ish approach is probably the best idea for most projects and I can also say that to be a complete manager in this day and age you need an understanding of agile, and if you’re the curious kind, with an appreciation for the conceptual, then this article is definitely for you.

Before we go on, two things:

  1. I will link to a million things and many of those links may be books or very large articles. You don’t have to read them all as you read this, but I do it so I can sort of build a comprehensive library of all the key ideas in the agile space; you can use it as such

  2. Forget process, any process. Scrum, Kanban, SAFe, whatever. The whole idea of this is to go beyond a sequence or another of steps and travel deeper to some more fundamental ideas, to the point where you will be able to see any process as just another tool, clay in your hands

I also promise I won’t mention Taylor other than this once, because namedropping this dude is a cliche these days, but you should also read about scientific management to understand that important part of history as well, because a lot of what we’re doing in management today is both a continuation and a counter reaction to it.

The three big problems

The first way I like to look at agile is through 3 problems it’s trying to solve:

  1. Customer

  2. Complexity

  3. People

By fixing the customer problem, I mean fixing the problem of building stuff nobody wants. Agile wants to help us make sure we work on the things someone wants to pay for, not the things we think that someone may want to pay for (yes, there’s an issue with radical innovation here, hold that thought for now). That’s the customer problem, building the right thing at the right time for someone that wants it.

The complexity problem is the over-planning illusion, the ludic fallacy of thinking that you can plan complex work for months and years in advance in detail and then just execute the plan with little change. Agile says this is impossible so instead of getting stuck in all this planning (which is not going to be “true”, accurate, anyway), plan a little bit and then do a little bit, and as you do a little bit you’ll learn a bit more, and then you plan your next step, and so on, and so on. This is not to say you can’t or shouldn’t have a direction or an end goal in mind, but what you don’t have to do, and you probably couldn’t even if you tried, is to map every step towards it. Just start walking. There’s a note to be made here in as far as the cost of rework, as in some cases rework is prohibitive so more planning is required. For example, if you go out into the desert without enough water, the cost of rework (i.e. going back for more) might be fatal. Typically, most projects in business have much lower rework costs, and much more planning complexity than calculating the required quantity of water. In other words, the cost of over-planning, with all its unwanted consequences, will usually easily exceed to the cost of extra rework due to less planning, especially if extra planning is a game of extreme diminishing returns and some kind of iterative method is followed. Viewed from another angle, you should find the optimal quantity of planning where the sum of the cost of planning + the cost of rework is least. You can’t usually calculate this point upfront, so the typical way to proceed is by starting with minimal planning for a first short iteration and seeing if it’s enough, too little, or too much, and adjust, try again, adjust again. “Minimal planning” will vary with industry, nature of project and others, but remember, it’s easy to overdo-it, so start smaller than your first instinct tells you.

The people problem is about the fact that agile is pretty unique among the “production methodologies” in treating humanity as a first class citizen, with all its strengths and weaknesses. Individual and group psychology is taken into account at every step, from how decisions are made, roles defined, planning done, work (self) assigned etc. People are not considered robots that can just do whatever once you train them to do it, but as the emotional and (charmingly or infuriatingly) imperfect creatures that we are.

These are the three fundamental problems agile is trying to solve.

Priority inverting ideas and the cost of not getting it

This is not specific to agile, this is specific to any big fundamental change of how we do things, but there’s a lot of this happening with agile adoption onto waterfall thinking and when the priority inversion doesn’t happen, because the new ideas isn’t internalized, that’s where the agile adoption is shallow, it’s not real, it’s fake. The form is copied but the reasoning behind it is not understood. The motions happen, but the mind is stuck in the past. Take an example: the idea that work should be split in smallish, testable, business focused chunks is a fundamental idea for many fundamental reasons, but if you don’t understand why, you take it lightly, not seriously. You try to split your work like this, but you can’t, because your teams are specialized in a certain way and you have architectural constraints and you end up with large, intertwined, poorly testable chunks of work that you now call user stories and you put them in a list you call a backlog, and it doesn’t seem like a big deal to you that you haven’t structured the work items quite right, but it is, because without that you haven’t really done anything beyond confusing your people with new terminology. Your priorities haven’t changed and you keep the old as it was, just painting a bit of new on it. On the other hand, if you understand how fundamental this new idea is, if you understand that you have to do it no matter what, then this is your main priority now and everything else becomes secondary and needs to change if it comes in the way of this. If the teams have to change, the teams will change. If the architecture has to change, the architecture will change. Nothing is sacred, because your priorities have been inverted and you are ready to throw out the old and make room for the new, because you’ve internalized the new idea. This is where many agile transformations fail, because people don’t really want to change anything in the existing structure (be it people or tech) because it’s hard and the larger the company the harder it is, because bureaucracies tend to be that way. Instead, they adopt the new terminology, change a few things here and there and call it done, but it’s not. Their priorities haven’t been inverted yet. Their paradigm hasn’t shifted.

Inventory: the problem with wanting to have more stuff rather then less stuff

Call it a lack of zen, an un-appreciation for minimalism, a hoarder’s craving, or just plain old biological instinct of eating as much as you can when you can cause you don’t know when your next meal is going to be, the fact is that, left alone to our instincts, a lot of people, and therefore a lot of organizations, intuitively prefer to have more stuff rather then less stuff. “Rather have it and not need it than need it and not have it” kind of approach. While there are situations when you need some of this thinking, the counterintuitive insight of lean is that, more often then not, much more often than not, having stuff you don’t need, or having more stuff than you need, or even having better stuff than you need, is more of a burden then an asset.

green-giant-mushroom-400x400.jpg

The classic problem is inventory. Take the simple example of a canned mushrooms factory that produces one thing and one thing only: cans of brined, salty, watery mushrooms. Inventory is when you have cans of mushrooms already made, sitting around, but not yet sold and delivered to a client. At first glance, some inventory may seem like a good idea, for example to have something on hand for unexpected orders. It also makes management simpler, as you don’t have to produce just enough cans, just in time for a new order, you just do a whole bunch at a time and you stack them up in a storage room. Simple, apparently smart, but wrong. Inventory is stuff you’ve paid for but you haven’t yet been paid for, so it’s a (hopefully temporary) loss. You may try to sound smart and call it an investment, but that’s an overused term. If you misjudge something and you don’t sell it, it becomes a permanent loss and even if that doesn’t happen, while you have it, you spend money to store it. If you buy a batch of bad mushrooms that spoil faster than usual, or if you make any kind of manufacturing mistake, you may only figure that out after you’ve canned a whole bunch of them, because you work in big batches. If it’s the kind of mistake that only the client can see (imagine your product to be more complex than a can of mushrooms), then tough luck, by the time your product has reach the client, you probably have half a warehouse full of more product with the same defect, which you’ve already built and which you can’t fix or have to spend even more money to fix. Inventory doesn’t just take space and money, it also takes time and mental effort. You have think about it, you have to manage it, you have to develop process for it. Worst of all however, this kind of “let’s build a metric shit ton of it so we don’t have to worry about it” mentality encourages lazy, stupid, rigid management.

But Andrei, but Andrei, there’s no way I can make just enough cans for every order I get, just when I get the order, not sooner, and manage and deliver all that in a reasonable time. It’s just too difficult, too complicated.

It is complicated, but it seems more complicated than it is because you haven’t accepted the idea of no inventory as a priority inverting idea. Once you do that, and you accept it as a cornerstone, you’ll reorganize your entire business around it: you’ll ask your suppliers to deliver differently, and if they can’t, you’ll change them. You’ll change your production processes, your methodologies, your sales approach, your job tiles and roles and maybe even some of your people, if they can’t be retrained. This is lean manufacturing and this kind of thinking is just in time just enough (do what you have an order for, when you have an order for it, not more, not earlier) and pull not push (don’t push production to meet internal targets, let the market, the client pull demand from you).

But Andrei, but Andrei, canned mushrooms is easy. My business is way more complicated.

Well, congratulations, and even more so you need to pay attention to inventory. When the developer abstracts a class before the abstraction is really needed, that’s inventory: stuff we built because it may add value later on, or it may not, but we’ve already spent to build it and we’ll keep spending to maintain it. When you think too much about something, you’re building an inventory of analysis effort, you’re spending and storing analysis you don’t need, and you’re going to carry it with you. When you come up with a business process that is earlier or more complex than it absolutely needs to be, that is there “just in case”, you’re again building an inventory, this time one of process. Process inventory is among the worst because you not only spend to develop it and to maintain it, but you actually have to follow it and there’s few things worse than having to follow unnecessary process.

Anytime you’re about to do something, be it actually building stuff, analyzing something, organizing stuff, documenting something, think really hard: do I really need this now, and is this a proportional response to the problem, am I doing the minimum required to move forward or am I over-engineering?

This is the lean way and the biggest impediment in thinking and working like this is our human weakness and desire to surround ourself with all kinds of crap, too much of anything, cause we have fragile egos and our mothers didn’t love us or whatever. We fail as managers in the same way we fail as people. Better management is frequently about more introspection and better self management.

Lean

I won’t fully explain Lean here, there’s too much to say and too much read about Lean, but I’ll quick describe it and place it in the other overall puzzle. Lean is about all these things, like I was just talking about minimizing inventories, about doing just enough, about building incrementally and reducing waste at every step. Iterative, minimalistic, empirically informed movement. Lean in its modern form came out of the Toyota Production System and its 7 Wastes, out of which overproduction (doing more or better than needed) is the worst of them all.

Lean is a really big deal, because it’s important in itself, and because it’s one of the three parents of agile, and maybe, just maybe, the most important one. A lot of what agile is trying to do, a lot of what all kinds of agile processes such as Scrum are trying to do is no more and no less than to implement lean principles in the specifics of that process. The backlog is just a way of documenting a value chain, and it forces us to not build stuff before we need to. Sprints which end in potentially shippable stuff that the client takes a look at are a way of forcing us not to build inventories. Agile wouldn’t exist without Lean and a lot of Agile is directly based on Lean ides.

The three parents of Agile and what is Full Agile?

Agility, in it’s biggest sense (like everything agility, business agility, not just software dev agility) has three parents:

  1. Lean, which I’ve mentioned already, from where it took the basic production ideas, from where agile learned how to build stuff

  2. The people aspect, rooted in the cultural revolution of the 60s and its after effects, later on also informed by psychology and sociology, resulting in a workforce that wanted and expected a different kind of work environment than what it had before

  3. Decentralization of the company structure and decision making process, as a reaction to complexity that can’t be tackled centrally, prescriptively, deterministically, but only adaptively, organically, through loosely connected ever evolving organizations. Decentralization also has the side benefit of pushing decision making down into the front lines, which ties into the previous point, empowering people

These three taken together make up for what I call Full Agile, which is when your entire company is agile: You are lean, you are decentralized and you pay attention to what drives people in this day and age and this (your) kind of context. Some companies are lean, but have an old mindset when it comes to people or are insufficiently decentralized. Some companies are all hippie and fluffy when it comes to the people, but are not sufficiently lean. Few have it all, and you should aim to have them all.

Decentralization: how far can it go? How far should it go?

The idea with decentralization, beyond the fluff, is simple: you have to do it. Companies don’t decentralize because they like it or because it’s cool, they do because the central corporate planning process can’t cope with a fast changing world, so you need to split into loosely connected, more or less autonomous units, small, nimble, empowered to make decisions and to act on it. You do this because you can’t predict what problems you’ll face, what your competition will do, what the customer will want, so instead of trying to centrally plan for that, which you can’t, you create many autonomous units that are designed to be reactive and responsive. You don’t solve strategy through plans anymore, you solve it through structure. Your strategic thinking becomes less about what you will do when, and more about how you will structure yourself so you’ll be able to respond to whatever, whenever.

How far can you go? I’ve referred you to the Team of Teams book above which covers that quite well, but another book even more focused on decentralization, and more wide reaching in this regard, is Reinventing Organizations. Read it to see example after example of real life companies that are decentralized to an extreme, delegating huge decisions down to the lowest levels, some even dropping HR concepts like job titles and job roles and working in a completely new, extremely loose, self organizing way.

What should you decentralize and what not? Decentralize what you can’t (excellently) plan and centralize what you can (usefully) plan. Ideally, do both: decentralize and centralize at the same time. How do you do that? Through big data, or, to avoid buzzwords and not too make it too specific, through a lot of data. Decentralize execution, record everything that happens and then (non exclusively) centralize analysis and prediction. What you have to be careful with is how you record and how you analyze and how you act on that analysis so that you don’t discourage the autonomy and decentralization that made it all possible in the first place. Measurements influence behavior and motivation, so when you measure stuff just for analysis, try to do it cleanly, automatically, don’t make your people do it, let them be free and focused on what matters and don’t restrain their courage by putting (explicit or implicit) KPI’s in their heads, other then those few you actually want to. You want to know how many lines of code and what kind of lines of code they write on an April Monday as opposed to an October Thursday? Fine, do it, but do it automatically, take it out of the way, don’t make them have to do it or even think about it, and in no way don’t make them think you’re going to rank them by it, because then you’re going to ruin your whole people mojo faster than you can say “oops!”.

Also, remember that stuff is always in flux. On any given day, some aspects of your business may be in need of more decentralization and some others in need of more centralization. It depends and it’s changing as we speak. Go with the flow, adapt, change, flex, push, pull.

Direct and indirect control

Traditional organizational control is direct. This is the process, follow it. This the template, fill it. These are the standards, meet them. This is your job description, act like it. I am your boss, I tell what to do. Traditional control is also cascaded through the clusterfuck that is called Management by Objectives and the strong contender for the most overrated HR idea ever, which is SMART Objectives. Twin brothers of management myopia, MBO and SMART are the inheritance we have to thank GM for, which under Sloan invented the modern, massive, uber-hierarchical, super centralized, bureaucracy infused (pre) modern corporation, who everybody then copied and some are still struggling to outgrow even to this day.

One of the most difficult things for a manager to do is to switch from direct control to indirect control, but switch they must in order to become agile. This is another one of those priority inverting ideas.

Indirect control is about creating the context that increases the likelihood for a desired outcome to happen more often.

Indirect control is different because the manager that practices indirect control can’t (fully) answer simple questions such as: what are your people doing right now? when will this exact piece be ready? what will happen tomorrow at mid day? what happened last Thursday? The indirect control manager can’t answer these because she isn’t directing work in this kind of deterministic way. What she can say is: I’m building a context here with the right kind of autonomous entities, trained and primed the right way, motivated and guided such that through their apparently unpredictable actions, statistically speaking, on a medium to long term, we are most likely to be successful in regards to this major objective compared to any other course of action we can realistically conceive. Indirect control coupled with decentralization is many times the best way to increase the overall (or repetitive) likelihood of success, but it can’t guarantee success in any local area or specific time horizon. Neither can direct control in many cases, but it pretends to, it offers the illusion and it also offers all the ass covering bureaucracy for managers to say “I did my part, something else didn’t work”.

True indirect control is not possible without enlightened leadership at the highest levels. Agile is a kind of pretty advanced indirect control and when top management wants to do agile but doesn’t know that it has to give up direct control, or can’t give up direct control, it creates a hobbled version of agile that has all the changes of being shallow, fake and ultimately a failure. SAFe for example tries to solve the problem of combining indirect control at the team level with a kind of statistically relevant dashboard at the higher levels, and to a point it does it, but it’s also dangerous, because while SAFe is an honest attempt, it’s not a sham, it also too easily gives a direct control illusion to top management and therefore avoids the painful but necessary priority inversion from direct to indirect control, and without this inversion, no agile adoption is truly possible.

Control spikes

Sometimes though, control needs to be direct, or, at least, results needs to be guaranteed. Legal compliance, health & safety, extreme urgency. I call these control spikes, as an analogy to the XP Spike, as a deviation from normal operating model, an exception, an emergency. You can establish strict areas of direct control, but keep them out of the way of the decentralized part of your organization as much as possible. You can also start special projects, specific task forces, for specific problems, ran in a different, more direct control way, maybe in a crisys like fashion, to tackle particular problems, where the “seed good intentions now and reap rewards later, probably” approach of indirect control doesn’t work. Indirect control is necessary because it’s required for decentralization, but not everything needs to be decentralized. Startups for example tend to be heavier on direct control than more mature businesses, because a startup needs to survive today, right now, get results tomorrow, it doesn’t have the luxury of seeding many fields and waiting to see what crops succeed best.

The wetware

Wetware = human brain cells or thought processes regarded as analogous to, or in contrast with, computer systems. That is, people.

People are affected by a whole bunch of things you wouldn’t necessarily intend or foresee to affect them, and they are affected in unpredictable ways. The way you measure, talk about, evaluate their work, the way they are organized, tasked, trained, guided, the environment in which they work, everything affects their minds, thoughts, emotions, motivation, performance and ultimately outcome of their work. That’s why complicated people management is the worst. Complex job descriptions, sophisticated KPI’s, numerous objectives, all these are borderline impossible to balance and almost always create confusion and demotivation. When it comes to people, keeping it simple is best.

Alongside simple, also keep it honest and if you also have that (rare but learnable) ability to think big and motivate towards unnecessary improvement, but without sounding like you’re drinking your own kool aid, then you’ve got it covered. The holy grain of development is what I call motivation to unnecessarily improve. Anyone will fix something that hurts right now. Anyone will take steps to remove a pain. Few will put in the hard work now to do something more, better, even when not doing it would, strictly speaking, be enough.

But Andrei, but Andrei, doesn’t this contradict the lean idea of just enough, just in time?

No, it doesn’t. Here we’re talking about people’s attitudes of (self) betterment, of pushing boundaries, which is always great. There we were talking about actually choosing what to work on based on value streams. These two can be balanced, no problem, all it takes is a bit of good management. Sounds the same, but it’s different.

When it comes to motivation, the fashionable thing to say is Dan Pink’s triad of Mastery, Autonomy and Purpose. If you don’t have time for the book, the 10 minute youtube video will do. It’s not bad, but it’s not enough. As a topic of critical importance, I’ve also written an in depth article on motivation: Self motivation, determination and professionalism.

The (relatively little adopted) concept of software craftsmanship is interesting too, through how it views the software developer less like an engineer or a scientist, and more like a master metal worker for example, a (partial) throwback to the medieval days of apprenticeship, communities of practices, guilds, the practicality of working with your hands, the iterative development, the shaping and refining of your creation right there in your hands.

The good and bad of process; question primed processes and exclusion processes

Everything that simplifies stuff is good because it simplifies it. Everything that simplifies stuff is bad because it simplifies it. There is no general solution to this (apparently shallow) zen-ish sounding contradiction. You have to flex, to push and pull, to adapt and to decide what needs to be simplified how much when. Management, leadership, is Ju Jitsu. You grapple with the problem and you see what you can do. Leadership is not a beautiful dance, nor is it ivory tower detachment. Leadership is a contact sport.

Take Scrum as an example. If you understand the movement of the sea underneath it, with all it subtleties, Scrum is a good little boat to sail on. But you wouldn’t believe how many people do Scrum (better or worse, usually worse) and don’t even know what Lean is, and by that I mean the word, not the whole thinking behind it. In this case, Scrum is bad because it gives unprepared people an illusion, the illusion that they are agile. They go sailing without understanding why things float on the water.

Another good view on simplicity and recipes is generational. A bunch of people with profound understanding of something go ahead and codify some steps, some processes, some best practices, which they can use very well, because they also have the profound understanding behind them and they’re not just following a set of steps without understanding why. Time passes, new people come in and have less and less of that profound understanding, instead just learning the codified motions by heart, until finally everything turns to shallow form and comes crashing down. This is one of the reasons why brilliant startups and highly successful companies have such a hard time staying equally great as decades pass and the initial founders/managers retire.

Process, best practice, come to solve the real problem of efficiency (so as to not reinvent the wheel each time) but create the problems of rigidity and organizational stupidity (lack of understanding of the fundamentals, lack of courage and mental breadth to make big decisions). How can the contradiction be solved, how can we have the best of both worlds? We’d ideally like everyone to think deeply and understand the fundamentals, answer anything from the principles up, but at the same time it would also be prohibitively expensive to have everyone go through this kind of long and arduous discovery process for every problem. It is a clear practical necessity to have some kind of documented knowledge that new people can absorb at higher speeds than it would take them to discover it, and get productive quicker. So, how do we design best practices and processes that optimize without dumbing down the organization?

One way is through what I call question primed processes: that is, design process not to give one a prescriptive set of steps to follow (that is, give them an answer, and a rigid one at that), but to ensure that the right questions are asked and that conversations of sufficient quality happen before a decision is made. Take the classic Scrum example, the sprint planning meeting: it doesn’t tell you what to decide, but it tells you that a conversation should happen and it structures that conversation to maximize its quality. Going back to some of those examples of extremely decentralized companies from the Reinventing Organizations book, the decision making process usually boiled down to “before doing something, consult those that will be impacted by it”. It didn’t tell people to do or not do something, or how to do it, but it told them to talk about it before doing it.

Another kind of process is the exclusion process: a kind of process which prohibits further constraining of a certain area through more process not yet developed at this time. A sort of a guarantee of a certain freedom. For example, in aviation, the captain has final authority on all decisions made during a flight and, despite the myriad of other rules and regulations, which are of courses valuable and any good captain would be always attempting to take them into account as much as possible, at the end of the day, regardless of anything else, the captain decides..

Put it another way, question primed processes and exclusion processes are also ways of creating intentional ambiguity, of leaving some open unregulated space where real thinking, debate and ownership can happen. Any process, any culture item, any best practices that create intentional ambiguity is great because it gives you a direction, a head start, but it also makes you think. Business process shouldn’t be there just to give answers, but also to create thinking tension, to generate debate, to put you in the unknown and make you deliberate.

Keep this in mind when recording best practices and designing processes. The traditional way is to make them as explicit and as “easy” as possible: that is, make them such they give simple, prescriptive, repetitive answers such that the person following them doesn’t have to really think about it. Instead, try to create process that guides people in general directions, but also makes them think.

Horses and electric cars

Ford_Model_T_01-grande.jpg

Sometimes, especially on the wave of major technological disruptions, the value chain can become your enemy when it comes to radical innovation. Staying close to your clients and satisfying their needs as they perceive them with their current understanding can, occasionally, keep you stuck in the past and quickly make you obsolete. “If I had asked people what they wanted, they would have said faster horses” Henry Ford is to have said (unconfirmed, but tellingly). Why did it take someone from outside the car industry, far from traditional car development centers, all the way in California, to build the first truly successful electrical car? Was there nobody at BMW smart enough to think about it? Of course there was, but BMW customers back in 2008 didn’t want a 200k EUR electrical car with a range of 300 km and no charging stations to speak of. They genuinely didn’t want it, what they wanted was an incremental improvement on the Series 3. BMW, and others, listened to their customers, and didn’t build the electrical car. Now, everybody wants one, but it took some radical innovation to get there. The actual engineering steps to build the Tesla were lean and incremental in many ways, but the big bet, the value proposition, the big idea, it was radical. This is where agile as a mythology can work, but incrementalism as a thinking fails. Agile understood as some kind of down to earth, step by step grounded in real actual concrete value that we can prove exists today, fails when it comes to disruptive innovation. Read this book to understand it better.

Agile adoption: all at once?

The diffusion of innovation is the understanding that not all people are equally open to new ideas. Keep this in mind when selling to your customers, and keep this in mind when introducing agile into an organization. The biggest decision you have to make is if you want to introduce it all at once, mandatory, like “guys, we’re all doing this now”, or if you want to sort of test it and see who bites first, and start with those early enthusiastic adopters and spread from there. I don’t have one true answer, it depends, but the market adoption perspective is a good one to keep in mind even internally, inside your company. Treat your management initiatives as products and your employees as your customers. The larger the company, the more it behaves like a free(ish) market and the products (ideas, initiatives) the managers come up with are bought (*sincerely* adopted) by the clients (employees, partners), or not. Look at it this way, present your ideas like this, market them like this, and you will likely benefit and be a better leader for it.