The history and future of management
Many organisations today are run according to management principles which are 50 to 100 years old. These methods are not appropriate to the fluid and fast-paced world we live in, but many organisations are struggling to change. In this article, I will talk about the history and future of management theory, so we can understand where we are, how we got here and where we can go.
First wave: Scientific Management – 1910s to 1950s
The first major attempt at formally defining management was by Frederick Winslow Taylor. He came from a manufacturing perspective and focused on efficiency. Taylor was trying to solve the problem of how you can define and improve the instructions for workers so they create the greatest output for a given amount of time. He published a landmark book in 1911 called The Principles of Scientific Management.
While Taylor’s methods were revolutionary and hugely successful at the time, there are some important assumptions we need to consider. Taylor was operating from a mindset where:
- The senior management of the organisation knew exactly what they wanted from their workers
- The workers knew and cared nothing about the organisation, what it was doing or why
- The workers didn’t really want to be part of the organisation or do the work, so they had to be bribed or coerced to work
- The criteria of success of management was simple quantitative output: how much stuff was produced per unit of input (cost, time, worker)
- The responsibility for measuring and improving the entire process was that of management.
Command and control
At an organisational level, the main model was command-and-control with bureaucratic structures.
- Bureaucratic structure means a formal hierarchy of roles and rules: a chief at the top with people reporting to them, and people reporting to them, all following a pre-defined set of policies
- Command-and-control means that power is concentrated at the higher levels of the bureaucratic structure. People lower down in the pyramid don’t just report to higher level people from a line management perspective, but they are also dis-empowered. They receive instructions from a higher level person, who receives their instructions from a higher level person, and so on. This is the traditional model of a modern military.
So this was pretty effective for the time. Businesses were mainly engaging in big industrial and manufacturing processes. They (and workers) were mainly turning stuff (raw materials, commodities) into more complex refined stuff. Turning ore into steel, steel into beams, or beams into buildings, cars, and so on. There was not a whole lot of what we would call “knowledge work” or service industries.
The world started changing
Throughout the 20th century, two important changes happened:
- Industrial manufacturing started becoming more and more automated, and industry needed less manual workers and more people able to do things like design and build complex machinery or operate complex machinery
- A large service industry popped up: accountants, lawyers, advertisers, journalists, marketers, travel agents, management consultants. And of course, computer programmers.
People started realising that the Scientific Management model had some big problems and didn’t work so well in this new economy.
Second Wave: Theory Y – 1960s to 1990s
Starting in the 1960s we saw what I would call the “second wave” of management theory, which we can call “Theory Y”. This came from a guy at MIT in the 1960s, Douglas McGregor.
McGregor basically said we’ve been looking at people and organisations in the wrong way. People aren’t dumb automatons to be bossed around, threatened and harassed; they are beautiful precious souls who need to be nurtured and grown.
The traditional Scientific Management approach he called “Theory X”, which is fundamentally characterised by the view that employees don’t want to work and don’t care for the organisation they work for. He proposed Theory Y: people DO want to work and people DO support the organisation they work for, IF we treat them with respect and kindness. It is the job of management to build relationships with employees, and develop their abilities and knowledge.
Some things changed
This might sound like common sense but it was quite a stretch at the time and took a little while to catch on. But it eventually did.
You now had big companies and government departments with these new-fangled “human resources” departments, and “development programs” and “all-hands sessions” and the like. Employees were now all special beautiful snowflakes and it was decided we had to be nice to them and nurture them and train them and develop them and give them hugs if they felt bad.
In the 1980s, there was a new focus on “corporate culture”, especially after the publication of Peters and Waterman’s “In Search of Excellence“. I don’t think they actually understood how culture worked, but that’s another story. Due to the increasing pace of technological change, people now started talking about innovation and disruption, though few companies understood what it really meant.
Some things didn’t change
So some things changed, but some things did not. At an organisational structure level, things were pretty much the same. The big organisations were now talking about HR and looking after people and developing them, but it was still command and control.
You still had pyramid structures with a CEO at the top giving orders to senior executives giving orders to middle managers giving orders to worker bees. People were still cajoled into work, though more with the promises of bonuses rather than the threat of sacking. Power (formal and informal) was still concentrated in a few small places. In fact, it was arguably becoming even more concentrated in a new group called the PMO (Project Management Office). Project Management was becoming a powerful discipline with a body of knowledge and enormous influence.
The project management \methodology (which came from civil engineering) formed the basis of what we now call the Waterfall method of software development. Organisations continued to assume that they knew what the customers wanted, they would define and scope and specify software (or any products) up front, and it was the customer’s responsibility to rush out and buy the wonderful things that the big companies had decided to build.
The problems with Theory Y / Second Wave
There are of course huge problems with this:
- Command and control is an inefficient way of distributing decision making, especially under conditions of rapid change and uncertainty, i.e. the late 20th century
- Empowered workers are happier and more productive (decades of research confirms this)
- Financial bonuses are a terrible incentive for knowledge-intensive work (decades of research confirms this)
- The traditional Project Management methodology is great for big slow slabs of work where we know up-front exactly what we want to build and how and why we want to build it, but horrible for building new innovative things with a lot of uncertainty around its value proposition, i.e. the late 20th century
- The traditional PMO ways of defining and funding work (by projects, rolling up into programs, rolling up into portfolios) is classic command-and-control thinking, unsuited to the new changing world of business.
So in the 1990s, things slowly started changing.
The Third Wave of Management – 1990s to 2010s
In the 1990s we saw the third wave of management theory emerge, although the origins actually started a little earlier.
It really started with Lean Manufacturing
W. Edwards Deming, whose ideas influenced Lean Manufacturing
The seeds for it were actually sown in Japan in the 1960s, with the birth of Lean Manufacturing. Well actually, the seeds were actually planted a little earlier by an American named D. Edwards Deming, who wrote a few books, but a particularly influential one called Out of the Crisis. Deming went to Japan because nobody in America would tolerate his heretical views. Deming’s background and focus were manufacturing and quality control, but many of his views concerned businesses, organizations and management in general.
Deming put forward views that contracted most of the major tenets of management, especially around manufacturing:
- the most efficient way to produce things is in very small batches, not very large ones (flexibility beats economies of scale)
- the best amount of inventory to have is none at all, rather than a lot (inventory is a waste)
- a focus on quality increases efficiency, not decreases it, because defects are a waste and slow down the process
- businesses should focus on optimizing the whole system, not small parts of the system
- the responsibility of improving the work is on the workers, not management.
A Japanese guy called Taiichi Ohno and some of his mates formed a system of management called the Toyota Production System, which became known as Lean Manufacturing. And Toyota started beating the pants off of everybody in the automotive industry.
Then came Agile Software Development…
In the 1990s, people started coming up with different ways of building software that rejected the traditional Waterfall model, which was based on the classic PMO / PMBOK way. The way of big up-front design, fixed scope, and big efforts to lock down and control time and cost.
Agile software development said instead:
- let’s work in small batch size instead of large batch sizes
- instead of fixing the scope, let’s be open to change, and learn as we go and discover value
- let’s build quality in from the beginning, rather than frantically test at the end
- and let’s empower our teams instead of dictating what they build and how they must build it.
These are from the Agile Manifesto in 2001. Some of these ideas are similar to those in Lean, some are not. I have written about the differences between Agile and Lean here if you want to go into more detail.
… And also Systems Thinking
The 1990s also saw the publication of an influential book “The Fifth Discipline” by an MIT researcher named Peter Senge. This book drew heavily on an obscure practice called Systems Thinking, which developed in the 1950s from earlier works in cybernetics. This book applied Systems Thinking to organisational theory and knowledge management. It was a powerful and brilliant book and revived Systems Thinking into mainstream management theory. It went on to spawn a series of critics, imitators and disciples. Systems Thinking has been influential on some of the later management theorists, especially Jurgen Appelo and David Snowden.
Where are we now?
Lean and Agile have made big strides. Many organisations are realising the limits of theory Y. Some of the smarter ones are moving towards more of the “Servant Leadership” model championed by people in the Agile movement: the boss is actually there to help the worker, not the other way around. This has been going on in Japan as part of Lean Manufacturing (to a certain extent) since the 60s. The worker gets to define (and improve) their work, the leader helps the workers by giving them all the tools and support they need to get the work done.
At a structural level, it is still a struggle and still not much has changed. That is because traditional organisation structures and bureaucratic models are so rigid and hard to change. That is why Lean hasn’t really adopted Servant Leadership properly – the Japanese manufacturers still have giant rigid bureaucratic structures with set roles and hierarchies. But there are some alternative models emerging, which you could consider the fourth wave.
Lots of organizations are of course trying to adopt Lean or Agile or both but failing miserably. I have already written about the main reason these attempts at Agile fail, and why they end up really just being an Agile cargo cult.
The Fourth Wave? – 2010s and beyond
There are some more radical systems of management and organisation emerging that you could see as a fourth wave.
XSCALE
XSCALE is a open agile ecosystem (rather than a “framework”), modelled after the Linux open-source model. It promotes ideas of “self-organising teams in self-managing streams” and “leadership-as-a-service“, borrowing ideas from the Iriquois “Great Law of Peace”. This is a flexible and powerful model that replaces traditional rigid authority-based decision making with a model based on consensus and influence. A “Leader” is someone capable of making a decision only if a “Coach” has decided that the team or squad is unable to come to its own decision.
The ideas here are influenced by ideas of Servant Leadership and Arthur Koestler‘s notion of “holarchy” (not to be confused with Holacracy, see below). I would recommend reading more about XSCALE’s original and powerful ideas at its website or wiki.
Radical Management
This isn’t as radical as it sounds, though it’s pretty good. An influential thinker named Steven Denning wrote a book called “The Leader’s Guide to Radical Management” in 2010. I recommend you read it if you haven’t. It is basically Denning’s attempt to apply the values and principles of Agile and Lean into the management patterns and structures of traditional companies, especially ones that aren’t building software. This is an interesting and difficult task and Denning is successful at defining how this would work.
The Spotify Model
Another not-so-radical movement is the Spotify model, which is a new version of the matrix organizational structure. Instead of line management and project managers, you have Tribes and Guilds. Tribes are cross-functional teams that focus on a particular work-stream / value stream. Guilds are a gathering of people with a similar role and basically centres of excellence in a particular practice. So you might have a “mobile” and a “web” tribe and a “UX” and a “developer” guild. This video by former Spotify Agile consultant Henrik Kniberg explains how this model works.
Holacracy
Holacracy is a much more radical model that attempts to pretty much get rid of managers altogether. Instead of a traditional hierarchy of positions, you have “circles” of influence, which consist of “roles”. And roles do not necessarily correspond one-to-one with people. A person may have one particular role with one circle, and another role in a different circle. And people can move around roles easily (since they are not changing their actual position or job title).
There are specific rules around governance and decision-making, defined in a lightweight “constitution”, and that’s about it. Online retailer Zappos is the poster child for Holacracy, but the jury is still out on whether it works.
Management 3.0
A Dutch writer called Jurgen Appelo published an important book in 2011 that ties together some of these “fourth wave” ideas into a system he calls “Management 3.0“. It fits snugly in the lineage of traditional Agile software development but draws heavily on Systems Thinking and Complexity Theory. The core tenets of the system are:
- Energize people
- Empower teams
- Align constraints
- Develop competence
- Grow structure
- Improve everything.
I’d definitely recommend this book if you want to get an overview of the latest thinking in management.
Cynefin
This is not really a management “system” or “theory”, but a framework for decision making. (It’s a Welsh word and you pronounce it “Kuh-Neev-In” by the way). It was invented by a management researcher and theorist named Dave Snowden. Cynefin is a framework to help people understand the decisions they make and the level of complexity surrounding those decisions. It doesn’t sound very exciting but it is pretty cool. I don’t see it as a full system of management, but rather a tool that people can use when required. Like Management 3.0, it draws mainly on Systems Thinking and Complexity Theory.
Teal Organizations
One of the exciting new forces in the fourth wave of management thinking is Frederick Laloux. He published a book in 2014 called Reinventing Organizations. This book is his attempt to describe a new type of organization that reflects a new stage of human consciousness: Teal Organizations. Laloux does a really good job of describing previous organizational patterns and mapping them to specific stages of history and consciousness. The book can occasionally slip into weird new-age pseudo-mysticism. But it has some really interesting case studies of organizations that have adopted radical models of management.
I hope you’ve enjoyed this tour through the history and future of management theory!
