Ralph Waldo Emerson reportedly once said “If a man can…make a better mouse trap than his neighbors, though he builds his house in the woods, the world will make a beaten path to his door.”1 Every engineer fervently wishes this were true. It is not.
The success of any interesting product is not just a function of its efficacy, but of a complex social and economic process generated by the interaction of the technology with the market and the sociopolitical establishment. I’m going to break this down using the MP3 player market as an example. You should read my last post, iTunes and the Basis of Competition in the MP3 Player Market, first. If you already have, here is a timeline to refresh your memory.
This post is about innovation from a technology point of view. It talks very little about the human side of it or company strategy, those are the next two posts.
Technological Innovation
We tell stories about technological innovations like this:
- a genius scientist discovers some fundamental insight into how the world works,
- one or a small group of against-all-odds inventors use that newly discovered phenomenon to create a new technology,
- a forward-looking company turns that new technology into a product and commercializes it.
These stories make the best narratives, and the narrative I have chosen is no different.
The fundamental insight was perceptual coding, the discovery that the human auditory system does not “hear” every sound and so some information can be removed from an audio stream without changing what the listener hears. The invention was turning this into a workable audio compression algorithm. The product was embedding this algorithm into a portable media player2, which was then commercialized. Along the way more mundane innovations changed the constituent technologies to make them more useful.
Some thoughts:
This is easier said than done. The process of turning a new scientific insight into a technology—something that can be used—is not always straightforward. It took Karlheinz Brandenburg’s team and its predecessors years to turn the insight of perceptual coding into a usable algorithm. It reportedly took DuPont decades after the discovery of polytetrafluoroethylene to figure out how to get it to stick to frying pans and call it Teflon. I was taught in college that gallium arsenide was to imminently supplant silicon in integrated circuits: this has still not happened a few decades later, mainly because no one has figured out how to do it economically. One of the core uncertainties in technological development is whether you can and how long it will take to turn a discovered phenomenon into a usable technology and then into a commercializable product. Often this is simply unknowable.
Invention is not always science-driven. There is an idea (especially prevalent among scientists) that technology is nothing but applied science. Sometimes it is, and sometimes it’s not. Sometimes phenomena are analyzed by scientists and that analysis allows engineers or others to create technologies. But sometimes the opposite happens: “Through most of history science has arisen from problems posed for intellectual solution by the technician’s more intimate experience of the behavior of matter and mechanisms.”3 Thomas Newcomen, the inventor of the first steam engine, was not formally educated and developed his engine through trial and error. Edison knew almost nothing of electromagnetic theory. The Wright brothers were not scientists. In these cases the explanations for the phenomena followed the development of the technologies rather than preceding them.4
Of course science plays a part. Scientists experimented with electricity before Edison. Scientists had discovered the vacuum before Newcomen. But phenomena are discovered through experimentation, and the inventor’s trial-and-error can be as powerful a process of discovery as the scientist’s formal experimentation. Sometimes a phenomena is discovered, then explained by scientists, and then harnessed in an invention. Sometimes the invention comes first, with the scientific explanation following.
With MP3 there was a bit of both. There was preexisting science to inform the algorithm, but there was also a long period of practical experimentation–tinkering–to make it work. Brandenburg listened to Tom’s Diner hundreds of times, testing tweak after tweak.
Innovation is more than invention. Invention is a matter of degrees. We can say that Brandenburg (with his colleagues) invented the MP3 algorithm. But did SaeHan “invent” the MP3 decoder chip? Was it an “invention”? We usually use the word invention to mean the creation of a technology, not the implementation of one. But technologies seem less invention-y to us the further they are removed from the fundamental phenomenon they instantiate: the MP3 algorithm was an invention, instantiating perceptual coding. SaeHan’s MP3 chip instantiated the MP3 algorithm, but the algorithm wasn’t a fundamental phenomenon so it seems a little less like an invention, and so on. Who deserves credit for “inventing” the MP3 PMP? Who invented the automobile, the personal computer, the pizza?
While we should care about discovery and invention because they are necessary precursors, real impact comes from innovation, not invention. An innovation is an invention that is useful and is put to use. Inventors and discovers get most of the glory, but innovators cause change and make money. Brandenburg is widely credited with the invention of MP3, Fraunhofer made millions from their patents, but it was Apple–who invented neither the algorithm, the device, nor anything else–that changed the industry and made billions.
How much is an idea worth? I do not talk much about intellectual property in my class. Patents and such-like are important to startups because they defend the company from others having interfering patents in the future. But predicating a strategic advantage on being able to defend a patent works for only a specific type of company. Patents can cost millions of dollars to defend so they are only a viable way to prevent competition if you can spend those millions of dollars once your patents are infringed. If you need time to build your market and company value after your product is released, better resourced companies can copy your idea or work around your patent while you are still too small to defend yourself.
Patents work when, by the time you release your product your company is valuable enough to spend the millions to defend them. For instance, if you develop a groundbreaking pharmaceutical launching into a new market, your company becomes very valuable simply because you have released the drug; you don’t have to wait until a market is created for it, the market preexisted your launch. On the other hand, Fraunhofer could not prevent others from stealing its intellectual property because there was not enough money at stake when it was stolen for it to be worth them suing everyone. (There was enough money later on, when the market was established, and there were patent suits aplenty. But by that time Fraunhofer had lost control of the market.)
Patents are valuable. But they will not necessarily win you the market.
What is a technology? The inventions we have been talking about are all technologies. Technologies do something useful. They are discovered or invented and then put to use in a process of innovation. Later they might spread through the economy in a process of diffusion.
Technologies are not always “things”. A technology does not have to be a physical object. The MP3 algorithm is a technology, even though it is intangible. The idea of a limited liability corporation is a technology, even more intangible.
Some new technologies are the result of turning a fundamental phenomenon into something useful, but most technologies are created by combining already existing technologies. The MP3 chip was the combination of the MP3 compression technology with integrated circuit technology. The MP3 PMP was the combination of the MP3 chip with digital storage and display technologies. The advent of a new technology makes many other new technologies possible because it is a new thing to combine. Some new technologies are so useful and can be combined in so many ways with already existing technologies that they create a wave of innovation. The internet is an example.
Some innovative technologies don’t depend on anything new at all: Apple’s entry into the PMP market contained very little new technology, and the new tech it had was not the primary cause of the iPod’s success. What Apple did was connect the component technologies in a new way, a way that was much more useful to its customers. This type of innovation is called architectural innovation.
This process of combining technologies is accelerated through modularity. People are ingenious at coming up with combinations of technologies to solve problems. Sometimes this is the result of deep understanding of the component technologies and what they will do when combined, and sometimes it is the result of making do with what the engineer has at hand, a type of “bricolage.” The usual difficulty in either case is getting the different component technologies to work together well. Much of the “tinkering” that goes on with new technologies is just this: the gradual improvement of components and linkages to make the combination of technologies function better.
One common way to make this easier is to modularize components. By having a clear delineation of what a component technology’s function is and by laying out how it works with other technologies—an interface—components can more easily be combined with other technologies. When SaeHan built the MP3 chip, they included a description of how it could be combined with other circuitry: SaeHan intended it to be used by others and hoped for it to be adopted widely. Technologies that are not meant to be used outside of the ambit of their manufacturer often are not modularized in this way, and this can slow their diffusion if unanticipated uses are later found.
Modularization is powerful because the interface is an abstraction of what the component does: the user does not need to take the messy details of the component’s internal functioning into account. This allows the modularized component to be improved or replaced entirely without disturbing the linked technologies, so long as the interface remains substantially the same (or has backwards compatibility).
The standards process is a way to encourage modularity, by guaranteeing functionality and an interface. The MPEG standard-setting process that resulted in the MP3 standard, among others, is an example. The standards group knew that by having a few, well-understood, standards, progress in technologies that used these standards would be faster. Fraunhofer worked hard to have its algorithm standardized: they knew having a standard blessed by a recognized standards group would reduce a couple of sources of uncertainty for adopters: it was a type of legitimation, a mark of approval that the technology worked as described; it was a guarantee of modularity, that the interface was stable and would not be implemented in incompatible ways between different products, allowing interoperability. More about legitimation in the next post.
Modularity encourages experimentation. The MP3 PMP itself was several technologies connected together. The MP3 PMP as technology consists of these component technologies as well as the architecture of their connection, forming a system.
Because the components are linked, the form and function of each component depends on the components it is linked to. Innovators improve a complex system either by adding new technologies to it (creating a new technology), improving the technologies and linkages that make up the system (sustaining or incremental innovation), or by changing the linkages between technologies (architectural innovation).
Modularization of the components encourages experimentation: new components can be added more easily or components can easily be swapped out for components that perform the same function in different ways, especially if the substitute has the same or similar interface. When Apple developed the iPod, they worked from a reference design they bought from another company. But their substitution of several new components—the touch wheel, the 2.5” disk drive, etc.—while still using the same reference design substantially improved the resulting MP3 PMP while avoiding the expense of re-architecting the entire thing.
Experimentation is a primary mode of technological improvement. When there is uncertainty about what exactly the market wants from a technology, or about how changes to a technology will affect its overall functioning (as often is the case when the technology is ahead of the science) then experimentation is the best way to improve the technology. This kind of experimentation often takes the form of tinkering by people with deep tacit knowledge of the tech gleaned through long periods of learning by doing.
Most technological change is improvement. We notice and talk about innovations that make a big difference, that are either a noticeable step-change in the functionality of a technology or are a technology that does something fundamentally new and interesting: radical innovations. But usually innovations are of a quieter sort, they take an existing technology and improve it. While each of these incremental innovations only adds a bit, they are so much more common that they can create much of the eventual value from a technology.
Compare the first MP3 PMPs and a late-stage iPod. They are fundamentally the same technology and serve the same use, but the PMP–in itself, as a standalone portable media player–was incrementally improved over time to become more and more valuable.
Incremental change mostly occurs along a very few metrics. When I say the iPod was an improvement I mean that, given the choice, people would prefer it to the Eiger MPMan. You could even suppose that if both were available at the same time that people would pay more for the iPod, and this would be one way to measure how improved it was. (This is difficult to do in the real world because technologies often become cheaper as they become more valuable. This is not because price and value are disconnected, but because several types of incremental innovation are usually ongoing at the same time resulting in simultaneous improvement in functionality and decline in manufacturing cost.)
To improve performance, innovators must first figure out what “higher performance” means. A higher performance disk drive is one that holds more data or reads/writes data faster, all else being equal. The performance metric always seems obvious in retrospect, but finding the answer to the question “What is Quality?” is one of the few true defining moments in a technology’s life-cycle.
A competitive basis is more than just a quality metric, it is a metric that becomes a primary differentiator between competing products in that market. You might prefer a black disk drive to a gray disk drive, all else being equal, but making black disk drives is not a basis for competition. It is not that important compared to other attributes of the disk drive and it is not how products can meaningfully differentiate themselves from the competition. In some cases there might be many metrics customers care about. If you are buying a car you might compare fuel efficiency, or comfort, or acceleration. If each of these metrics appeals primarily to a different customer group, the market becomes several markets. If you care most about acceleration you don’t shop for a car, you shop for a sports car. Others shop for a fuel efficient car or a luxury car.
The MP3 algorithm itself was developed with two quality metrics in mind: the heard fidelity of music, and the computational power needed to encode/decode. The MPEG authorized several standard digital music algorithms at the same time (there was a MP1, MP2, etc.) and MP3 was superior to them in terms of lower computational power, but inferior in sound quality. This was intentional: it was meant for bandwidth-constrained channels. This compound metric–better in bandwidth-constrained channels–was, at the time of introduction, a solution in search of a problem. It wasn’t until the consumer internet started to grow that this metric came to the fore as the competitive basis. MP3 outcompeted MP2 in the internet sharing of music market because it was better suited. It also outcompeted because it was made free. While the free software was illegal, it created enough users and MP3-encoded music that it became a de facto part of the interface that linked the various pieces of the digital music ecosystem together.
Though this quality metric guided the design of MP3, it was not one that was especially compelling while the algorithm was being developed. Fraunhofer worked for quality along this metric because that was the unique strength of perceptual coding, the technology that Brandenburg happened to know. Researchers often create the technology they know how to create without regard to what the market wants. If they are lucky, as Fraunhofer was, the market appears, but this is in no way a given.
In other cases, innovators have to search for the right metric to compete on. They can do this by talking to customers and trying to determine which problem they are solving, but customers often do not know. So the search is very often done through experimentation: many product designs are tried until one is found that resonates with customers.
At the beginning of the MP3 PMP market companies competed on many quality metrics, trying many variations of the product. Some players held more songs, some were smaller, some had better user interfaces, some were better for exercising with, some integrated with the user’s PC better, etc. The profusion of companies and PMPs between 1998 and 2003 was a result of this search for the basis of competition.
It is hard to build a market with many different bases of competition. It is confusing to the customer and the market fragmentation means the resources available for incremental innovation is spread across many technologies so each improves less. But what tends to happen, especially when the technology in question is a compound technology—one that is comprised of many components in a system—is exactly what happened in the MP3 PMP market: there is a profusion of varieties of the technology, each emphasizing a different combination of strengths. This profusion of products is usually accompanied by a profusion of companies. At some point a product hits upon the set of features that satisfies customers while focusing improvement on a metric (or a couple of metrics) that address the customers’ primary problem. This metric or metrics then becomes the sole basis of competition.
At this point, companies that cannot compete on the new basis leave the market. The few companies that can compete remain. The product design that embodies this newly discovered basis of competition is called the dominant design. When a dominant design appears, competitors, their customers, suppliers, and everyone else in the market orients around it. This makes radical innovation hard, though incremental innovation continues. Primarily, companies compete on cost, and innovation moves from product innovation to process innovation, where companies direct their resources to improving internal processes to lower costs and increase fidelity to the dominant design. The below chart shows the rise and decline of the number of US automobile manufacturers. The dominant design was established in 1923 by Dodge Brothers: the all-steel, closed body automobile.
Apple realized that the metric most important to customers was ease-of-use. Previous to the iPod, getting digital music, organizing it, and putting it on the device was a clunky process. Apple streamlined this with its combination of iPod, iTunes, and iTunes Store. This became the dominant design, with ease-of-use as the basis of competition. It did not mean that the iPod could then be inferior in other respects: the number of songs, the portability of the device, etc. were all comparable to the best of the other devices–these were constraints; this allowed the iPod to be accepted as an alternative to other PMPs. But the only way to compete with Apple’s dominant design would be to create a competing system that included PMP, software, and store.
The System often extends beyond the ‘technology’. Apple realized that to make the PMP easy to use it had to alter the technological system not just of the device itself, but of the things it connected to. The MP3 PMP itself was just a small part of the overall digital music system that emerged. It had to connect to a computer that had jukebox software. This software had to connect to a channel that delivered music and information, either the CD drive plus a database of album and song titles or a music store accessed over the internet. These channels had to be fed by a source of music, either the old CD distribution system or the new connections between the online music store and the record labels. Etc. Although calling this system of interconnected parts a technology itself makes the definition of technology a bit tenuous, the extension is clearly analogous. Apple was in a unique position to make this larger system easy to use because it built not just the PMP, but the jukebox software, some of the personal computers the jukebox ran on, and the store the music was bought in.
One company does not usually control all of the things that make up a useful technological ecosystem, especially when a major new technology is emerging. Things that are the needed but are controlled by someone other than the innovator are called complementary assets. In this system both music and the internet are complementary assets to Apple. The system will not work without them.
When new technological systems require complementary assets, innovators must find a way to integrate them. We’ll talk about the business strategy of this later, but from a technological point of view, this means having a clear and stable way of allowing other companies to work with the new system. Much as technologies were more useful if they had a clear path to interoperability, so are technological systems. The interfaces of systems, though, are often far more complicated than the interfaces of technological components because the things that must be considered span the spectrum from set technological choices (the algorithm used to encode the music, for instance) to business model considerations (how will the value created by shared) to hard-to-quantify human interactions (who will get the power and glory from the success of this new ecosystem?)
Technology improves along the competitive basis. Once a competitive basis is established, companies and researchers will direct resources towards incremental innovation that improves along that metric. Initially this improvement tends to grow at an increasing rate. Moore’s law is an example: the primary metric of performance in integrated circuit manufacturing is density of components; density has been increasing exponentially for more than 50 years.
The reasons given for the commonality of increasing growth in performance vary:
- Accumulating technological knowledge allows faster improvement: the learnings from one generation of improvement makes the next generation of improvement easier;
- Success attracts more resources: as integrated circuits became a big business, more resources (people, capital) were devoted to increasing density;
- Companies know their competitors will plan to increase quality by a certain amount in a certain time, enough resources must be spent to remain competitive with them (this becomes a bit of a self-fulfilling prophecy).
This process leads to a technological trajectory and momentum in the direction of improvement. Because stakeholders in the technological system think improvement will come along the competitive basis, they factor this improvement into their plans.
This trajectory of improvement in the competitive basis makes the types and rate of improvement predictable, which helps users of the technology plan for the future. Because the dominant design is locked in, producers focus all resources on improving along the competitive basis, knowing that if they do not they will lose the market to competitors who do. Because company processes have their own momentum, performance often continues to improve even after it exceeds what most customers can use.
When this happens, an opportunity opens up for upstart producers of the technology to compete in the market by offering an inferior but cheaper solution. This picks off customers who don’t need the full potential of the technology. But as the inferior technology undergoes its own exponential growth it starts to compete in the full market with the incumbent producers. This process is called disruption.5
Disruption also can happen when the inferior technology offers some functionality along a metric different than the mainstream basis of competition. This sustains the technology until its growth along the mainstream basis is sufficient to reach a larger market.
The bases of competition for recorded music equipment and media (not the music itself obviously) prior to MP3 were fidelity and portability. When the primary technology was vinyl records, the best quality gear was that which produced the most accurate reproduction: “audiophile” stereo equipment was always the leading-edge. Other technologies competed on portability: eight-track tapes and then cassette tapes. These two types of technologies competed primarily in separate markets: the makers of hi-fi equipment did not tout portability as a benefit and while the makers of portable gear would talk about sound quality, improved portability would trump fidelity in any new product. The two markets were briefly joined when compact discs were introduced, which offered excellent fidelity and, eventually, portability.
MP3s ended up competing on a completely different metric, ease of use. While cassette tapes were easy to use in the simplest sense—you put the tape in the tape deck and pressed play—they were unwieldy in a broader sense. If you wanted to make a playlist it was difficult, if you wanted to juggle thousands of albums it was difficult, if you wanted to buy and listen to something not found in your local record store it was difficult. Even when they were first introduced, MP3 devices made these things easier. It was more difficult to listen to any given song, but the overall music acquisition/organization/listening experience was easier.
The MP3 ecosystem was disruptive. The devices were far inferior on the fidelity metric, a metric which had far overshot what the majority of music listeners needed. Meanwhile they offered an improved experience on a new metric. This new metric brought in enough users to keep the technology commercially viable while sound quality was improved (through skip-prevention caches, more storage for higher bit-rate MP3s, and even newer codecs that were better than MP3). I would argue that the streaming services that dominate today are a continuation of this competitive basis.
Meanwhile, the technologies that could not improve along the new dominant competitive basis lost market share.
The ease-of-use metric existed all along, of course. CDs were easier to use than cassette tapes, and this was part of their appeal. The greater part of their appeal was audio quality, of course. But CDs could not compete with MP3 because they could not be further improved along the ease-of-use dimension when customers started to consider ease-of-use of the entire music getting/organizing/listening process, not listening alone. Exponential curves can not grow forever, technologies have their limits. Even Moore’s Law will eventually (and arguably already has begun to) tail off because of fundamental physical constraints on how small transistors can be. This tailing off as the opportunity space of technological improvement is exhausted results in the famous S-curve.
Customers, however, want their problem solved, they don’t generally care how. They demand improvement of their experience–not of specific technologies–along the competitive basis. If continuing ease-of-use improvement means switching from CDs to MP3s, customers as a whole are not generally sentimental about it (though there are certainly pockets of nostalgia, like current purchasers of vinyl records).
As an example, integrated circuit manufacturing, and thus Moore’s Law, depends on semiconductor lithography technology. Improvements in lithography result in higher density ICs. Any given lithography technique goes through its own S-curve. As each S-curve runs its course, IC manufacturers switch to newer, higher-performing techniques to continue the growth of the density metric as seen by their customers.
Radical change is often driven by customers. In this post I’ve tried to isolate the technological aspects of innovation. But other considerations also matter. Treating technology innovation as if it’s exogenous—not influenced by the larger system that includes everything else with an interest in it—is a common simplifying assumption. It is clearly not true: even here we have to note that the competitive bases along which technologies are developed come from the technology’s customers, not from innovators contemplating a platonic ideal of the technology in isolation.
Even more, the technology’s developers are not necessarily the ones that decide what the technology is or how it will be used. The MP3 algorithm was not developed with the technology of music sharing in mind; to the contrary, Fraunhofer hoped to embed it in the old studio model, with expensive encoders and inexpensive decoders. The transformation of this model into the widespread creation and sharing of MP3s, against Fraunhofer’s wishes, was done entirely by users. User innovation is often a major source of innovations.
This user innovation not only changed the uses of the technology but shaped the metrics that guided its further development. While Fraunhofer probably hoped its technology would be implementable by trained engineers, they could not have foreseen that its entire point would be enabling the ease-of-use of a music listening system.
It’s not really surprising that technology released into the wild is used in unanticipated ways. The human reaction to innovations is one of the biggest sources of uncertainty in the commercialization process. Not just how people will use a new technology, but how they will view it and react to it, how quickly they will adopt it, how they might fear it and ask it to be regulated. Technologies are shaped as much by their interaction with the sociopolitical sphere as they are by the underlying phenomena they harness. We will talk about that in the next post.
Bibliographic Note
A great overview is Clay Christensen’s “The Evolution of Innovation” 6 but most of the information in this post is part of the basic underpinnings of the theory of technological innovation rather than direct quotes, so I have not footnoted every idea. The books below are some of the best treatments of these ideas.
1. W. Brian Arthur, The Nature of Technology
Technologies as embodiments of phenomena or combinations of other technologies.
2. Baldwin and Clark, Design Rules
Modularity and interfaces. Architectural innovation.
3. Christensen, Innovator’s Dilemma and Innovator’s Solution
Competitive bases. Disruption. Modularity.
4. Thomke, Experimentation Matters
The importance and process of experimentation in product design.
5. Utterback, Mastering the Dynamics of Innovation
Incremental vs. radical innovation. Dominant designs. Product vs. process innovation. S-curves.
6. Von Hippel, Sources of Innovation
User innovation.
Whether he actually said this or not is open to question. See https://quoteinvestigator.com/2015/03/24/mousetrap/ ↩
I will refer to the MP3 player as a PMP, a portable media player. When I say MP3 by itself I am referring to the algorithm. ↩
Cyril Smith, quoted in David Nye’s Technology Matters. ↩
Nye, David, Technology Matters, p.9. ↩
This is an unfortunate name: the word “disruption” has so many other, more common meanings that its use as a narrowly-defined aspect of technological innovation causes confusion. I will try to use it in its technical sense. ↩
Christensen, Clayton M. “The Evolution of Innovation.” In Technology Management Handbook, edited by Richard Dorf. Boca Raton, FL: CRC Press, 1998. ↩