Reaction Wheel

Generative AIs can do some things better than people: they can code faster in many instances, they can write junior high school level essays faster, they can create detailed images on demand no matter what the demand, and they can write a mediocre sonnet about whatever you want. But it’s still obvious that human intelligence is more general. Humans are not yet bested. Even so, the recent advances in AI have caused a widespread belief that AIs will imminently surpass human intelligence: AI is advancing so rapidly, it is not hard to imagine how good it will be in a couple of years, or even a few months.

Of course, every technology looks exponential until it doesn’t. The rate of improvement starts to level out when a technology starts to reach its inherent limits, because every technology has inherent limits. At least, every technology to date. But the belief that AI will become smarter than humans is predicated on AI being smart enough to not only improve itself but to figure out ways to expand its own capabilities beyond any inherent limits. The current question is not “is AI smarter than humans?”, it is “can it become so?”

This post argues that there may someday be an AI that can improve itself beyond any bounds humans can imagine, but generative AI isn’t it. The current AI technology is not only not smarter than humans, it can’t improve itself to be smarter than humans. My argument relies on fundamental conceptual limitations on types of reasoning. This is important not just to bound what current AI can do, but to expand our thinking to what would need to be done to create true AGI.

1950s-1970s: Deductive AI

“Let us now return to the history of AI. One of the early things which people attempted to program was the intellectual activity of theorem proving.”

Douglas Hofstadter, 1979, Gödel, Escher, Bach, p. 609.

Theorem proving¹, game playing², machine translation³, general problem solving⁴, and conversation⁵ were some of the earliest things AI pioneers attempted. All of these efforts used variations on a single way of producing output: they used known rules to deduce from known facts.

Deduction is a way to create new knowledge. If we know that all men are human and that Socrates is a man, then Socrates must be human. (The rule we are following here is modus ponens: if you know that “if p then q” is true, then if p is true q must also be true.) Must be, not might be: deduction takes truths and creates new truths; certainties, not likelihoods.

That Socrates is human seems implicit in what we already knew. But making it explicit makes it new knowledge. This is a subtle point, so here’s a better example. Consider the equation \( ax^2+bx+c=0 \). Before Brahmagupta described the quadratic formula in 628 AD, people did not know an algebraic way of solving this.⁶ But, of course, the quadratic formula follows rather simply from the equation given and the rules of algebra.⁷ Even so, just as it was new knowledge to you when you learned it in junior high school, it was new knowledge to the world when it was first derived. Deduction creates knowledge by using known rules on known facts.⁸ This is what mathematicians do when they prove theorems. They explore the inevitable consequences of an accepted set of rules and axioms.

The first wave of AI was primarily deductive: it solved problems by applying known rules to known facts. The facts might have been the state of a board in a checkers game and the rules were the rules of checkers. Or the facts might have been a sentence in Russian and the rules were the definitions in an English-Russian dictionary. It’s relatively easy to program a computer to perform deduction. The hard part is not in knowing the facts or applying the rules, it is in knowing which rule to apply at any given time and whether applying that rule moves you closer to your goal. AI researchers came up with several clever ways to do these things, but they primarily used brute force.

Some of these programs were fruitful and some were dead ends. Machine translation, for instance, did poorly. Language has too much ambiguity to deduce meaning. But, overall, AI using deduction was successful on its merits. By 1987, an expert system developed at CMU was reportedly cutting the time it took DEC salespeople to configure new VAXen by hours, saving DEC some $40 million over five years.⁹ But this was peak Deductive AI. Progress slowed markedly towards the end of the 1970s; improvements in capabilities started to rely primarily on increases in compute speeds, not fundamentally new techniques. Use cases were limited, and human-level intelligence was never approached. With plateauing performance, funding dried up and the “AI Winter” of the 1980s-1990s set in.

1990s-2020s: Inductive AI

Holmes: For example, observation shows me that you have been to the Wigmore Street Post-Office this morning, but deduction lets me know that when there you dispatched a telegram…an explanation is superfluous; and yet it may serve to define the limits of observation and of deduction. Observation tells me that you have a little reddish mould adhering to your instep. Just opposite the Wigmore Street Office they have taken up the pavement and thrown up some earth, which lies in such a way that it is difficult to avoid treading in it in entering. The earth is of this peculiar reddish tint which is found, as far as I know, nowhere else in the neighbourhood. So much is observation. The rest is deduction.

Watson: How, then, did you deduce the telegram?

Holmes: Why, of course I knew that you had not written a letter, since I sat opposite to you all morning. I see also in your open desk there that you have a sheet of stamps and a thick bundle of postcards. What could you go into the post-office for, then, but to send a wire? Eliminate all other factors, and the one which remains must be the truth.

Arthur Conan Doyle, 1890, The Sign of Four.

“In 1986, when we launched the journal [Machine Learning], machine learning was still viewed as a branch of artificial intelligence. By 2000, many researchers committed to machine learning treated it as a separate field with few links to its parent discipline. There are now active PhD-level researchers who have never taken a course in artificial intelligence and who see no reason why they should, as their interests lie in completely different areas.” 

Pat Langley, 2011, “The changing science of machine learning“.

The AI pioneers focused on deduction. Computer systems are defined with a type of formal logic, and this may have helped set the path. Deduction can also be effective in a limited universe of knowledge and rules, and computers were memory constrained at the time. But I also think the human tendency to idolize deduction was at work. Deduction seems like the epitome of reasoning. Knowledge you create through deduction is guaranteed to be correct if the antecedent knowledge and the rules are correct. Deduction is rigorous, it is pure. We consider people who are good at deduction—like mathematicians and philosophers—very intelligent, and so emulating intelligence by becoming good at deduction makes a kind of sense.

This is selection bias. We think deduction is a superior kind of intelligence only because we are so naturally bad at it. Our brains don’t seem to work that way. To be good at deduction you have to have enough mental horsepower to work around your own limitations. Trying to make a machine emulate human intelligence by programming it to deduce was exactly wrong: like working towards the subtlety of the human hand by building a vise. What the human brain does best is not deduction; what Sherlock Holmes is illustrating in the above quote is not deduction. The human brain naturally does induction; what Holmes is doing is induction.

Deduction creates new facts using known rules and existing facts. Induction takes known facts and creates rules; these rules can then be used to infer new facts. But while deduction’s new facts are guaranteed to be true, induction’s new facts are only likely to be true.

Induction is extremely useful. The ancients observed the sun rising every day and assumed the sun would rise again tomorrow, even though they did not know why. But induction does not provide the same assurance that deduction provides. A turkey sees the sun rise every day and so assumes he will see it rise the following day. And he will be right, until Thanksgiving.

Induction works when there is a constant mechanism that is generating the future. (Or at least, constant within the bounds of how the information generated by induction will be used: we know that someday our sun will not rise but we assume this is some way off.) The beauty of induction is that we don’t need to know what that future-producing mechanism is to use induction to predict the future. Induction creates the knowledge that the facts you have imply; it deals with the messiness of real-world data and our usually incomplete understanding of why things happen.

Induction—or probabilistic reasoning, one aspect of induction—has been used in AI since the beginning. It just didn’t work that well early on. When the AI winter set in, researchers started called their inductive reasoning systems anything but artificial intelligence, as Pat Langley emphasizes above. One of the earliest inductive learning programs I remember was Firefly, in 1995. The website would collect information about music you liked and make recommendations about other music you might like. It did this by learning from its users the connections between various artists. If people who like both Guided by Voices and The Beatles often also like Neutral Milk Hotel, then if you like Guided by Voices and The Beatles, you might also like Neutral Milk Hotel. You got recommendations by telling the system what music you liked and this then fed the system more information to make better recommendations to others in the future. It learned linkages even though it had no idea why GBV, The Beatles and Neutral Milk Hotel might be related. A computer asking someone what music they like and then recommending other music they like is far more similar to human intelligence than proving theorems, but no one considered it AI. It was called collaborative filtering or collective intelligence, implying the intelligence was supplied by the collective, not generated by the computer. This is a version of the Chinese Room Argument. Firefly wasn’t “thinking”, it wasn’t “figuring things out”. It was just telling you what the data said. (For an excellent history of machine learning, see Chris Wiggins and Matthew Jones’ How Data Happened, especially chapter 9.)

Humans do what Firefly did all the time, and we consider making good recommendations a sign of intelligence. But Firefly and its ilk were not considered intelligent precisely because they did something humans do so naturally: inductive inference. We consider human intelligence paramount but then decide things humans can do easily are not really intelligence. It’s ironic.

Firefly became useful at the time it did because the internet had become widespread and there was suddenly an easy way to gather data. Unlike the expert systems of the 1980s, this wasn’t curated data boiled down to a single “correct” point of view, it was real-world data. The messy data was made predictive using mathematical techniques borrowed from statistics. The predictiveness of these statistical techniques went up an order of magnitude when huge amounts of data were available. Eventually, human programmers no longer needed to tell a program exactly how to solve a problem, they could let the patterns in the data teach the program.

It would be hard to make a list of the most influential inductive AI programs because they were not part of a structured research program, they were much more commonplace. In some cases, like Firefly, they were academically unambitious. In others, like Google Translate, they were huge and visible advances. In almost all cases they were not called AI. They were called machine learning, deep learning, and big data. Critics quipped that “machine learning is just statistics”. This is not untrue, but the problem with the criticism is that it denigrates induction. It’s is a bit like criticizing philosophy by saying it’s just arithmetic. Statistics should be appreciated. Like other inductive techniques, statistics allows us to take an incomplete set of facts, infer some possible relationships between them, and then use those relationships to predict the future. This is close to magic.

Levels of Inference

KIRK: Everything Harry tells you is a lie. Remember that! Everything Harry tells you is a lie!

HARRY: Now listen to this carefully, Norman: I AM LYING!

NORMAN: You say you are lying, but if everything you say is a lie then you are telling the truth, but you cannot tell the truth because everything you say is a lie, but…you lie, you tell the truth, but you cannot for you l…Illogical! Illogical! Please explain! You are Human! Only Humans can explain their behavior! Please explain!

Star Trek, 1967, “I, Mudd”.

The supreme triumph of reason…is to cast doubt upon its own validity.

Miguel de Unamuno, 1921, The Tragic Sense of Life, London: MacMillan and Co., p. 104.

Here’s something interesting about deduction: You can’t deduce deduction. Lewis Carrol argued this (in a very Lewis Carrolian way) in 1895.¹⁰ In a dialogue between Achilles and the Tortoise, he shows that to accept the results of a deduction, you first must accept the rules of deduction themselves. If you do not, no deductive argument can convince you of deduction because, of course, the argument would rely on rules you have not accepted.

If deduction is not self-proving, how did we decide to accept deduction as valid? My guess is that we decided to accept it because it worked. We noticed that when we proved something deductively, like the quadratic formula, the result was always true. This is a powerful argument, but it is an inductive argument.

Of course, it’s not true that deduction always works, as Bertrand Russell demonstrated in 1903 with Russell’s Paradox. The simplest version of his paradox is to take a sheet of paper and write on it “This sentence is a lie.” If the sentence is true, it is false; if it is false, it is true. This is the paradox Captain Kirk used on the artificially intelligent robot Norman in the quote above. In the show, Norman starts to emit smoke and then, presumably, shorts out. (I just tried this on ChatGPT. Please let me know if you see smoke issuing from their datacenter.)

And, in 1931, Kurt Gödel showed that no mathematical system could be complete, that there would always be true things about any deductive system you could not prove using the system. This opened the door to a series of papers undermining the foundations of mathematics and, by extension, any deductive system.¹¹ But, despite Gödel, deduction works almost all the time, so any good inductionist will be comfortable using deduction. A good deductionist, of course, would shun deduction. That mathematicians still exist is a paean to induction.

Induction has a similar problem: you can’t deduce induction. This was shown by David Hume, whose “problem of induction” is the first thing every philosopher brings up when discussing induction.

It was David Hume’s argument against induction that hooked me on philosophy. Surely we have good reason to believe that the sun will rise tomorrow, even though it is possible that it won’t; yet Hume provided an apparently unanswerable argument that we have no way to show this. Our inductive practices have been reliable in the past, or we would not be here now to think about them, but an appeal to their past successes to underwrite their future prospects assumes the very practices we are supposed to be justifying.

Peter Lipton, 1991, Inference to the Best Explanation, p. xi.

It’s not surprising, since you can’t deduce deduction, that you can’t deduce induction. But it might surprise you that you also can’t induce induction:

There’s a joke about a planet full of people who believe in anti-induction: if the sun has risen every day in the past, then today, we should expect that it won’t. As a result, these people are all starving and living in poverty. Someone visits the planet and tells them, “Hey, why are you still using this anti-induction philosophy? You’re living in horrible poverty!”

“Well, it never worked before…”

Scott Aaronson, 2013, Quantum Computing Since Democritus, p. 229.

As with deduction, you need circular reasoning to convince someone of induction using only induction.

But if you can’t deduce induction and you can’t induce induction, why do we believe in induction? Just as deduction is believable because of induction, there must be a more general form of reasoning that allows us to use induction comfortably. I’ll call it Invention for the moment, but it is clearly something humans can do that encompasses induction (and thus also encompasses deduction.) There must be layers of inference, from the most specific to the most general. The most specific, deduction, offers the most assurance of truth, and that assurance wanes as inference becomes more general. Invention might not even be inference at all. Regardless, it is a way to think, and so a type of intelligence.

Intelligence can mean many things, but the hallmark of human intelligence is its generality. In this sense induction is more “intelligent” than deduction because it is more general, and Invention is more intelligent than induction. The outward progression is the path towards AGI.

AGI

AI has existed since the birth of computers. It has been around long enough that it has had periods of excitement where some leading experts make extraordinary claims. Claude Shannon was one of the greatest pioneers in computer science and AI. In 1961 he said: “I confidently expect that with[in] a matter of ten or fifteen years something will emerge from the laboratory which is not too far from the robot of science fiction fame.” His prediction was that by the mid-1970s we would have walking, talking, thinking autonomous machines. Forty years later, we can still barely make a robot walk. It certainly cannot think for itself. Today, there are surveys (containing large variation of views) concluding that there is a “50% chance AI will outperform humans in all tasks in 45 years”. It all sounds so familiar.

Peter J. Bentley, 2018, “The Three Laws of Artificial Intelligence: Dispelling Common Myths”¹²

AGI stands for “Artificial General Intelligence”, but what it tends to mean is artificial intelligence that rivals human intelligence. Intelligence is ill-defined. Chess playing computers are intelligent in a way, and the current crop of LLMs are also intelligent in a way. The latter are even “generally” intelligent though not yet as generally intelligent as we are. But we designed these AIs and we improved the design over time. At some point the AIs become intelligent enough to improve themselves and, since electronic circuitry should be faster than our wetware, they should be able to improve themselves faster than we can improve them.

People worry about this. There have always been people who have predicted that some new technology is more than just dangerous, it poses an existential risk. We’re still here, so they’ve been wrong each time. But with every new breakthrough technology someone argues that “this time is different.” It’s easy for us tech folk to dismiss this (because induction, actually.) But like the turkey waiting for Thanksgiving, we need to ask ourselves if this time is different.

Even during the first wave of AI there was widespread fear. The homicidal computer HAL, in 2001: a Space Odyssey (1968), and the genocidal computer WOPR, in WarGames (1983), bookended the 1970s. The crux of the fear was that if AI could find a way to make itself slightly more intelligent then, as something more intelligent, it could find a way to make itself even more intelligent, and so on, unleashing an exponentially increasing cycle of self-improvement, until the AI was so much smarter than humans that it could decide to end humanity on a whim. This was called by John Von Neumann (no relation) in 1958 “the singularity”. When there is suddenly a large change in AI capability, people wonder if it is the beginning of that exponential climb. They look at AI today, compare it to two years ago and say “at this rate of growth in intelligence, humanity will be outstripped in a matter of years, if not months.”¹³

Deductive AIs turned out not to be a threat. While intelligent, they were nowhere near as intelligent as humans. The fear dissipated in the 1980s, as deductive AI progress plateaued. This is typical of new technologies: change is slow as people figure out how a technology works, speeds up as they master it, and then plateaus as its promise is tapped out. Technologies follow an s-curve. It should be expected (inductively) that inductive AI would also follow an s-curve, and extrapolating from the last couple of years of progress would be the wrong way to think about it.

Technology s-curves do not flatten out because we are not smart enough to continue their improvement. They flatten out because the technologies reach their potential. Technologists evolve technologies along this s-curve until they plateau and then they look for new technologies that can take their place. Moore’s Law, for instance, is a single s-curve of improvement of transistor size in silicon, and thus speed of computation in computer circuits. This s-curve was a long-standing one, but has recently started to flatten. And, if you look deeper, the technologies that enabled it also follow s-curves.

But, say some very smart people, if the AI could improve itself, then it need not plateau. We know that Deductive AI did not improve itself to Inductive AI, and we know from the last section that this is because induction can’t be deduced. It was never a question of just providing better algorithms or more horsepower to Deductive AI before it could figure out induction, it required a change in the mode of thinking. No deductive system could have come up with induction. Jumping from one s-curve to the next requires the development of a new technology.

In assessing the improvement in intelligence in AI, we have already seen a jump from one s-curve, Deductive AI, to the next, Inductive AI, spurred by the new technologies of machine learning and RNNs. There was a long rise in Deductive AI followed by a plateau, the AI winter. Then there was a long rise in Inductive AI.

What happens now?

There are two questions that bear on how intelligent inductive AI can become.

How long until the Inductive AI s-curve starts to plateau?

How far along the s-curve are we with induction? It’s hard to know, but signs are that we are already starting to plateau. Sam Altman, CEO of OpenAI said “I think we’re at the end of the era where it’s going to be these, like, giant, giant models. We’ll make them better in other ways.”¹⁴ One of these other ways, speeding up of the underlying computing power, also seems close to being tapped out.¹⁵ The time from start to end of the first s-curve was about 30 years. It has been ~30 years since the start of the second s-curve. This by itself means little, but it’s a base rate.

If we have already reached the beginning of the plateau, it doesn’t mean the current technology won’t continue to be commercialized in interesting and “disruptive” ways (in fact, technologies often find their best use as they start to plateau in performance because they are both more predictable and because investment in uses becomes more lucrative than investments in improvement), it just means we won’t see the same rate of improvement in performance that we’ve seen for the past 15-20 years.

Can Inductive AI itself find the next technology and cause the jump to the next s-curve to happen sooner?

I don’t doubt that Inductive AIs can improve themselves. Given their own code, and their ability to write code, they could rewrite their code to be better. Induction, like deduction, can create new knowledge. But this kind of improvement is incremental, it can’t jump curves. Inductive intelligence can’t be used to conceptualize what I’ve called “Invention”, just as deductive intelligence could not conceptualize induction. This hard barrier to improvement is an impassable roadblock to the singularity. If you believe that human intelligence is something fundamentally more than induction, then we are not currently on the road to AGI.

Until we use our inventive intelligence to invent artificial Invention, the next curve jump, this AI can’t become AGI. And this assumes we even can: it may be that humans can’t conceptualize our own intelligence. We can’t deduce deduction or induce induction, so why should we believe we can invent Invention? If we can’t, then the creation of the next kind of intelligence will probably only happen in the same way that our intelligence happened: not designed but chanced upon by trial and error through the process of evolution.

Postscript: what is Invention?

Humans do something more than induction. I’m calling it Invention because I think it’s more than a mode of reasoning, but I’m not sure. I’ll explain what I mean.

Here’s a quote from a recent article on AI:

Dr. Williamson considers mathematics a litmus test of what machine learning can or cannot do. Reasoning is quintessential to the mathematical process, and it is the crucial unsolved problem of machine learning.

Siobhan Roberts, New York Times, 7/2/2023, “A.I. Is Coming for Mathematics, Too“

When Williamson talks about “machine learning”, he is talking about induction. But what does he mean by “reasoning”? Deduction is the type of reasoning that constitutes a mathematical proof. And machine learning/induction is clearly also reasoning. When an epistemologist talks about inference (ie. reasoning), they mostly mean deduction and induction. But obviously Williamson doesn’t mean deduction and he doesn’t mean induction. I believe what he means is that machine learning can’t decide which deductive path to go down. This is a problem familiar to every 13 year old taking geometry: they may know the relevant theorems, but when asked to construct a two-column proof, they have no idea which theorem to use when. Choosing the deductive path to take in a proof is partly induction (in the form of learned patterns and heuristics) but is also something more. This “something more” is what Williamson means when he says machines can’t reason.

The reasoning he is thinking of is probably abduction, or inference to the best explanation (IBE). (There are arguments in the philosophical literature about the exact definitions of these two things and how they overlap, but I am going to use them interchangeably.)

Abduction is the process of taking the available data and figuring out the best explanation for it. This is akin to induction in that you are creating “rules” to explain the data, but it is crucially different because it is an attempt to discover why, not just what. For example, we may use induction to find a rule describing how far a dropped ball has fallen after a certain amount of time. But this rule does not tell us why the ball is falling. We have to use abduction for that.

The philosopher Charles Peirce said “Abduction…is the only logical operation which introduces any new idea; for induction does nothing but determine a value, and deduction merely evolves the necessary consequences of a pure hypothesis. Deduction proves that something must be; Induction shows that something actually is operative; Abduction merely suggests that something may be.”¹⁶ This is, in my opinion, not quite right: as mentioned previously, deduction can be used to introduce a new idea, such as the quadratic formula, and induction can be used to introduce a new idea, such as Newton’s Laws. But, as Peirce said, ideas created through deduction are entailed by the data and rules, while ideas created through induction summarily describe what has actually happened. IBE/abduction, on the other hand, has no claim to truth. If I infer an explanation from data, it may be one of innumerable possible explanations. And yet it seems we are pretty good at finding explanations that later prove to be true or, at least, useful. This has to be part of what Williamson means when he says “reasoning”. And yet this type of reasoning is far less reasonable than deduction or even induction.

[Edit, 12/17/2023: I was reading Michael Huemer’s excellent Understanding Knowledge and it struck me that I never explained why “invention”, aka IBE, can justify induction. Induction assumes that there is an unchanging (or, at least, unchanging during the validity of the inductions) mechanism that generates observations. By making observations and using induction we can know something about that mechanism and thus predict what observations it will generate in the future. The problem is that we can’t induce that there are unchanging mechanisms without begging the question. But we can use IBE to infer that there are constant mechanisms that generate some sets of regular observations, because this is the best explanation for them. Thus, while we can’t use induction to justify induction, we can use IBE. I don’t remember exactly where in the book Huemer says something similar to this, but it was close to the end of chapter 12.]

Descriptions of IBE distinguish between the “likeliest” explanation and the “loveliest” explanation. The likeliest is the one best supported by the evidence, but it is the loveliest, the explanation that provides the most understanding, that we gravitate towards.¹⁷ Mathematicians often talk about proofs being “beautiful”, and use this as a mark in their favor. Scientists do this too. In The Double Helix, James Watson called his proposed structure of DNA “too pretty not to be true.”¹⁸ We seem geared to do this, even to the point, as Lipton notices, that we are biased to accept conspiracy theories which are often undoubtedly “lovely” even when they are extremely unlikely. This is clearly unreasonable, but whatever mechanism results in conspiracy theories also results in lovely but at-the-time-unlikely explanations, like Einstein’s special relativity.

Note that as we are forced by lack of knowledge of the world (either rules or data) outward from deduction to induction to abduction, we lose precision. Thinking becomes less mechanical, less susceptible to speed of computation and brute force. Humans are able to do something machines still can’t do: reason effectively in the relative or even complete absence of objective data. (For the latter, consider Descartes’ proof of his own existence, his cogito.)

IBE is probably the type of reasoning we need to make a leap to if we want to come closer to AGI. IBE seems to be outside of induction and you can clearly abduce induction, so induction is entirely contained within IBE. And, as you’ve probably now come to expect, IBE can not be used to prove IBE: it would be a circular argument. In the same way anti-inductionism justifies itself, Inference to the Worst Explanation (IWE) does as well:

• Most attempts at explanation are unsuccessful;
• These explanations are arrived at by application of IWE;
• The worst explanation of this is that IWE is a reliable rule of inference.¹⁹

If you want to extend my argument to say that there must be a type of reasoning beyond abduction to explain why IBE makes such evident sense and IWE does not, be my guest; it’s above my pay-grade.

Abduction/IBE can’t be all of invention. It goes backward from data to produce an explanation of the data, but then needs to go forward from that explanations to create something new. An explanation of thermodynamics must still be operationalized into a steam engine. IBE might be the hard work, in terms of intelligence, but the ability and drive to invent relies on more than just reasoning.

Invention requires the willingness to be wrong, to try things that seem like good ideas and see what happens, even to imagine things that seem like good ideas and imagine what will happen. Many intellectual breakthroughs occurred because someone imagined something they thought to be counterfactual and determined its consequences. Einstein, for instance, imagined light traveling at the same relative speed no matter the velocity of his frame of reference and did the math to see the consequences. The math itself was not the breakthrough, it was the supposition, counter to all lived experience, that something would have the same speed in any frame of reference. Of course, a supposition isn’t a breakthrough either, but when its consequences explained previously unexplained things, when the explanation was lovely, it became an avenue worth exploring.

All of these modes of thinking also require the thinker to ask questions and to want to “make meaning“. They require the thinker to have intentions, what Vaughn Tan calls “self-directed intentions”. Humans flourish with meaning, even when the meaning they are using is false. (Consider religion, for instance. Not your religion, of course. I mean all the religions that don’t agree with yours.)

Meanings are not goals, though goals may be instrumental in making meaning. Self-directed intentions are flexible, self-aware and self-modifying. An over-strict attention to goals precludes intelligence. Nick Bostrom’s paperclip maximizer—an AI that has been tasked with producing as many paperclips as possible and that figures out how to turn all matter in the universe, including humans, into paperclips—is not a consequence of artificial intelligence: this type of goal-directed behavior is incompatible with general intelligence.

The paperclip maximizer is an overpowered problem solver, it is not intelligent. Intelligence requires the ability to discard goals, it requires the ability to think autonomously. You can see this in considering how we abduce, how we play, how we do thought experiments, how we make meaning. These things are not separate from our intelligence, they are its quintessence. An AI that was programmed to have a strict and unalterable goal would not be able to invent and so would be restricted to a lower sort of intelligence, not the general intelligence humans have.²⁰

Human intelligence is not single minded because to move from induction to abduction we have to let go of strict goal determination. Human intelligence is bound up in our autonomy and our meaning-making. These can be subtly at odds, but any system that would move beyond induction would have to somehow reconcile them. This is the challenge of the next generation of AI.

---

1. Newell, Shaw, and Simon’s Logic Theorist, 1956. ↩

2. Shannon’s 1950 paper “Programming a Computer for Playing Chess” and Samuels’ checkers playing programs through the 1950s. ↩

3. The Georgetown-IBM Experiment, 1954. ↩

4. Newell and Simon’s General Problem Solver, 1957; Bobrow’s STUDENT, 1964; Slagle’s SAINT, 1961. ↩

5. Feigenbaum’s ELIZA, among others. ↩

6. People did solve quadratics before this, of course, but Brahmagupta is the first person documented to have laid out the quadratic formula the way we use it today. In any case, grant me that there was a time before people knew the equation. ↩

7. You can easily derive the formula using math you learned in high school: complete the square and then solve for x. ↩

8. I’m using “facts” to mean something accepted as true within the context of the system being reasoned about. ↩

9. Feigenbaum, McCorduck, The Rise of  the Expert Company, New York: Basic Books, 1988, p.225. ↩

10. Carroll, Lewis, “What the Tortoise Said to Achilles”, April 1895. Mind. IV (14): 278–280. doi:10.1093/mind/IV.14.278. ↩

11. For a great introduction to this see Morris Kline’s Mathematics: the loss of certainty, chapter XII: “Disasters”. ↩

12. In Should We Fear Artificial Intelligence, European Parliamentary Research Service, March 2018, p.10. ↩

13. There’s also the less existential fear is that even if we do not reach the singularity, the current crop of AIs will be intelligent enough to supplant all human labor, leaving us with nothing to do and no way to support ourselves. This, of course, makes no sense. Trade is circular. If machines did all the farming, for instance, but humans could not afford to buy the food, what would the machines do with the food? They would either have to give it away, or stop growing it. If they stopped growing it, humans would step in and start farming again. This is true of the economy as a whole. There would be, of course, a serious rebalancing of jobs based on comparative advantage. How this would work out is up for dispute. I’ve written about this before and it’s a boring argument to have for the umpteenth time, so I’m going to skip it, aside from this note. ↩

14. Will Knight, “OpenAI’s CEO Says the Age of Giant AI Models Is Already Over”, Wired, Apr 17, 2023, https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/  ↩

15. https://www.fromthenew.world/p/diminishing-returns-in-machine-learning ↩

16. Peirce, Charles Sanders. Collected Papers of Charles Sanders Peirce, vol. 5. United States: Harvard University Press, 1931, p. 106 or 5.172. ↩

17. Lipton, op cit., pp. 59-60. ↩

18. Watson, JD, The Double Helix, ch. 28. ↩

19. Psillos, S. “The Scope and Limits of the No Miracles Argument”, 2011, In: Dieks, D., Gonzalez, W., Hartmann, S., Uebel, T., Weber, M. (eds) Explanation, Prediction, and Confirmation. The Philosophy of Science in a European Perspective, vol 2. Springer, Dordrecht. ↩

20. You could imagine an AI “superego” that takes the inventions an abducing AI makes and filters them for paperclip-maximizing. This superego would not be intelligent, but the system would be similar to Bostrom’s construct. But this would not be a result of AI, it would be a use. Any technology can be put to bad use. In this sense the paperclip maximizer is as banal as the atomic bomb: it is no more reason to fear AI research than the atomic bomb was to fear research into nuclear physics. ↩