The discussion about whether or not progress in science and technology is slowing down could greatly benefit from a thorough reading of Michael Polanyi’s 1966 classic, The Tacit Dimension. In it, he developed a compelling case for why the advance of science depends on a particular type of culture which is embodied in what he calls the society of explorers. Around the world, it appears that the traditional methods which more closely approximated the society of explorers were slowly replaced with a process designed for perfectionism, derived from the Enlightenment belief that the sum of all knowledge is that which relates given facts using explicit rules.
This was an abstract introduction. Let’s get into Polanyi’s argument so we can emerge understanding why science may be slowing down, and how we can speed it back up.
The first two chapters of Polanyi’s book can be tedious, but they provide important insights. First, he introduces the idea of tacit knowledge, which is the idea that, for example, you can ride a bicycle without being able to describe the mechanics of your movements using scientific laws and engineering data. Here is Polanyi:
“I shall reconsider human knowledge by starting from the fact that we can know more than we can tell. This fact seems obvious enough; but it is not easy to say exactly what it means.”
Indeed, how do you speak intelligibly about the knowledge you cannot describe? He uses the example that we can recognize a face without communicating how we know this. We can, however, reconstruct faces pretty well by recognizing the pieces if we are presented them, like noses, mouths, etc. But we cannot tell how these particulars are tied together to make a whole. Comprehension of the whole is tacitly held, even though it is constructed from explicit particulars.
In the second chapter, Polanyi discusses integrative levels, and how explicit knowledge of one level does not inform us about the functioning of the next higher level. He illustrated the concept nicely:
“You can see, for example, how, in the hierarchy constituting speechmaking, successive working principles control the boundary left indeterminate on the next lower level. Voice production, which is the lowest level of speech, leaves largely open the combination of sounds into words, which is controlled by vocabulary. Next, a vocabulary leaves largely open the combination of words into sentences, which is controlled by grammar. And so it goes.”
Polanyi applies integrative levels to the sciences. For example, physics creates boundary conditions for biological systems, but the rules governing biological systems cannot be explained by reference to the rules of physics, just like grammar cannot be explained using vocabulary.
For the question of a research and development program, we have a higher level technological system which is intended to achieve some result in a human environment. Whether it is successful depends on the rules of physics and engineering, but explicitly relating those facts, such as through a systems analysis, cannot fully describe the value of the resulting technology. The value the technological arrangement has depends on tacit knowledge. Here is Polanyi, discussing this point in a different way:
“A living function has a result which it may achieve or fail to achieve. Processes that are expected to achieve something have a value that is inexplicable in terms of processes having no such value. The logical impossibility of such explanation may be affiliated to the dictum that nothing that ought to be, can be determined by knowing what is.”
After this introduction, we can get to the heart of the matter. R&D policies followed by government have over the last century been moving inexorably toward rationalist methods. As Polanyi described scientific rationalism, it “would permit us to believe only explicit statements based on tangible data and derived from these by a formal inference, open to repeated testing.”
That might sound like a good basis for a system of R&D – and indeed seems to be exactly what governments have strived for – but it ignores the implications of tacit knowledge across integrative levels. We do not know all aspects of the rules at any level of analysis. No one claims we have a complete description of mathematics or physics, let alone biology. So there are hidden truths yet to be discovered.
Even if we did have a complete description of the rules with zero technical uncertainty, there is still uncertainty of how they should be aggregated to fulfill human requirements which, though they can be expressed in technical measures and performance characteristics, are inadequately described by them.
So explicit analyses of given facts cannot form the basis of choices across the wide range of alternative R&D projects. If it were possible — as believed by defense reformers like Alain Enthoven, Charles Hitch, and Robert McNamara — then a small group of the smartest individuals at the very top would be in the best position to evaluate designs and select the optimal portfolio across the economic enterprise. Indeed, this rationalist system of project choice became established defense policy with the rise of the Planning-Programming-Budgeting System in 1961.
Polanyi, however, recommended a different approach to resource allocation, which essentially amounts project choice:
“It might seem impossible to compare the complex scientific value of marginal contributions over such different areas as, for example, astronomy and medicine. Yet I believe this is in fact done, or at least is reasonably approached in practice. It is done by applying a principle that I have not seen described elsewhere, although it is used in various fields; I would call it the principle of mutual control. It consists, in the present case, of the simple fact that scientists keep watch over each other. Each scientists is both subject to criticism by all others and encouraged by their appreciation of him. This is how scientific opinion is formed, which enforces scientific standards and regulates the distribution of professional opportunities. It is clear that only fellow scientists working in closely related fields are competent to exercise direct authority over each other; but their personal fields will form chains of overlapping neighborhoods extending over the entire range of science…
“All that I have said here about the workings of mutual adjustment and mutual authority… suggests a way by which resources can be rationally distributed between any rival purposes that cannot be valued in terms of money. All cases of public expenditure service collective interests are of this kind. This is, I believe, how the claims of a thousand government departments can be fairly rationally adjudicated, although no single person can know closely more than a tiny fraction of them.”
Polanyi argued that because important aspects of knowledge are tacitly held, such as what new paths of inquiry and technology are likely to prove fruitful, a system of progress cannot do better than letting individuals decide for themselves the experimental trials that they will pursue. Some will result in errors. But, being constrained by what is possible in nature, including the hidden realities, the individual is pushed to find what actually works. Moreover, the opinion and praise of peers also engaged in R&D provides a kind of control that also allows for an openness to revolutionary insights.
The fact that we cannot know what will prove successful at higher system levels using only explicit rules and known facts from the lower levels, and we agree that in any case our view of nature is at best incomplete, means that in order to make advances into the unknown – to make real discoveries that propel us to new understandings – we must accept our ignorance. We must take action based on the conjecture of reasonable individuals, because we cannot know what will turn out to be true ahead of time. The asymmetric upside to new discoveries – increasing effectiveness by 10x or even 1,000x – means that we cannot afford not to take part in these experiments.
Yet defense R&D policy making takes the view that the only projects worth pursuing – those that can be justified to Congress and the public – are those that have a complete technical plan backed by explicit rules and known facts. That is the modern version of oversight. Control is not based on evaluating what R&D work has been done, but on what will be done in the future.
But the reason researchers or technologists chooses a particular path is because they anticipate a fruitful endeavor. The anticipations are based on tacit knowledge, which cannot be adequately conveyed to the bureaucracy. That means small and less meaningful projects are pursued — those less likely to achieve 10x+ returns — because they conform to what is known. Even worse, the compiling of new requirements at each stage of bureaucratic approval means the resulting project is one which no individual anticipates to be correct.
Here is an excerpt from Polanyi which I think will prove enlightening:
“There are two possible ways of viewing the progress made by the front line of scientific discoveries as it advances over a period of time. We may look upon such progress as the growth of thought in the minds of gifted people along the pathways of science. The frequent occurrence of simultaneous discoveries may appear to support this image… This seen, the growth of new ideas appears altogether predetermined…
“Yet, looking forward before the event, the act of discovery appears personal and indeterminate. It starts with the solitary intimations of a problem, of bits and pieces here and there which seem to offer clues to something hidden. They look like fragments of a yet unknown coherent whole. This tentative vision must turn into a personal obsession; for a problem that does not worry us is no problem: there is no drive in it, it does not exist. This obsession, which spurs and guides us, is about something that no one can tell: its content is undefinable, indeterminate, strictly personal. Indeed, the process by which it will be brought to light will be acknowledged as a discovery precisely because it could not have been achieved by any persistence in applying explicit rules to given facts. The true discoverer will be acclaimed for the daring feat of his imagination, which crossed uncharted seas of possible thought…
“Note that there is a widespread opinion that scientists hit on discoveries merely by trying everything as it happens to cross their minds. This opinion follows from an inability to recognize man’s capacity for anticipating the approach of hidden truth. The scientist’s surmises or hunches are the spurs and pointers of his search. They involve high stakes, as hazardous as their prospects are fascinating. The time and money, the prestige and self-confidence gambled away in disappointing guesses will soon exhaust a scientist’s courage and standing. His gropings are weighty decisions…
“His vision of the problem, his obsession with it, and his final leap to discovery are all filled from beginning to end with an obligation to an external objective. In these intensely personal acts, therefore, there is no self-will.”
And so we find how the principle of mutual control allows an R&D program to proceed at its fastest possible pace. The participants themselves have partial information, but it is the best information available as to what might prove fruitful. It includes explicit and tacit information.
Not all projects are tried. Based on their grounding in what is already known, researchers make conjectures about how the particulars can be related differently, or whether there is a new way of casting the rules. The opinion of their peers holds them in check, about the reasonableness of their claims and empirical evidence. But it is also possible to overturn what is already known, and this is indeed the most exciting aspect of engagement – that you can make important contributions to human knowledge.
The society of explorers are allowed to do just that: explore. Imagine if Christopher Columbus, on his speculative endeavor, had to convince not just one monarch willing to finance the effort, but convince all monarchs in Europe. That is analogous to what is required for a new defense R&D project, approval from all the “fiefdoms” across the DOD including requirements, programming, budgeting, contracting, cost, and various other communities.
In times past, there were several Army technical departments and Navy bureaus, each of which had their own chiefs which independently pleaded for funds, and were able to route those funds to projects they found most promising without a defense-wide consensus. It more closely resembled the society of explorers which Polanyi talked about. It allowed for project choices based on “judgment” and “intuition,” two words that came under constant attack after WWII in favor of an idealistic view of science in which there was no such thing as tacit knowledge.
It is ironic that the liberal and democratic United States implemented R&D policies that were very close to the perfectionist ideals of socialism — that central planning is the best form of technical advancement. Having been installed nearly 60 years ago now, it is difficult for professionals working in government funded R&D to recognize just how radical the current system really is.
The Soviet R&D program, as RAND analyst Arthur Alexander reminded us, had many of the features of capitalism absent in defense planning. There were dozens of research institutes, airframe and engine design bureaus, and so forth, which competed against each other to get designs into production and operations. Chiefs who didn’t deliver got replaced. As Stalin was reported to have said:
“… the designer was the one individual who could be held responsible for success or failure, that the designer has the duty of protecting the integrity of his design from the demands of others… The designer must not be at everybody’s beck and call. He has to protest irresponsible demands… It is hard to make a good machine and very easy to spoil it and it is a designer who is responsible.”
There is something the government R&D program can learn from methods in the Soviet Union, France, modern China, nineteenth century England, and so forth. While a society of explorers is an idealism itself — only imperfectly represented in the past — I believe that the the direction is correct. Marginal movements in that direction and away from central planning of the R&D program are important steps toward rejuvinating scientific progress.
Leave a Reply