Can we speed up scientific progress?

Scientific progress appears exponential but is made up of smaller paradigm shifts, with major discoveries becoming rarer as fields mature. True innovation faces challenges in gaining attention...

Can we speed up scientific progress?

In 1874, Lord Kelvin observed:

Scientific wealth tends to accumulate according to the law of compound interest. Every addition to knowledge of the properties of matter supplies [the physical scientist] with new instrumental means for discovering and interpreting phenomena of nature, which in their turn afford foundations of fresh generalisations, bringing gains of permanent value into the great storehouse of [natural] philosophy.

If Kelvin’s statements had sufficiently described the accumulation of scientific wealth, we would expect to see continuous exponential progress.

Fig.1 Exponential Growth

However, progress only superficially appears to proceed exponentially. Indeed, on closer observation, it is actually made up of many smaller paradigm shifts, with each shift creating an entirely new field.

Such a breakthrough initially facilitates many significant follow-on discoveries, but as the new field matures, scientists are able to make successively fewer and fewer major discoveries within it.

One such breakthrough was the Germ theory of disease, which states that some diseases are caused by microorganisms. Girolamo Fracastoro was one of the first to formulate the theory back in 1546, but it took hundreds of years for the theory to gain acceptance.

Countless versions of the theory were formulated over the centuries. For example, Ignaz Semmelweis, a Hungarian obstetrician working at the Vienna General Hospital, provided evidence that washing hands reduced mortality from childbirth from 18% to 2.2% in his hospital. Semmelweis’s observations conflicted with the established scientific and medical opinions of the time, and some doctors were offended at the suggestion that they should wash their hands. Semmelweis lost his job, was pronounced insane, and died of septicemia (bacterial infection) in an insane asylum only 14 days after being committed.

By 1880 Pasteur and Koch, both already established and well respected scientists, finally provided evidence considered substantial enough for the germ theory to gain acceptance. It was then that a golden age of bacteriology started. We illustrate this in the logistic S-Curve below.

Fig.2 S-Curve of a new paradigm

Suddenly huge amounts of resources and manpower began to flood into the field, and between 1879 and 1889, German microbiologists isolated the organisms that cause cholera, typhoid fever, diphtheria, pneumonia, tetanus, meningitis, gonorrhea, as well the staphylococcus and streptococcus organisms. But while the initial discovery was a revolutionary paradigm shift that started a golden age, many further discoveries — at least initially — were merely incremental additions to human knowledge. All the low-hanging fruit quickly had quickly been harvested.

Exponential progress requires exponential innovation.
By 1890 Viruses were discovered.

Fig.3 Multiple S-Curves overlayed

It’s not enough for a new insight to occur every once in a while. This would lead to merely linear progress. Truly fundamental insights open up new avenues for research.

We believe that we’re living in times of ever-increasing faster exponential progress. But there are those who believe that in many fields this progress has been slowing. 

For a new idea to succeed on the market of ideas, an inventor or scientist needs to first achieve a significant result, then draw sufficient attention to the result, and finally either convince the established cabal in their field of the validity of the result, or alternatively decide to sidestep the academic process altogether, and directly, entrepreneurially, implement their discovery in the form of a product or service.

Depending on the type of project, the inventor might require funds, or even institutional permission, to develop and test their hypothesis and generate sufficient evidence to move forward. But when it comes to funding, in the last decades, radical experiments have been increasingly stymied in favor of safe, predictable, incremental improvements. Institutionalism, centralization, standardization, and risk-averseness have homogenized the kind of research that can get funded. 

SMBC Comics #2866

True innovation is by definition difficult to recognize. Everything that is obvious has already been discovered. New ideas require rethinking and challenging what we believe to be true and absolute. It is tough for outsiders - and most innovators are outsiders - to garner the level of attention they need to get their ideas funded and accepted. 

Only after an outsider has succeeded do we begin to admire their way of thinking, and search for people who fit the same superficial pattern — looks, thinking style, education, personal history, type of approach —to begin the next wave of innovation. However, the previous generation of rebels often are not the ones who can recognize the significance of the next. By the very nature of innovation, every new wave will be unlike the previous one. Truly new ideas always look unworkable, boring, crazy, and absurd.

It does not help that those invested in the dominant paradigms form cartels that try to keep out upstarts and alternative views. To name but a few examples: If during the last 30 or so years you had, worked on quantum gravity, but did not believe in string theory, or researched vaccines, but did not believe subunit vaccines were the only viable approach, or were an oncologist, but did believe immunotherapy was a valid approach, you’d have struggled to find funding, or even a faculty position.

Not only do the dominant thought collectives at each university often actively conspire to keep out alternative form of inquiry on a local level, but given the invention of the internet and the emergence of globalized citation rankings, it has become international career-suicide to work on anything but the dominant theories in many scientific fields. We’ve traded parallel lines of inquiry for having to look good to as many of our peers as possible.

As discussed at the beginning of this essay, incremental improvements to dominant theories are unlikely to yield the kind of results we do require for sustained exponential progress. It’s only when we challenge the status quo that we can truly move humanity forward. We’re wasting valuable resources by condemning our best and brightest to dig in mines that have already been depleted.

The few that do succeed in bringing us true innovation must serve two masters: Being intellectually far enough outside the group consensus to discover original ideas, but skilled enough as a courtier to attract funding and talent to their radical vision. The kind of innovator that can truly shine both as an independent thinker, and at a court that systematically punishes divergence from group think, is rare indeed. If we want to see more innovation and faster progress, we’ll need to find ways to bridge that gap and make it a goal to have many parallel competing lines of inquiry pursued.

Paul Stefan Bohm (follow me on twitter) writes about self-organization in Computers, Biology and Society.

Thanks to Kumar Thangudu, Christine Peterson, Kai Peter Chang, and Silver Keskküla, Michael Solana, and Geoff Lewis for reading a drafts of this article and providing feedback.