How Data Deceives Us, and Theories Let Us down


The confidence revolution has dramatically improved the quality of economic and political decisions. Hypotheses and theories can now be supported or refuted by data. In a column for GURU, NES Professor Gerhard Toews talks about the data-related opportunities that have recently opened up – and that are not always made use of. 


Since ancient Greece, those interested in understanding the world had the option to use two different methods of reasoning: deductive reasoning and inductive reasoning. Deductive reasoning uses logic to link clearly defined assumptions or self-evident premises to theoretical predictions. Inductive reasoning, on the other hand, relies on repeated observations to gather evidence for an empirical relationship, without necessarily understanding the underlying theoretical mechanism. Both inductive and deductive reasoning contribute to our understanding of the world around us. In fields like economics, it is often emphasized that these two forms of reasoning are highly complementary and should both be present in a good thesis. 

Using only one form of reasoning can often lead to misunderstandings. For example, a naive firm owner might theorize that raising the price of a product will increase profits. However, if the price increase leads to a decrease in demand, the owner will be negatively surprised to learn that while the profit margin increases, the number of products sold decreases leading to ambivalent consequences for overall profits. Thus, the naive theory of the firm’s owner may prove to be wrong.

Similarly, relying solely on inductive reasoning to generalize empirical observations, without considering the underlying theoretical relationships through deductive reasoning, can lead to fundamentally incorrect conclusions. For instance, a student might observe that wearing a specific shirt during exams is associated with better performance and conclude that wearing the shirt enhances academic performance. However, this observation could be coincidental such that repeatedly wearing the shirt may not actually improve grades. 

Nobel Prize winner Daniel Kahneman gives a good example in his book Thinking, Fast and Slow, which shows us how relying solely on empirical evidence can lead to mistakes. Teaching flight instructors in the Israeli Air Force he told them that rewards for improved performance work better than the punishment for mistakes. One of those instructors claimed that his experience refuted this statement. Why was the instructor wrong? He ignored the theory – regression to the mean. Cadets whose results were better or worse than average could fail or improve their next attempt regardless of what the instructor said or did.

Therefore, both inductive and deductive reasoning are necessary to construct sound arguments. However, constructing theoretical and empirical arguments is time consuming and requires some effort. Theoretical arguments necessitate careful consideration of all the assumptions made and adherence to logical rules in order to derive a theoretical prediction. Empirical arguments require the systematic collection of data and thoughtful consideration of the sources of empirical variation, which would allow for the causal interpretation of the data. 


New opportunities for economic research and more

Over the past 20 years, there has been an empirical revolution in economics and the world in general. Technological advancements have enabled the collection of large amounts of high-quality data, while statistical advancements have allowed economists to utilize this data in more sophisticated ways, providing causal evidence for previously unknown or badly understood relationships. This resulted in the fact that making an inductive argument started being relatively less difficult and empirical work started dominating the activity of economic researchers. In the following section, I will provide three examples of how the availability of data has improved the empirical aspect of economics.

At the micro level, advancements in financial transaction technology have facilitated the precise tracking of people's consumption behavior. For instance, analyzing online shopping data that is linked to individual characteristics like gender, age, and education allows economists to investigate factors such as the impact of recommendations or online reviews on consumer choices. Additionally, advancements in neuroscience have enabled the study of the neural processes underlying consumer decision-making using techniques like functional magnetic resonance imaging. This allows researchers to understand how factors such as pricing and branding impact brain activity and influence consumer choices.

At the macro level, technological progress has made remote sensing and its application in mainstream economics more accessible. For example, instead of relying on surveys to calculate Gross Domestic Product, researchers have found that observing economic activity from space using lights as a proxy can provide valuable insights. Regions with greater economic prosperity are more likely to have well-lit streets, shops, and restaurants, which can be captured through satellite imagery at night. This approach, while a proxy, is not influenced by a country's political regime or ability to collect data which influences publicly provided official statistics. Similarly, remote sensing improved our ability to identify the location of informal mines in remote locations which has the potential to improve a government's ability to levy taxes. 

Finally, and going back in time, technological progress in reading and transforming archival documents has simplified the collection of data from historical records. For instance, in the past one would hire an army of research assistants who would manually type numbers from an old document into a spreadsheet which would then be imported into a software of your choice to be analyzed. Now, many softwares exist which can transform a scanned pdf document into a spreadsheet of numbers within a few seconds. Thus, what used to take years or decades can now be accomplished in a matter of months or weeks which greatly facilitated research in economic history and allowed economists to understand and learn from past events.


Analysis improves the quality of economic decisions

While these examples illustrate how micro and macro-level data, both current and historical, can enhance our understanding of the world, the mere collection of data is meaningless without proper analysis. Over the past decade, as data generation has become less costly, there have been significant improvements in the way we analyze data. Economists now emphasize the importance of data variation being driven by exogenous variation, ideally achieved through randomized treatments of groups while leaving a control group untreated. And if such controlled experiments are technically or morally impossible, economists resort to natural experiments in history, where unexpected and indiscriminate events such as natural disasters provide plausibly exogenous variation. 

It is evident that economists should utilize both inductive and deductive reasoning to advice policymakers in a world characterized by radical uncertainty. It is ideal to empirically test sound theories and intuitive predictions before implementing them in real-world scenarios. In the past, due to limited high-quality data and empirical tools, economists relied more heavily on theories when providing policy advice. This approach often resulted in unintended and negative economic consequences. For example, the implementation of price controls, such as setting maximum or minimum prices, may seem like a viable solution to ensure the affordability of essential goods and services. However, empirical evidence has consistently demonstrated that price controls often lead to short-term shortages or surpluses, diminished quality of goods and services, and the emergence of black markets. In the long-run, such measures can discourage investments and hinder innovation.

Fortunately, with the advent of improved data availability and empirical tools, economists now have the ability to test their theories before introducing new policies. However, it is important to note that the benefits of such progress in economics can only enhance the well-being of citizens if policymakers are genuinely committed to maximizing the welfare of the people in the first place.


The views expressed in the section "Opinions" do not necessarily reflect the views or positions of the GURU.