BIAS & FORENSICS


Whereas other human activities are dominated by fashions, fads, and personalities; science is supposed to be constrained by agreed rules of procedure, rigorous tests... (and) is an impersonal, dispassionate and thoroughly objective enterprise. 

This is, of course, manifest nonsense.

 

Paul Davies - Director: BEYOND - Center for Fundamental Concepts in Science 

Introduction to Richard   P. Feynman’s Six Easy Pieces: Essentials of Physics

 


For nearly three decades I have been a forensic engineering investigator; at times called an expert witness. Unlike the fight against criminal law breakers by the heroes and heroines portrayed on TV and in the movies, a forensic engineer most often operates in the calm and relatively slow-moving world of civil law. It involves working alongside, and in opposition to, experts in other academic fields – chemists, physicists, and statisticians - in the search for clues and facts to establish [in my case] why and how a material, structure, or product failed in service.

 

Counter-intuitively civil forensics can be more complex than criminal forensics. Criminal law often only requires the ostensibly clear-cut task of “matching” one pattern to another: where evidence collected from both the suspect and the crime scene are compared. Patterns may also be available from a data base (a fingerprint, a DNA sequence, or a facial likeness); developed by tests performed under known conditions (blood splatter patterns, fire spread patterns, characteristic skid marks, marks on a bullet, or even decaying biological matter); or based on data established from past cases. The science and engineering of civil forensics is much more broad-based.

 

The term forensic science, or forensic engineering, refers to the application of science and technology to legal matters. The terms have also come to mean the specialist investigation of any matter that occurred in the past. Discovering how a previous event played out involves three procedures: 1] the observation and analysis of objects as they currently exist - after the event; 2] the simulation of how the event may have played out; and 3] if feasible, the analysis of the information generated while the event was actually happening.  From the combination of all three procedures (ideally) an opinion is formed which is then subjected to critiques by experienced scientific and legal minds. As a consequence one’s ideas and understandings are continually under inspection for their accuracy, logic and rationality.

 

These processes create a passionate need for precision. But they also show the sharp differences between the professions of engineering and research science. While the professions appear to be closely related there are significant and important differences which can influence each at a philosophical level.

 

The priority for engineers is the practical aspects of design, human safety, and property security. The priority for research scientists can be summed up as producing knowledge or discovering “facts”. One profession builds while the other explores. Engineers live in a world of mostly existing data; scientists in a world of finding data; and both operate in an arena where complete knowledge is almost always unattainable. To get around this substantial problem each must make one or more assumptions.

 

Due to its practical nature engineering assumptions can use straightforward tests to establish probabilities on the facts of a matter and determine appropriate degrees of safety and durability. Engineers can readily tell if their work is below standard: parts may not fit together properly; they are not sufficiently stiff or strong; materials age prematurely or cost too much to build; or the client is dissatisfied with the product for other practical reasons.

 

The unfortunate scientist, in pursuit of facts and truth in a complex world, has little or no opportunity for making such assessments. A scientist cannot predict probabilities regarding a complex experiment when the exact conditions have never been previously encountered. The best that can be done is that tests do support the assumptions made; or the work is supported by other scientists - who likely made the same set of assumptions as the original researcher. 

 

The 2013 report by Kassina, Drorb, and Kukucka entitled The Forensic Confirmation Bias: Problems, Perspectives, and Proposed Solutions was highly critical of forensic scientific evidence.  It said: the forensic sciences are subject to contextual bias and fraught with error.  It continued: … context can taint people's perceptions, judgments, and behaviors. … Forensic Confirmation Biases … corrupt lay witness perceptions and memories as well as the judgments of experts in various domains of forensic science.

 

A further example of the difficulties with science was highlighted by Dr. P. A. Ioannidis in his 2005 report Why Most Published Research Findings Are False where he demonstrated statistically that 98% of scientific papers have errors (this report being based upon statistics, and not assumptions, would likely not cause it to fall so far short of the truth). His work also analysed some earlier broadly accepted research findings and found that 32% were subsequently contradicted (16%) or shown to have effects that were smaller than originally determined (16%).

 

To address the human biases in the assumptions that are evidently driving these mistakes a number of straightforward procedures could be instigated for scientific research. Some of these ideas are currently applied within the engineering world:

  • Scientific researchers should be required to be licensed and held responsible for the accuracy and precision of their work. Procedures should be developed to analyse published work along with any educational or public work the scientists perform.
  • A work study specification should be published prior to the start of any research. Any change of specification should be published as an addendum similar to the original specification.
  • The idea of parallel research should be promoted where teams work on identical questions but use different assumptions or paradigms.
  • Scientific reports should be required to include a list of all assumptions made and any anomalies that result. Such a list would provide a reviewer an easier way of assessing the work and the prior knowledge of the researchers. 
  • A study's publication should always disclose the peer reviewers’ names and resumes including any educational or public work they perform.  

 

For more on bias and human reasoning in science see our Topics pages - What is Unbiased Reasoning and How to be Wrong