scientific discovery is not a one-step process

To counter this universal tendency of the untrained mind to see causal relations and trends where none exist, responsible science requires a very skeptical attitude, which demands cogent evidence for an effect; particularly one which has captured the popular imagination. Thus we can easily understand and sympathize with the orthodox conservatism in accepting new effects. There is another side to this; skepticism can be carried too far. The orthodox bias against a real effect does help to hold irresponsibility in check, but today it is also preventing recognition of effects that are real and important. 

The history of science offers many examples of important discoveries that had their origin in the perception of someone who saw a small unexpected thing in his data, that an orthodox significance test would have dismissed as a random error. Jeffreys (1939, p. 321) notes that there has never been a time in the history of gravitational theory when an orthodox significance test, which takes no note of alternatives, would not have rejected Newton's law and left us with no law at all. Nevertheless, Newton's law did lead to constant improvements in the accuracy of our accounting of the motions of the moon and planets for centuries, and it was only when an alternative (Einstein's law) had been stated fully enough to make very accurate known predictions of its own that a rational person could have thought of abandoning Newton's law. The discovery of argon by Lord Rayleigh and of cosmic rays by Victor Hess are examples that come to mind immediately. Of course, they did not jump to sweeping conclusions from a single observation, as do the disaster-mongers; rather, they used the single surprising observation to motivate a careful investigation that culminated in overwhelming evidence for the new phenomenon. 

It is fortunate that physicists and astronomers do not, in practice, use orthodox significance tests; their own innate common sense is a safer and more powerful reasoning tool. In other fields we must wonder how many important discoveries, particularly in medicine, have been prevented by editorial policies which refuse to publish that necessary first evidence for some effect, because the one data set that the researcher was able to obtain did not quite achieve an arbitrarily imposed significance level in an orthodox test. This could well defeat the whole purpose of scientific publication; for the cumulative evidence of three or four such data sets might have yielded overwhelming evidence for the effect. Yet this evidence may never be found unless the first data set can manage to get published. How can editors recognize that scientific discovery is not a one-step process, but a many step one, without thereby releasing a new avalanche of irresponsible, sensational publicity seekers? The problem is genuinely difficult, and we do not pretend to know the full answer. (p.504ff) Probability Theory, The Logic of Science, E.T.Jaynes