Why You Shouldn't Conclude "No Effect" from Statistically Insignificant Slopes (repost)

The following is not a Stats Make Me Cry original, but rather something I came across and found very interesting. If you are interested in the topic, please read the preview and follow the link that follows to the original site.

It is quite common in political science for researchers to run statistical models, find that a coefficient for a variable is not statistically significant, and then claim that the variable "has no effect." This is equivalent to proposing a research hypothesis, failing to reject the null, and then claiming that the null hypothesis is true (or discussing results as though the null hypothesis is true). This is a terrible idea. Even if you believe the null, you shouldn't use p > 0.05 as evidence for your claim. In this post, I illustrate why.

To demonstrate why analysts should not conclude "no effect" from insignificant coefficients, I return to a debate waged over blogs and Twitter about a NYT article. See Seth Masket's original take, my response, and Seth's recasting. The data come from Nate Silver's post, which adopts a more nuanced position that I think is appropriate in light of the data.

Read the rest of Carlisle's article here...

9 Comments /Source

Jeremy J. Taylor

Stats Make Me Cry is owned and operated by Jeremy J. Taylor, Ph.D. Jeremy completed his doctoral training in Clinical Psychology at DePaul University and completed his pre-doctoral internship at the Kennedy Krieger Institute, Johns Hopkins School of Medicine. He is currently a Senior Research Associate at the Collaborative for Academic, Social, and Emotional Learning. Although Jeremy's background is in Psychology, he consulted on dissertations for more than 100 students, from 13 countries, and from a variety of disciplines.