Study on Skin Cream and Gun Control (Title is slightly misleading)

This is decision research.

For this thread, post your answer (or just simply remember it). Resist the urge to google search the answer and resist the urge to change your answer after reading my spoiler text.

After you have come up with your answer, read the below text:

Spoiler:
But what happens if you swap the two labels for the columns?

What happens if you change the labels (but keep the numbers the same) to be about gun control?

This study did that with what I think are hilarious results:

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2319992

For those of you with the time, I recomend reading that study (download it. It is a 37 page pdf and is a good read).

Here is a Youtube video that does a great job of explaining, in simple terms, what exactly is going on:

http://www.youtube.com/watch?v=CfoKor05k1I

Discuss the results and your answer. I think the results of the study would have been much more pronounced if they did a prelim filtering before they selected their sample: they should have gotten 500 people that strongly oppose guns and 500 that strongly supported gun freedom. Then the charts would have shown much more hilarious clustering. 😄

So lets see ~116 people benefited from the cream and ~54 were harmed by it. A product that makes things worse about a third of the time is not something I would be rushing to bring to market.

Originally posted by dadudemon

Spoiler:
But what happens if you swap the two labels for the columns?

Then the cream is makes things qualitatively worse. About 20% of people who do nothing will see worsening while about 33% of people who use the cream will see things get worse.

Originally posted by dadudemon

Spoiler:
What happens if you change the labels (but keep the numbers the same) to be about gun control?

That's a bit vague. There are at least six different readings of that which come to mind, two of which involve skin cream causing gun violence. Which row should be which?

Originally posted by Symmetric Chaos
So lets see ~116 people benefited from the cream and ~54 were harmed by it. A product that makes things worse about a third of the time is not something I would be rushing to bring to market.

That's stood out to me the most, when I first read it. Who in their right mind would use that cream? Unless it was my last hope, I don't think I would be using it any time soon.

While its definitely interesting that people's ability to solve these problems declines in the face of political stakes (especially that even in the 90th percentile only 57% got the right answer) I'm not sure that's the best explanation for the discrepancy between public opinion and scientific data. My experience has been that a lot of people simply think "the other side" is lying or distorting data, interpretation never comes into it.

Also: Woooo! Data reported as probability distributions!

hmmmm, that is an interesting question...

Spoiler:
so, going by proportions, it appears that the cream makes things worse, but I'm not sure that "1/4" of ~300 people is significantly different than "1/6" of ~130 people. I actually forget how to do t-tests for proportions (I may be confusing this with correlation, but I think there is a specific method), especially when dealing with different n's, but here is what I'm thinking (going by what I found here http://www.ltcconline.net/greenl/courses/201/hyptest/hypprob.htm and here http://www.dummies.com/how-to/content/how-to-compare-two-population-proportions.html):

The null for the hypothesis that there is a benefit would be that there is no difference between 223/298 (298 is the n for the first group) and 107/128 (128 is the n for the second group) [n is the number of observations within an experimental group... though I'm fairly sure that is common knowledge]. So, is .75 different than .84, at 298 and 128 observations, respectively...

so... the numerator for the t-test is:

[1] .75 - .84
[2] -.09

the denominator is... ugh...

[1] root(.84(1-.84)(1/298+1/128))
[2] root(.84*.16*.0114)
[3] root(.0015)
[4] .0387

giving us a t-stat of:

[1] -.09/.0387
[2] -2.326

... I don't actually know how to calculate the degrees of freedom in a proportions test with different n's, but even using the lowest n of 128 (making the t-test the most difficult to find significance), -2.326 is significant at an alpha of .05, though maybe not at .01 [one tailed test is borderline at ~100 observaions http://mecholsky.mse.ufl.edu/EMA4714/statisitcs/t-tables.jpg].

I can only assume a similar thing would be found for the "worse" category, given proportions are relative like that. So, even at .01, you would find the cream makes the rash worse. [EDIT: actually, not at .01... .01 for a one-tailed test only reaches significance with a t-score of 2.326 with infinite observations... so it would only be significant at an alpha of .05... A more conservative alpha would find no differences between these groups... **** I wish I knew how to calculate these DFs...]

Isn't degrees of freedom on a contingency table always the number of cells minus 1?

[edit] No, that's wrong. NVM.

most of the time (sometimes it is -2), but when you have different numbers of cells in each group, you have to "pool" them to come up with a single value. That normally involves an equation that uses the variance from both groups. In the case of proportions, variance isn't as easy to determine, at least in a way where it doesn't seem redundant. I'm sure there is a way, I just don't know it. In any case, the pooled DF wouldn't be infinite, and would be higher than 100, so it wouldn't change the results. at .05 it is significant, at .01 it isn't.

Originally posted by Symmetric Chaos
Isn't degrees of freedom on a contingency table always the number of cells minus 1?

[edit] No, that's wrong. NVM.

Yes. Your degrees of freedom are the number of x minus 1.

I made some excel spreadsheets to make t-tests (and even z-tests) much easier to calculate. Just type in your numbers, it figures out your degrees of freedom, look at the results on a chart, type in some more numbers, BAM!

Your results.

I tried to attach the spread sheet but it will not allow it on this forum.

I may be able to zip it up?

I'll try and then edit my post.

Spoiler:
But we are missing the point, lol!

when you have different n's, DF is not n-1...

Does this work?

Edit - It doesn't work. It is missing quite a few formulas and, specifically, the one with the chi-squared test, z-scores, t-scores, etc. That's on my home PC.

it downloads as "attachment.php" for me

Originally posted by Oliver North
it downloads as "attachment.php" for me

Save the php (save target as a zip file), rename the PHP extension to a zip file (if you're using windows, you'll have to change the file type to "all files"😉. Then unzip.

so, in the examples in your excel files, you know the SD, and it looks like it is a single group comparison, so you are right, df = n-1.

when you have more than one group, and they have different numbers of observations, like in the example you posted, it isn't that simple. Generally, you use a Welch's t-test, which is different from a typical Student's t-test, but to calculate DF in Welch's, you need to know the variance. My issues is that I don't know how to calculate variance for a proportion, and it isn't given in the question.

The question you posted is just a more complex statistical test than the ones you are solving in those sheets.

http://en.wikipedia.org/wiki/Student%27s_t-test#Unequal_sample_sizes.2C_equal_variance

Isn't the variance always calculable from the SD?

Originally posted by Symmetric Chaos
Isn't the variance always calculable from the SD?

afaik, yes

what I mean is, if we are just given a proportion (223/298), I don't know how to calculate what the variance of that would be...

If we coded it as 1 = effective, 0 = worse, variance would be redundant with the proportion itself (ie: the proportion IS the variance)... otherwise I'm sort of at a loss...

Like I said, I'm sure there is some simple answer I just don't know. But in any case, using either the most conservative or most liberal DF available (128 or 298), it wouldn't change the end result in terms of significance.

Originally posted by Oliver North
so, in the examples in your excel files, you know the SD, and it looks like it is a single group comparison, so you are right, df = n-1.

when you have more than one group, and they have different numbers of observations, like in the example you posted, it isn't that simple. Generally, you use a Welch's t-test, which is different from a typical Student's t-test, but to calculate DF in Welch's, you need to know the variance. My issues is that I don't know how to calculate variance for a proportion, and it isn't given in the question.

The question you posted is just a more complex statistical test than the ones you are solving in those sheets.

http://en.wikipedia.org/wiki/Student%27s_t-test#Unequal_sample_sizes.2C_equal_variance

Varience for sample proportion is p(1-p)/n.

Uhhh...the one for two population proportions is a bit more complicated:

http://www.socscistatistics.com/tests/ztest/Default.aspx

I think that's what you want.

I'm pretty sure I built that into another spreadsheet, at home.

Originally posted by Oliver North
afaik, yes

what I mean is,

HA!

Originally posted by Oliver North
Like I said, I'm sure there is some simple answer I just don't know. But in any case, using either the most conservative or most liberal DF available (128 or 298), it wouldn't change the end result in terms of significance.

Using the calculator I just posted at .95, there is not significant difference between the two samples. Meaning,

Spoiler:
the cream is not that great or has no effect with that sample.

Originally posted by dadudemon
Using the calculator I just posted at .95, there is not significant difference between the two samples. Meaning,

Spoiler:
the cream is not that great or has no effect with that sample.

its the same calculation I used above

I did a one tailed test though, which is why we got different results

[edit]

So R actually has a function specifically for this (2-sample test for equality of proportions) which outputs df=1 from that table and basically runs a X2 test that ends up with p = .063.

on the study you posted, maybe it is my background, but I'm not sure what the surprise is suppose to be? People's beliefs and expectations impact how they perceive incoming information. In fact, in almost all cases it is hugely beneficial that they do so, as the opposite would be behaving in a world where previous experience had no impact on your current behaviour, meaning organisms wouldn't learn.