Study on Skin Cream and Gun Control (Title is slightly misleading)

Text-only Version: Click HERE to see this thread with all of the graphics, features, and links.



dadudemon
This is decision research.

For this thread, post your answer (or just simply remember it). Resist the urge to google search the answer and resist the urge to change your answer after reading my spoiler text.

https://www.motherjones.com/files/study_image_1.png





After you have come up with your answer, read the below text:



But what happens if you swap the two labels for the columns?



What happens if you change the labels (but keep the numbers the same) to be about gun control?


This study did that with what I think are hilarious results:

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2319992

For those of you with the time, I recomend reading that study (download it. It is a 37 page pdf and is a good read).

Here is a Youtube video that does a great job of explaining, in simple terms, what exactly is going on:

http://www.youtube.com/watch?v=CfoKor05k1I


Discuss the results and your answer. I think the results of the study would have been much more pronounced if they did a prelim filtering before they selected their sample: they should have gotten 500 people that strongly oppose guns and 500 that strongly supported gun freedom. Then the charts would have shown much more hilarious clustering. big grin

Symmetric Chaos
So lets see ~116 people benefited from the cream and ~54 were harmed by it. A product that makes things worse about a third of the time is not something I would be rushing to bring to market.

Originally posted by dadudemon
But what happens if you swap the two labels for the columns?

Then the cream is makes things qualitatively worse. About 20% of people who do nothing will see worsening while about 33% of people who use the cream will see things get worse.

Originally posted by dadudemon
What happens if you change the labels (but keep the numbers the same) to be about gun control?

That's a bit vague. There are at least six different readings of that which come to mind, two of which involve skin cream causing gun violence. Which row should be which?

dadudemon
Originally posted by Symmetric Chaos
So lets see ~116 people benefited from the cream and ~54 were harmed by it. A product that makes things worse about a third of the time is not something I would be rushing to bring to market.

That's stood out to me the most, when I first read it. Who in their right mind would use that cream? Unless it was my last hope, I don't think I would be using it any time soon.

Symmetric Chaos
While its definitely interesting that people's ability to solve these problems declines in the face of political stakes (especially that even in the 90th percentile only 57% got the right answer) I'm not sure that's the best explanation for the discrepancy between public opinion and scientific data. My experience has been that a lot of people simply think "the other side" is lying or distorting data, interpretation never comes into it.

Also: Woooo! Data reported as probability distributions!

Oliver North
hmmmm, that is an interesting question...

so, going by proportions, it appears that the cream makes things worse, but I'm not sure that "1/4" of ~300 people is significantly different than "1/6" of ~130 people. I actually forget how to do t-tests for proportions (I may be confusing this with correlation, but I think there is a specific method), especially when dealing with different n's, but here is what I'm thinking (going by what I found here http://www.ltcconline.net/greenl/courses/201/hyptest/hypprob.htm and here http://www.dummies.com/how-to/content/how-to-compare-two-population-proportions.html):

The null for the hypothesis that there is a benefit would be that there is no difference between 223/298 (298 is the n for the first group) and 107/128 (128 is the n for the second group) . So, is .75 different than .84, at 298 and 128 observations, respectively...

so... the numerator for the t-test is:

.75 - .84
-.09

the denominator is... ugh...

root(.84(1-.84)(1/298+1/128))
root(.84*.16*.0114)
root(.0015)
.0387

giving us a t-stat of:

-.09/.0387
-2.326

... I don't actually know how to calculate the degrees of freedom in a proportions test with different n's, but even using the lowest n of 128 (making the t-test the most difficult to find significance), -2.326 is significant at an alpha of .05, though maybe not at .01 http://mecholsky.mse.ufl.edu/EMA4714/statisitcs/t-tables.jpg].

I can only assume a similar thing would be found for the "worse" category, given proportions are relative like that. So, even at .01, you would find the cream makes the rash worse.

Symmetric Chaos
Isn't degrees of freedom on a contingency table always the number of cells minus 1?

No, that's wrong. NVM.

Oliver North
most of the time (sometimes it is -2), but when you have different numbers of cells in each group, you have to "pool" them to come up with a single value. That normally involves an equation that uses the variance from both groups. In the case of proportions, variance isn't as easy to determine, at least in a way where it doesn't seem redundant. I'm sure there is a way, I just don't know it. In any case, the pooled DF wouldn't be infinite, and would be higher than 100, so it wouldn't change the results. at .05 it is significant, at .01 it isn't.

dadudemon
Originally posted by Symmetric Chaos
Isn't degrees of freedom on a contingency table always the number of cells minus 1?

No, that's wrong. NVM.

Yes. Your degrees of freedom are the number of x minus 1.

I made some excel spreadsheets to make t-tests (and even z-tests) much easier to calculate. Just type in your numbers, it figures out your degrees of freedom, look at the results on a chart, type in some more numbers, BAM!

Your results.


I tried to attach the spread sheet but it will not allow it on this forum.

I may be able to zip it up?

I'll try and then edit my post.



But we are missing the point, lol!

Oliver North
when you have different n's, DF is not n-1...

dadudemon
Does this work?

Edit - It doesn't work. It is missing quite a few formulas and, specifically, the one with the chi-squared test, z-scores, t-scores, etc. That's on my home PC.

Oliver North
it downloads as "attachment.php" for me

dadudemon
Originally posted by Oliver North
it downloads as "attachment.php" for me

Save the php (save target as a zip file), rename the PHP extension to a zip file (if you're using windows, you'll have to change the file type to "all files"wink. Then unzip.

Oliver North
so, in the examples in your excel files, you know the SD, and it looks like it is a single group comparison, so you are right, df = n-1.

when you have more than one group, and they have different numbers of observations, like in the example you posted, it isn't that simple. Generally, you use a Welch's t-test, which is different from a typical Student's t-test, but to calculate DF in Welch's, you need to know the variance. My issues is that I don't know how to calculate variance for a proportion, and it isn't given in the question.

The question you posted is just a more complex statistical test than the ones you are solving in those sheets.

http://en.wikipedia.org/wiki/Student%27s_t-test#Unequal_sample_sizes.2C_equal_variance

Symmetric Chaos
Isn't the variance always calculable from the SD?

Oliver North
Originally posted by Symmetric Chaos
Isn't the variance always calculable from the SD?

afaik, yes

what I mean is, if we are just given a proportion (223/298), I don't know how to calculate what the variance of that would be...

If we coded it as 1 = effective, 0 = worse, variance would be redundant with the proportion itself (ie: the proportion IS the variance)... otherwise I'm sort of at a loss...

Like I said, I'm sure there is some simple answer I just don't know. But in any case, using either the most conservative or most liberal DF available (128 or 298), it wouldn't change the end result in terms of significance.

dadudemon
Originally posted by Oliver North
so, in the examples in your excel files, you know the SD, and it looks like it is a single group comparison, so you are right, df = n-1.

when you have more than one group, and they have different numbers of observations, like in the example you posted, it isn't that simple. Generally, you use a Welch's t-test, which is different from a typical Student's t-test, but to calculate DF in Welch's, you need to know the variance. My issues is that I don't know how to calculate variance for a proportion, and it isn't given in the question.

The question you posted is just a more complex statistical test than the ones you are solving in those sheets.

http://en.wikipedia.org/wiki/Student%27s_t-test#Unequal_sample_sizes.2C_equal_variance

Varience for sample proportion is p(1-p)/n.


Uhhh...the one for two population proportions is a bit more complicated:

http://www.socscistatistics.com/tests/ztest/Default.aspx

I think that's what you want.

I'm pretty sure I built that into another spreadsheet, at home.

Originally posted by Oliver North
afaik, yes

what I mean is,

HA!

Originally posted by Oliver North
Like I said, I'm sure there is some simple answer I just don't know. But in any case, using either the most conservative or most liberal DF available (128 or 298), it wouldn't change the end result in terms of significance.

Using the calculator I just posted at .95, there is not significant difference between the two samples. Meaning, the cream is not that great or has no effect with that sample.

Oliver North
Originally posted by dadudemon
Using the calculator I just posted at .95, there is not significant difference between the two samples. Meaning, the cream is not that great or has no effect with that sample.

its the same calculation I used above

I did a one tailed test though, which is why we got different results

Symmetric Chaos

Symmetric Chaos
So R actually has a function specifically for this (2-sample test for equality of proportions) which outputs df=1 from that table and basically runs a X2 test that ends up with p = .063.

Oliver North
on the study you posted, maybe it is my background, but I'm not sure what the surprise is suppose to be? People's beliefs and expectations impact how they perceive incoming information. In fact, in almost all cases it is hugely beneficial that they do so, as the opposite would be behaving in a world where previous experience had no impact on your current behaviour, meaning organisms wouldn't learn.

Symmetric Chaos
Originally posted by Oliver North
on the study you posted, maybe it is my background, but I'm not sure what the surprise is suppose to be? People's beliefs and expectations impact how they perceive incoming information. In fact, in almost all cases it is hugely beneficial that they do so, as the opposite would be behaving in a world where previous experience had no impact on your current behaviour, meaning organisms wouldn't learn.

Seems like they're looking specifically to deal with the argument that people are being influenced mainly by simple ignorance, ie the general public doesn't know how to think about the problems at all so the influence of political beliefs might be very small. What they found is that even people who are good at math (which they show matches with performance on the neutral version of the test) are very poor at the political version of the test.

dadudemon
Originally posted by Oliver North
on the study you posted, maybe it is my background, but I'm not sure what the surprise is suppose to be? People's beliefs and expectations impact how they perceive incoming information. In fact, in almost all cases it is hugely beneficial that they do so, as the opposite would be behaving in a world where previous experience had no impact on your current behaviour, meaning organisms wouldn't learn.

I think the study was checking to see if humans would think for themselves rather than using their bias to make an assessment of a situation. Is this called "critical thinking" in your field of study?


IMO, there is nothing natural about comparing 4 sets of data. I don't think "bias" should play a part when critically thinking (please, correct my use if it is wrong) about this data when they run across it. The default should not be to ignore that data.

However, one could argue that selecting the response that best fits that person's idea of "how it should be" is also a measure of what evolution does for default human decision. Selecting the one that does not fit with their "social group" could result in group exclusion.

Would you say that it is possible that selecting the correct answer (when it specifically runs against their political agenda) shows that the person is either ignoring their instinctual behaviors or that they have a new ability (this may be a false dilemma fallacy).

Oliver North
Originally posted by Symmetric Chaos
Seems like they're looking specifically to deal with the argument that people are being influenced mainly by simple ignorance, ie the general public doesn't know how to think about the problems at all so the influence of political beliefs might be very small. What they found is that even people who are good at math (which they show matches with performance on the neutral version of the test) are very poor at the political version of the test.

ah, alright, that makes more sense

Oliver North
Originally posted by dadudemon
I think the study was checking to see if humans would think for themselves rather than using their bias to make an assessment of a situation. Is this called "critical thinking" in your field of study?

how do you think for yourself without being biased by yourself?

that is sort of my point... there is no neutral "self" detached from your past experience that can think about things

Originally posted by dadudemon
IMO, there is nothing natural about comparing 4 sets of data. I don't think "bias" should play a part when critically thinking (please, correct my use if it is wrong) about this data when they run across it. The default should not be to ignore that data.

"bias" is also what allows you to know what the numbers and words you read mean... I'm sympathetic with the idea that people should try to let data speak for itself, but in this case, it took three of us, who are fairly knowledgeable about stats, a page worth of discussion to even figure out the right way to analyze the data, and we got several results depending on what we set alpha to or whether we used one or two tailed tests.

math, especially stuff relating to statistics, is something humans are terrible at, even people with advanced math degrees. for all the people who can calculate the root of 238474593457 in seconds in their head, there aren't many savants of probability.

Originally posted by dadudemon
However, one could argue that selecting the response that best fits that person's idea of "how it should be" is also a measure of what evolution does for default human decision. Selecting the one that does not fit with their "social group" could result in group exclusion.

I don't think they were suggesting people were trying to fit in...

actually, you just fell victim to the very type of "critical thinking bias" you bemoaned above. wink

if anything, it would be closer to cognitive dissonance

Originally posted by dadudemon
Would you say that it is possible that selecting the correct answer (when it specifically runs against their political agenda) shows that the person is either ignoring their instinctual behaviors or that they have a new ability (this may be a false dilemma fallacy).

no

dadudemon

Oliver North
I imagine the issue is, when you ask people a question of a political nature, they no longer look at it in strictly numeric terms. They don't think it is a math question any more.

why some people do or dont, idk

Zampanó
For pooled sample variance, where you assume that there is common variance between the two samples, we have

S_p^2 = (n_1 s_1^2+n_2 s_2^2)/(n_1+n_2-2)

that is, sample pooled variance is equal to the size of sample 1 times the st dev of sample one plus the size of sample 2 times the st dev of sample 2 all over the sum of samples 1 and 2, minus 2

Wolfram alpha code:

http://www.wolframalpha.com/input/? i=%28n_1*s_1%5E2+%2B+n_2*s_2%5E2+%29+%2F+%28n_1+%2
B+n_2+-+2%29

There are better versions of this test, but this is all i have time to fill in for now

dadudemon
I could not find my master spreadsheets at home. I have no clue where they are. But, basically, it takes you through all the formulas (has examples and tables) you would need to be a researcher. I know I gave it to some students. sad

It was probably on my old work PC (changed jobs...old PC had lots of good stuff). Had about 20 different worksheets with multiple formulas, examples, and sometimes images. It was many week's of effort. I am so saddened by this that I may end up sulking. sad

Text-only Version: Click HERE to see this thread with all of the graphics, features, and links.