NBC/WSJ Poll: Oversampling

By Jeralyn, Section Elections 2008
Posted on Wed Mar 26, 2008 at 09:57:28 PM EST
Tags: Hillary Clinton, Barack Obama, 2008 (all tags)

Share This: Digg!

The new NBC/WSJ poll (pdf) out today has Hillary Clinton and Barack Obama tied at 45%. The margin of error is 3.7%. The poll was of all registered voters, not just Democrats. Both Hillary and Obama's unfavorability ratings increased somewhat from two weeks ago. Combining the "somehat "and "very negative" categories, Obama was at 28% unfavorable two weeks ago and is at 32% now. Hillary was at 43% and is now at 48%.

Given that these numbers include Republicans, I'm not surprised. What did surprise me is that the poll says it oversampled African Americans.

In addition, we oversampled African-Americans in order to get a more reliable cross-tab on many of the questions we asked in this poll regarding Sen. Barack Obama's speech on race and overall response to last week's Rev. Jeremiah Wright dustup.

I don't get it. Does anyone have an explanation or think the results are more reliable or less because of the oversampling?

< Bloggers File Fundraising Complaint Against John McCain | Greta Interviews Hillary >

Legal

Nothing on this site should be construed as legal advice. TalkLeft does not give legal advice.

TalkLeft is not responsible for and often disagrees with material posted in the comments section. Read at your own risk.

NBC/WSJ Poll: Oversampling | 51 comments (51 topical)

Readers Must Login or Create an Account to Comment

If they're competent pollsters (5.00 / 5) (#1)
by andgarden on Wed Mar 26, 2008 at 10:02:00 PM EST

they'll have corrected for the over-sampling with weighting. I'm sure they must have.

The reason to over-sample is so that you get a lower margin of error for the subgroups.

Think of it this way: it's very easy to take a big digital picture and make it smaller, but if you try to blow up a small (low resolution) picture, it will look terrible.

Yes (5.00 / 4) (#6)
by Steve M on Wed Mar 26, 2008 at 10:08:15 PM EST

It seems like standard polling procedure to me. The oversampling is specifically to find out what a particular subgroup says; surely they correct for the oversampling with respect to the overall poll sample, unless they're as incompetent as John Zogby.


Wouldn't it increase the overall error? (5.00 / 1) (#61)
by ineedalife on Thu Mar 27, 2008 at 06:54:31 AM EST

After weighting doesn't that essentially convert their 700 person survey to a 602 person survey, and therefore increase the overall error rate? And don't they still have a 7% error rate within the AA population since, after the over-sampling, it is only 175 people? It seems like they burned themselves both ways.


I'm not a poling expert (none / 0) (#62)
by andgarden on Thu Mar 27, 2008 at 07:02:32 AM EST

but I don't see why additional error would be introduced by over-sampling a particular subgroup.


It doesn't (none / 0) (#63)
by tek on Thu Mar 27, 2008 at 07:43:14 AM EST

seems that it would skew the poll because the AAa are all Democrats and all for Obama.


Should (none / 0) (#64)
by tek on Thu Mar 27, 2008 at 07:43:53 AM EST

read: Seems it DOES.


Not if you reweight in the final results (5.00 / 1) (#65)
by andgarden on Thu Mar 27, 2008 at 07:46:14 AM EST

Which is obviously what this poll did.


I Understand Weighting And How That Works (none / 0) (#71)
by flashman on Thu Mar 27, 2008 at 10:39:52 AM EST

But what I don't understand is how they get the "margin of error" result. As I remember it, the error depends on the sample size, and these polls are typically conducted using only 1k interviews, or so. How do they get ~3% error from such a puny sample?


: Margin of error (none / 0) (#79)
by muffie on Thu Mar 27, 2008 at 01:47:43 PM EST

doesn't really depend on the population size, at least when the population is large. If the population of the US was 100 million or 1 billion, the poll would have almost exactly the same margin of error.
Basically, we suppose the popular preference is fixed. We can model this preference as a weighted coin, which lands heads some unknown percentage of the time. If you do 1000 coin flips, you'll have a pretty good idea of the odds of getting heads vs. tails.

oversampling (5.00 / 2) (#4)
by Nasarius on Wed Mar 26, 2008 at 10:06:58 PM EST

I'm not an expert (all I know about opinion polling I learned in analytical chemistry), but I believe the idea is to get a broader base of opinion to start with, but still weight as usual. More data points are usually a good thing.

The upshot is that their data on African Americans might be a little better than usual, but otherwise it's Just Another Poll.

Exactly (none / 0) (#5)
by andgarden on Wed Mar 26, 2008 at 10:08:05 PM EST

by oversampling, you increase the resolution.


Which is interesting because... (none / 0) (#12)
by Maria Garcia on Wed Mar 26, 2008 at 10:17:11 PM EST

...if I am reading the data correctly, African Americans still have a pretty favorable view of Hillary Clinton.


Yup (none / 0) (#13)
by andgarden on Wed Mar 26, 2008 at 10:18:26 PM EST

and 60% for Bill.

Down, I'm sure, but not a catostrophy by any means.

~70%, in the heat of the primary battle, say they'll vote for Hillary over McCain.


But, but but . . . (none / 0) (#31)
by nycstray on Wed Mar 26, 2008 at 11:24:42 PM EST

I thought they were going to leave the party? /snark


The key finding of this poll: Clinton's Negatives (none / 0) (#59)
by sar75 on Thu Mar 27, 2008 at 06:30:55 AM EST

....is that only 37% of respondents are willing to say they have positive feelings about Hillary Clinton, while 48% have a negative feelings. Obama is 49-32 the other way. I wonder why that's not giving some Clinton supporters pause.


What has given me (none / 0) (#68)
by waldenpond on Thu Mar 27, 2008 at 09:58:38 AM EST

the most pause is where 28% of Clinton supporters will not vote for Obama and another 13% will stay home, total: 41%. For Obama supporters not voting for Clinton it is I believe 19 and 9, total: 28%. The medida seemed to focus on that quite a bit yesterday. Even CNN tried the unfavorable stat and other talking heads kept bringing it back to the 28/19 and one kept making the dig of 41%. Some talking heads think Clinton supporters will vote D no matter what. I question that belief as the number unwilling to vote for Obama has steadily grown with successive polls.


Her negs don't always effect her vote (none / 0) (#82)
by nycstray on Thu Mar 27, 2008 at 04:41:23 PM EST

if they did, why is this still a race?

His negs are rising, so it will be interesting to see if his voters stick.

I don't like everything either of them do, but for me (and I suspect for many of her voters), it's who I feel will do the best job on the things I care about once in office. I live under the rule elections are ugly, what's next when it's over.

And I don't care how 'nice' anyone says they'll play. It's never true. Our latest example: Obama.
McCain called him on it yesterday . . .


: A couple of things (none / 0) (#85)
by cal1942 on Thu Mar 27, 2008 at 08:17:45 PM EST

Remember that this poll included Republicans. I wonder how they were weighted. Were they weighted by what the pollster believes to be crossovers or were they just lumped in with Democrats. It seems odd that ANY Republicans would be included in such a poll because even if weighted by some arbitrary crossover factor it would seem that it would be difficult to place that fine a point.
It certainly would be cheaper to poll anyone who answers the phone as opposed to making enough calls to get a sample of Democrats only.
The other THING is that post-Lewinsky Bill Clinton's personal approval (for what it's worth) was well under 50% but his job approval ratings were (as I remember) at 60% and better.

Other info from the WSJ story interesting (5.00 / 2) (#9)
by Cream City on Wed Mar 26, 2008 at 10:09:10 PM EST

"It did indicate that a substantial number of voters question whether the first-term senator would be a safe choice, or whether more needs to be known about him. Mr. McInturff said some voters are wondering, 'Do we know enough about this guy?'

"While the senator's support among Democrats is little changed, he did slip among conservatives and Republican voters, groups that had shown some attraction to Sen. Obama's message of changing partisan politics in Washington. 'I think the survey does indicate that this has taken a little of the patina off Sen. Obama,' Mr. McInturff said."

Yep (none / 0) (#14)
by Lou Grinzo on Wed Mar 26, 2008 at 10:18:28 PM EST

This is something a lot of us have been saying all along: We know a lot about Obama, but many mainstream voters who aren't afflicted with a political addiction don't.

Even at this stage, thanks to people dealing with a staggering economy, higher energy prices, etc., I bet a lot of them from outside of his region, if asked about Obama, would say something like, "Oh, I've heard of him. He's the skinny guy from Ohio. Or Indiana. Iowa? One of those states in the middle of the country. Gives a good speech, but I'm not sure where he is on the issues."

Note that this is not meant to be a criticism of Obama or these voters, just my assessment of the situation. Many people are struggling economically and haven't yet started paying attention to this election cycle.


Little of the patina? (none / 0) (#17)
by RalphB on Wed Mar 26, 2008 at 10:20:19 PM EST

Really, I'm shocked. Do we know enough about this guy? I see all kinds of people saying that Obama doesn't believe anything Rev Wright says. I've seen absolutely nothing that would prove that to me, other than some words. He's such a cypher that I don't know what to believe.


: That's the ticket (2.00 / 1) (#53)
by Prabhata on Thu Mar 27, 2008 at 01:25:09 AM EST

Obama doesn't believe everything Wright says. He picks and choses: Ini Mini Miny Mo.

"has taken a little of the patina off" (none / 0) (#23)
by nycstray on Wed Mar 26, 2008 at 10:34:55 PM EST

but wouldn't that make him more shiny?


Anything from NBC (5.00 / 0) (#70)
by TeresaInSnow2 on Thu Mar 27, 2008 at 10:38:54 AM EST

is suspect, as far as I'm concerned. Anything they do is about pushing the candidate they want.

Less reliable (3.00 / 3) (#2)
by dissenter on Wed Mar 26, 2008 at 10:02:49 PM EST

Anytime you oversample anything. You might as well toss the results and not print it unless there is another agenda going on.

I guess... (5.00 / 2) (#7)
by mbuchel on Wed Mar 26, 2008 at 10:08:22 PM EST

Sorry, dissenter, but the first commenter is dead on. All it does is reduce the error among subgroups. It does not mean that AAs made up 50% of the poll or anything. Keep trying though!


Nonsense (none / 0) (#3)
by andgarden on Wed Mar 26, 2008 at 10:05:06 PM EST

not unless the released unweighted poll results.


Your assumption (none / 0) (#8)
by RalphB on Wed Mar 26, 2008 at 10:09:04 PM EST

considering it's NBC/WSJ may not be well founded.


: The numbers don't seem wildly (none / 0) (#10)
by andgarden on Wed Mar 26, 2008 at 10:10:49 PM EST

out of whack.
As Steve M says, we're not talking about Zogby.

Less reliable. (1.80 / 5) (#11)
by ajain on Wed Mar 26, 2008 at 10:12:33 PM EST

NBC oversampled African Americans to make Sen. Obama look good. They are just too amusing.

7.4% moe (none / 0) (#15)
by Coldblue on Wed Mar 26, 2008 at 10:18:35 PM EST

on African-Americans.

Usually it would be higher, I think (none / 0) (#16)
by andgarden on Wed Mar 26, 2008 at 10:20:05 PM EST

considering that they aren't even 15% of the population.


It was a small sample (none / 0) (#21)
by Coldblue on Wed Mar 26, 2008 at 10:28:09 PM EST

177 AA v. 520 White

700 total = 25% AA


: A sample that's almost double (none / 0) (#22)
by andgarden on Wed Mar 26, 2008 at 10:34:33 PM EST

their actual percentage in the general population.

That's a fairly large MOE (none / 0) (#18)
by Cream City on Wed Mar 26, 2008 at 10:21:07 PM EST

from what I recall of stats class. (I try to forget that class, the stuff of nightmares for me.:-) I was taught that anything over 4% is not reliable.


: no (none / 0) (#26)
by rilkefan on Wed Mar 26, 2008 at 11:04:20 PM EST

Assuming the population is well sampled (a.k.a. the systematic uncertainties in the measurement are small) a large statistical uncertainty is fine - one makes the same test of an n-sigma effect regardless of what sigma is. The calculation of the uncertainty is different for samples on the order of handfuls (Poisson instead of Gaussian), and there are very tricky problems when the estimator can go negative (say you measure something and estimate the background noise and the latter is as large as the former), but this sample is ok for measuring things at the 15% level (assuming again that the sample is representative of the population).

Um (none / 0) (#20)
by Steve M on Wed Mar 26, 2008 at 10:26:04 PM EST

While the oversampling seems to be a non-issue, it seems really weird to me to ask the overall electorate for their opinion on the primary. Obviously the relevant universe ought to be Democratic primary voters.

Not like most commenters will accept it... (none / 0) (#67)
by jr on Thu Mar 27, 2008 at 08:48:51 AM EST

...but Josh has the explanation from NBC over on TalkingPointsMemo right now. http://talkingpointsmemo.com/archives/185845.php

The poll (none / 0) (#72)
by Andy08 on Thu Mar 27, 2008 at 11:10:34 AM EST

is useless. First the oversampling is giving a weighted results. Second, the margin of error
on the oversampled black vote (AA) is of absurd at 7.4%

This is not a statistical poll; this is a political poll: i.e useless.

From AP:

The Journal-NBC poll was conducted March 24-25 and included telephone interviews with 700 registered voters, with an oversampling of 177 black voters. The margin of sampling error was plus or minus 3.7 percentage points for all registered voters, 4.3 points for white voters and 7.4 points for black voters.

ugh (none / 0) (#74)
by mindfulmission on Thu Mar 27, 2008 at 11:35:42 AM EST

The oversampling was done so that they could actually get statistically relevant results on the black sample. Because blacks are such a small portion of the population, it is often difficult to get statistically relevant data.

Now... you are right in saying that 7.4% is somewhat absurd. Unfortunately there isn't much choice in such a sample.

Further... the poll was waited so that the sample of black voters was NOT disproportionate to the normal population.


thank you mm (none / 0) (#78)
by Jeralyn on Thu Mar 27, 2008 at 12:01:36 PM EST

for so quickly taking out the tagline.


: Could you please clarify (none / 0) (#83)
by Andy08 on Thu Mar 27, 2008 at 06:26:36 PM EST

your comments from a statistical analysis view point? What method was it used and what was the
the "proportion" you refer to when you say
"was NOT disproportionate to the normal population"?
(what was the deviation from the normal?)
used ? And also could you explain why was one minority oversampled over another one?

this thread is about the poll (none / 0) (#73)
by Jeralyn on Thu Mar 27, 2008 at 11:35:15 AM EST

and has been cleaned of off-topic insults, race-baiting and accusations.

Actually, (none / 0) (#75)
by cmugirl on Thu Mar 27, 2008 at 11:38:17 AM EST

Isn't what this says is that his "Speech to End All Speeches" had little effect?

I flipped on MSNBC to watch the news at 5:30 this morning (I know, I know - curse my eyes!), and that's even what part of their scroll said.

: not really (none / 0) (#76)
by mindfulmission on Thu Mar 27, 2008 at 11:40:02 AM EST

The issues around Obama/Wright seemed to swing the momentum, and the national polls, over to Clinton, but Obama has recovered and seems to have regained his lead in national polls.
This could imply that the speech helped him recover.

What I get from this poll... (none / 0) (#77)
by sweetthings on Thu Mar 27, 2008 at 11:47:22 AM EST

Is that Clinton is not immune to mudslinging. Her negatives have gone up, significantly, which runs counter to the argument that Hillary has been so thoroughly vetted that there's nothing the right-wing smear machine could do to her.

Fortunately for McCain, the right wing machine doesn't have to do anything but wait. Democrats are willing to do all the heavy lifting themselves.

People should be aware (none / 0) (#80)
by rilkefan on Thu Mar 27, 2008 at 01:59:30 PM EST

that probably every poll oversamples subpopulations. There's an as-far-as-I-know unresolvable methodological question about what to do when one samples say registered voters and finds that there are e.g. rather more Democrats than one expected. Does one reweight to the known ratio of D/R? Does one say there's been a shift and go with it?

Here they've deliberately made a sub-poll of data which can be safely used with the other data by simply multiplying the subtotals by the ratio #_AA_expected_in_total_sample/#_AA_sampled. They probably do this correction routinely using women or people of SES tranche blah instead of AA because they are overrepresented among poll respondents relative to the population.

oversample (none / 0) (#81)
by mindfulmission on Thu Mar 27, 2008 at 03:49:14 PM EST

Hmm...

This might be good for those of you are who are critical of this poll: What "oversample" means

Okay the article clarifies (none / 0) (#84)
by Andy08 on Thu Mar 27, 2008 at 06:31:46 PM EST

what they did; why didn't they clarify this initially in their wide-press reports.
Oversampling may mean different things depending
on what are you doing/looking for/ trying to measure.


i have no idea (none / 0) (#86)
by mindfulmission on Thu Mar 27, 2008 at 09:21:13 PM EST

Maybe it was a poorly written article?

But regardless, I thought it was pretty obvious what they did and how they did it, before reading their explanation.

I do think it is funny that people will attack anything, even without understanding it, just because it isn't as favorable to Hillary Clinton as they would like.


: You are mistaken (none / 0) (#87)
by Andy08 on Thu Mar 27, 2008 at 09:46:21 PM EST

I know statistics pretty well, and it was not clear at all what they were doing from the news article.
They could have been doing something completely different. I couldn't care less about who is benefiting; but it annoyes me to the extreme the manipulation and bad use of statistics, especially for political pollings purposes.
I still have to see what on earth they were trying to achieve by oversampling AA (why ? why not Hispanics?) I couldn't care less about who is benefiting.