|
Frequently Asked
Questions
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Why did you do this experiment?
The purpose of this experiment
was to extend current social science research on the impact of racial
diversity on college students. The extant research contains clear and
consistent evidence that racial diversity contributes positively to
students' social and intellectual development across a variety of domains.
These studies, however, do not include designs that incorporate random
assignment of participants, and they typically rely on self-reported outcome
measures. Our study extends the knowledge base by using an experimental
design and a behaviorally-based outcome measure.
-
What were the independent and dependent variables in the
experiment?
There were two dependent
variables, Integrative Complexity and Perceived Novelty. Integrative
Complexity, the primary dependent variable of interest, is a measure of
complex reasoning or thinking developed in the social psychological
literature. Scores of Integrative Complexity are coded by raters from
student essays, and is therefore a "performance-based" measure or a measure
based upon a production task. As a companion to the behavioral data,
Perceived Novelty measures students' perception of how novel the arguments
of the collaborator were and how much the collaborator made the student
think.
There were two primary
independent variables. Racial diversity, the main focus of the study, was
manipulated with the use of either a Black or White research collaborator
who participated in the group discussion with three White participants.
Since theory predicts enhanced integrative complexity with the presence of a
minority opinion, we also manipulated (via the research collaborator)
minority opinion in the group.
We controlled for several
variables in the data analyses. They included university site, discussion
issue, age, gender, and current contact with racially diverse others.
-
What hypotheses were tested in the experiment?
There were four hypotheses
tested with regard to racial diversity.
With regard to Integrative
Complexity:
(1)
In essays written prior to discussion, students in groups with a
Black collaborator will exhibit higher levels of integrative complexity,
relative to those in groups with a White collaborator.
(2)
In essays written after discussion, students in groups with a Black
collaborator will exhibit higher levels of integrative complexity, relative
to those in groups with a White collaborator.
(3)
In the "transfer" essays written on a different topic, students in groups with a Black
collaborator will exhibit higher levels of integrative complexity, relative
to those in groups with a White collaborator.
With regard to Perceived
Novelty:
(4) After discussion,
students will view the collaborator as contributing more to novel
perspectives and as more influential when the collaborator is Black than
when the collaborator is White.
-
Why did you choose Integrative Complexity as the dependent
measure?
We were interested in
measuring outcomes that reflected behavior and relied less on self-reports.
We also preferred a measure that had a been used in the academic,
scientifically-based literature. We consulted broadly with colleagues in
experimental social psychology and discovered one measure being used by
Professor Deborah Gruenfeld of the Stanford Business School, a social
psychologist specializing in small group interactions. Integrative
Complexity, originally developed by Peter Suedfeld and Philip Tetlock, is a
versatile measure that has been used in hundreds of studies. The content of
essays and speeches are coded for integrative complexity along two
dimensions: the extent of differentiation of ideas, and the extent to which
the ideas are integrated using overarching principles. The measure yields
an overall score of integrative complexity, ranging from 1 to 7, and the
scale is independent of specific attitudes toward an idea. The scale has
been used in a variety of applications. One study analyzed Supreme Court
opinions for integrative complexity, and found that unanimous decisions were
less complex than split decisions. Another study found a positive effect
for the presence of members of groups who held minority opinions.
Specifically, those who participated in discussion groups where diverse
opinions were expressed resulted in higher integrative complexity.
Integrative complexity has also been linked with higher grades in college
students. These considerations led us to decide that integrative complexity
would be an ideal measure for our experiment.
-
Why did you expect to find race effects on
Integrative Complexity?
Our expectation is based upon
the theory of minority influence in social psychology. The presence of a
minority opinion stimulates more complex or divergent thinking because an
individual feels they must take diverse views into account in their
reasoning processes. We expected the presence of racial diversity to have a
similar kind of effect in that the presence of racial diversity stimulates
an individual to consider the possibility of difference or deviance within
an intellectual exchange. In the American cultural context, racial
diversity is commonly associated with variance in experiences, aspirations,
values, and attitudes. Individuals responding to racial diversity, then,
are responding to this variance. In our experiment, we scripted the
arguments of the collaborators so that both Black and White collaborators
held the same opinions, and yet race effects were detected. As we
predicted, therefore, race carries with it an additional "stimulant" to
complexity.
We had thought that racial
diversity might interact with opinion diversity such that the effects of
opinion diversity would be stronger when the collaborator was a member of a
different race. We found no such interaction, perhaps indicating that when
members of a different race agreed with participants, that too created
greater complexity.
-
Can you elaborate on the reliability of the Integrative
Complexity measure that was used in the study?
We had six different coders
who read essays. We trained our coders using the methods developed by
Suedfeld, Tetlock and others. In all, the coders scored over 180 pieces of
writing each during the training process. Different sets of three coders
read each of the participants' essays. We preformed a multilevel analysis
that examined how much the coders agreed with one another to determine the
reliability of ratings. We found that the reliabilities were 0.70 for the
pre-discussion measure of integrative complexity and 0.62 for the
post-discussion measure. There are two related reasons why the reliability
of the post-discussion measure is not so high. First, essays were shorter
because some participants only wrote what new thoughts they had as a result
of the discussion. Second, the post-discussion measure is in essence a
change measure and change measures typically have low reliability. However,
it is important to note that despite their typically low reliability, change
measures can nonetheless show experimental effects. For example, the paper
by Overall & Woodward cited in the paper shows that our ability to detect an
existing effect can be maximized in the analysis of change scores when the
reliability of change scores is zero.
-
Why did you use the topics of child labor and death penalty
for your discussions?
Topics were chosen based upon
the degree of racial content in the issue. The primary topic, child labor,
was set in a fictional developing country in Asia and judged to contain very
little racial content. In other words, we believed the topic would not
stimulate the issue of race in an American context. The death penalty was
chosen as a topic without any explicit racial content, but one on which
opinion polls suggest that Blacks and Whites hold opposing positions.
-
How did you come to decide on your sample size?
Several factors entered into
our decision concerning sample size. Among those factors were having enough
power (i.e., the probability of obtaining a statistically significant effect
when the effect actually exists in the population we are studying) and our
budget. We reasoned that if we had a small to moderate effect size of 0.3
(an effect size is a standardized measure of difference between the means of
the populations we are studying), we would need a sample size of 350 to have
an 80% chance of detecting such an effect. Fortunately we were able to
secure funding for a study with about 350 cases. Our final same size was
357.
-
What differences did you find between the three sites:
Stanford, UCLA and Maryland?
In all of our analyses, we
controlled for differences between the three sites because we were not
specifically interested in the ways in which students at the three sites
differed, for example, in Integrative Complexity. However, we did check to
see if the effects we found, for example, for racial diversity, varied
across the different sites and we discovered that they did not (i.e., there
were no interaction effects of any of the independent variables with site).
This suggests that the study was implemented in a consistent way across the
three sites.
-
Did you encounter any problems in conducting the experiment
as planned?
We intended to manipulate
majority and minority opinion by selecting participants who all held the
same opinion with regard to the discussion issue (in the case of child
labor, for example, they were all against it) and then randomly placing them
in groups where the collaborator either shared their position or disagreed
with their position. After reading the discussion prompt, some participants
(15%) took a position that was inconsistent with their opinion on the
screening instrument. The result was that instead of just having groups
where either the collaborator agreed with the three other students or
disagreed with all of them, we also had some groups where the collaborator
agreed with one other student and some groups where the collaborator agreed
with two other students. In other words, we had a continuous condition of
student agreement with the collaborator. The effect on our analysis was to
simply change opinion diversity from a dichotomous variable to a continuous
one. Moreover, our results are essentially the same when we analyze only
groups in which the participants did not take a position different from the
one they had indicated in the screening survey.
-
Why did you report two types of statistical analyses – one
at the level of individuals and one at the level of groups?
The data from the study occur
at different levels. They are at the level of the group, the three persons
who discuss the issue together, and the level of the individual. If there
is a “group effect,” it means that the groups differ from one another in
ways that can not just be explained by the differences between the
individuals in the groups. On the other hand, if there is no true "group
effect" and the only level of importance is the individual, then the
variation between groups can be entirely explained by who is in the group.
We tested for the presence of group effects in all of our analyses. In the
analyses of integrative complexity we found no evidence for a group effect.
However, in the analysis of perceived novelty, there was an effect due to
group. The analysis of this variable, therefore, also included the group
effect in the formal model.
-
Isn’t an experiment unnatural? How can you draw conclusions
from an experiment to the real world?
Any experiment is by
definition artificial because its key characteristic is the isolation,
control, and manipulation of variables. Putting an idea to experimental
test is the most rigorous way to determine if a variable really causes
another variable, what has come to be called "internal validity." In every
area of science, theories that are developed through observations and
experiences are usually eventually tested by using an experiment. In our
case, we believed from the results of other non-experimental studies of
college students that there was sufficient evidence to test experimentally
the hypothesis that racial diversity leads to positive cognitive outcomes.
Despite the necessarily artificiality of an experiment, we worked very hard
to ensure that we tested the hypothesis in a way that matched the context to
which we hope our results will generalize: the university context. We
tried to create an experiment that mimics reality in essential ways. Our
experiment used actual college students attending selective universities,
the exact population to which we wished to generalize our findings. The
students sat in small groups with peers and discussed and wrote about
complex social topics, similar to what they are expected to do in seminars
and sections of lecture courses. The artificiality was introduced by the
fact that the treatment was a single session that lasted one hour, rather
than discussions that take place over the course of an entire academic
term. We would like to have had an experiment in the context of a real
course and observed students over the entire course, but that would have
made the design more complicated and prohibitively expensive.
-
Diversity as represented in your study means Blacks and
Whites. Why did you make that choice?
This was purely a pragmatic
decision. The theory about diversity contributing to cognitive complexity
can be applied to all forms of diversity, but we felt in this first study,
we could only look at one minority group. We choose African-Americans for
several reasons, the most important being that White participants would
immediately know that the person was different from them. We would hope that
future lines of research would extend this work to examine whether the
results can be replicated with other forms of diversity (e.g., adding a
White person to an all Black group).
-
Why did you find an effect for race in the pre-discussion
essay?
We asked our participants to
write their first essay before any discussion had taken place to see whether
there would be an effect simply from the anticipation of participating in a
discussion with a racially diverse group. Our results showed a marginally
significant effect, indicating evidence of the "priming" of complexity, due perhaps,
to the expectation of novelty coincident with racial diversity.
-
Why do you think that current contact with diversity was a
relevant factor in your results?
Contact with racially
different others, based on self-report, had an effect such that those who
had little or no contact showed the biggest gains in Integrative Complexity
after discussion had taken place. Indeed, for those with frequent contact,
there were much smaller differences between those in racially diverse or homogeneous
groups. This speaks directly to the argument for why campus diversity is so
important, in that students who have not been previously exposed to racially
diversity are intellectually challenged by the new social setting.
-
In your results, minority opinion expressed by the
collaborators seemed to exert a larger influence than did race. Does that
compromise your claim that race matters in higher education?
No. With regard to the
importance of race in higher education, our experiment should not be seen as
the sole or decisive source of evidence. The existing evidence clearly
demonstrates beneficial effects of race for students in the domains viewed
as central to the missions of higher education institutions. Our study adds
further evidence supporting that general conclusion. Opinion diversity
could also be seen as important for higher education, and that point is not
disputed here. Rather, in addition to previous work demonstrating the role
of racial diversity in increasing cultural awareness, enhancing social and
intellectual self-confidence, increasing the frequency of discussions about
race and other social issues, and positively impacting community involvement
and other civic participation, our study shows that racial diversity also
contributes to the development of higher-order thinking processes. Opinion
diversity has not been shown to have a similar broad range of effects among
college students.
-
What important questions from your study should social
psychology address?
There is an unfortunate
paucity of research in social psychology on the effects of inter-racial
interactions on cognitive outcomes. Our study indicates that racial diversity,
in classroom-like settings as well as among students' informal contacts, does affect people's
integrative complexity. We would hope that the field of social psychology,
which has often benefited from the study of applied social problems (e.g.,
the study of authoritarianism following World War II), would begin to
explore these phenomena in more detail.
-
What relevance does the experiment have to legal challenges
to affirmative action in college admissions?
The constitutionality of affirmative
action rests essentially on one premise – that racial diversity serves a compelling
interest for higher education. A large body of social science research has
been conducted over the past five or six years with the intention of
determining whether that premise can be substantiated. In the most recent
judgment on affirmative action in undergraduate admissions (Gratz v.
Bollinger & Grutter v. Bollinger), the Supreme Court ruled that the social science
evidence presented a strong rationale for the compelling interest premise.
Critics of the research, however, continue to raise concerns over specific aspects of
how the scientific research was conducted. Our experiment addresses those
concerns directly.
-
What are your qualifications to conduct the study?
The team envelops expertise in
higher education research and experimental psychology. Anthony Lising
Antonio, Mitchell Chang, and Jeffrey Milem all received their training at
the pre-eminent center for higher education research at UCLA. Each of these
three experts has published widely in the area of diversity and higher
education. Kenji Hakuta, David Kenny, and Shana Levin come out of the
tradition in psychology where experiments are the norm. Hakuta is an
experimental psycholinguist by training, and has not conducted research in
higher education, but has extensive experience in bringing empirical
research to bear on important social policy issues. Kenny is an
experimental social psychologist who has expertise in statistics and
research methodology and has conducted substantive work in analyzing dyadic
and small group data. Levin is an experimental social psychologist whose
expertise is in the psychology of race and racism. This combination of
expertise in theory, methodology, and policy makes the team unique.
-
Who funded your study?
The study was funded by
Carnegie Corporation of New York, the Ford Foundation, the William and Flora
Hewlett Foundation, the James Irvine Foundation, and a private donation from
the Richard Parsons Family Foundation.
-
Did you design the study so that it was guaranteed that you
would get results that you found?
While we expected to obtain
results that diversity would lead to greater levels of Integrative
Complexity, one could argue that the results might have been the opposite.
For instance, one might think that Whites might have been uncomfortable with
a Black student in the group and that this anxiety might have interfered
with the processing of information. Additionally, Blacks as outgroup
members to Whites may not have been listened to carefully by White
participants, resulting in no stimulant to complex thinking. Thus, it was
hardly a certainty that we would have found the results that we found.
Moreover, given the short duration of the experiment, the effect might have
been too small to measure. |
