Slides

Harnessing Complexity
Analysis methodology and ethical framework
to facilitate utilization of video data in evaluations
Kurt Wilson, Ph.D.
September 17, 2014
Outline
1. Context
Complexity and visual culture
2. Study Design
Process, problem and literature review
3. Overview of 3-Paper Dissertation
Data, design, conclusions
4. Discussion
Questions, next steps
Motivating Domain: Complexity
International Development
Teen Pregnancy
Education
Faith-Based Nonprofits
Hospice
Adoption
Environmental Care
Homelessness
Addiction Recovery
Foster Care
Urban Renewal
Path to the Proposal
Big Question: Complexity
Visual Data:
Exceptionally Rich
Analysis of
User-Generated
Video Data
3 Paper Dissertation
Context: Visual Culture
Long History
• Photography was invented in 1798 (Gernsheim, 1986)
• Edison patented the movie camera in 1888 (Green, 2013)
VERY Broad Participation
• Over 1 billion camera phones used worldwide by
2010 (The Economist, 2010)
• 300 million photos uploaded to Facebook every day,
viewed at a rate of several million a second (Madrigal, 2013)
• YouTube visitors view 6 billion hours/month and upload
100 hours/minute (YouTube, 2013)
Context: Overall Field
Data Visualization
Visual Data
Before
After
Video
Study Problem
Underutilization of visual data
• Rich data source is lost
• Cultural relevance is threatened
“Whilst texts are associated with reason and
higher mental faculties, images are seen as
subversive, dangerous and visceral.”
(Emmison & Smith, 2000, p. 14)
Approach to the Problem
Three studies designed to address some of
the reasons for underutilization:
1.
2.
3.
4.
5.
6.
7.
Not appropriate for specific project
No awareness of visual methods
Expense of analysis
Time for analysis
Fear of ethical complexity
Lack of equipment
Etc.
New Understanding of Problem
Client Planning Process:
Eval Terms of
Reference (TOR)
Familiar Data
(Documents,
surveys,
interviews)
Eval Goals
Practical Data
(Docuents,
surveys,
interviews)
Analysis &
Conclusions
Evaluation Planning Process:
TOR / Design
Literature Review: Underutilization
Keyword
AJE
NDE
JMDE
ER
Video
63
28
7
42
Photo
22
0
1
16
"Video Analysis"
0
1
0
0
"Video data"
1
1
1
0
"Photo-elicitation"
1
0
0
0
"Visual data"
1
0
2
3
"Visual methods"
1
2
0
2
"Visual survey"
0
0
0
0
"Image use"
0
0
4
0
Crowdsource
0
0
0
0
Crowd-sourced
1
0
1
0
Papers 1 & 2
Linked Pair
• Address sequential steps of analysis
• Same user-generated product review videos
• Pilot-test new analysis method using crowd-sourcing
Amazon’s ‘Mechanical Turk’ (Mturk)
• Crowd-source platform
• 500,000 people registered to work on simple tasks
• MANY applications for evaluators
• Continues work started by Dr. Azzam
Paper 1: Crowd-Sourced Open Coding
How does crowd-sourced open coding of video data
compare with coding provided by trained evaluators?
Cellphone
ipod Touch
Board Game
Moisturizer
Toy Helicopter
Primary Data
143 surveys with descriptive open codes of video data
Instrumentation
5 internet surveys distributed through Mturk & Email
Participants
*Treatment: 106 Mturk participants (25/video with
some completing multiple surveys)
*Control: “Typical” qualitative team (3 evaluators)
Qualitative content analysis
Analysis
Paper 1: Conclusions
1. Mturk workers provided higher average number of
descriptive words
• Take job seriously, provide good quantity of
analytic data
2. Evaluator coding was professional and evaluative;
Mturk coding was descriptive and sometimes crass.
3. Harnessing the crowd provides new method to
address reality that “the researcher is the
instrument.” (Marshall & Rossman, 2011, p. 112)
Paper 2: Crowd-sourced categorization coding
What is the cost, speed, and accuracy of crowd-sourced
categorization coding of video data? How do different
questions and size cohorts impact results?
Cellphone
ipod Touch
Board Game
Moisturizer
Toy Helicopter
Primary Data
169 total surveys with categorization codes of video data
Instrumentation
5 internet surveys distributed through Mturk
Participants
103 Mturk participants hired in 3 Cohorts – 3, 10, 20
Analysis
Fidelity analysis (percentage) & Pearson’s r coefficient
Paper 2: Conclusions
1. Crowd-sourcing can effectively address both time
and cost concerns related to analysis of video data.
• Time: 15 min. for 3 coder cohort! 4 hrs. for others
• Cost: Average of .$71/video
2. Nearly 100% reliability in simple / clear
categorization (e.g., gender, obvious video quality).
3. More study is needed to refine the method for
reliable coding from diverse categories.
4. No evidence of subjective coding bias related to
personal relevance of video content
Paper 3: Informing ethical visual methods
What guides the use of images in entertainment,
journalism, advertising and the internet? Can broad
principles be identified to provide a useful reference for
guiding ethical decisions?
Primary Data
Intentionally sampled image use forms & procedures
Instrumentation
None
Participants
None
Analysis
Content analysis; two stages of coding and inductive
analysis of underlying assumptions
Paper 3: Conclusions
1. Visuals are primary communication vehicle;
words/numbers are support. (Title slides, subtitles,
logo/phone number…)
2. Don’t force visual data through ethical approaches
designed for words/numbers (i.e. anonymity).
3. Focus on maintaining integrity of relationship with
participants, sensitive to the context. Established
norms can provide useful reference.
Discussion
• Questions?
• Dissertation topic ideas?
• Other evaluation applications for Mturk, or
user-generated data?
• Follow Up
Kurt Wilson
(616) 446-9678
[email protected]

Download Report