Tracking the Progress of E-Mail Vs. Snail-Mail

Tracking the Progress of
E-Mail Vs. Snail-Mail
Gap narrows on response rates, but
applications still limited.
By Duane Bachmann, John Elfrink, and Gary Vazzana
Advances in computer technology and the increased popularity of electronicmail applications have enhanced the potential for conducting survey research
through e-mail. In an experiment comparing mail and e-mail data collection, email fared well with respect to response rates, item omission, response time,
and data quality. The authors make a case for using e-mail to conduct research,
but only under specific circumstances.
T
he widespread use of personal computers in tbe workplace and at home, the
opening of the Internet to tbe general
public, and tbe dispersion of software
and hardware that allows linking of
computers all are contributing to the feasibility of
electronic mail as a survey mode,
A 1995 Gallup poll found that uine out of 10
Eoriune 1,000 companies and balf of smaller businesses now run e-mail applications on networks.
The study also projects the number of e-mail users
to reaeh 38.9 million by 1998.
Although increased use of e-mail in tbe workplace seems apparent, tbe New York Times reports
tbat home access and usage is growing even more
rapidly. A 1995 survey of Internet users found that
most people are tapping into tbe World Wide Web
and commercial on-line services from bome ratber
tban from tbe office or school. And, according to
tbe "Information and Interactive Services Report,"
approximately 14,000 new consumers are linking
up to the Internet daily.
E-MAIL RESEARCH
E-mail questionnaires appeared almost simultaneously witb tbe introduction of e-mail and tbe
opening of tbe Internet. Intra-company networking
allows managers to gather data from fellow
employees for in-house purposes through e-mail.
And participants of on-line discussion groups regularly receive requests for survey responses on topics of interest to the groups. However, despite tbe
increasing use of e-mail surveys, very few
researchers bave investigated tbis new form of data
collection.
Tbe earliest studies of e-mail surveys were
restricted to populations sampled from within a
single company or university. For example, AT&T
conducted an e-mail survey of its international
MARKETING RESEARCH:
b, Vol, 8 No. 2 3 1
employees in 1990 and obtained a 60% response use of e-mail to conduct surveys, its use is approrate, and Lotus Development Corp, achieved a priate for certain specialized populations.
56% response rate to a 1994 employee survey.
In 1986, Sara Kiesler and Lee Sproull surveyed Research Methodology
students and employees in a university setting who
We developed a survey requesting information
worked in a computer-intensive environment. from business scbool deans and division cbairperTbese researcbers obtained a 67% response rate and sons on tbe use of total quality management
found that respondents were more likely to give less (TQM)-type processes in higher education. We also
socially desirable responses tban were those asked tbe subjects to judge tbe usefulness of TQM
responding on paper, Tbe e-mail subjects also left processes in the university and business environfewer questions blank.
ment and tbe extent of its usage in tbe future.
In 1992, Barbara Scbuldt and Jeff Totten surveyed
The topic is an important and timely issue in
a national sample of marketing and MIS faculty education, business, and industry, so we anticipatmembers, comparing mail and e-mail metbods of ed a response rate tbat would provide a good test of
delivery. The mail response rate was 56.5% while the tbe effectiveness of eacb of tbe two different
e-mail rate was 19.3%. Tbe e-mail respondents were research methodologies.
primarily MIS professors wbo bad a greater interest in
We used a combination of several sampling
and familiarity witb the technology involved.
frames—including tbe 1994-95 Membership Directory
The amount and quality of e-mail survey research, for the American A.ssembly of Collegiate Schools of
especially outside of a corporate or university setting, Bminess (AACSB), the 1995 Hasselback Accounting
is very limited. Trying to generalize from studies that Faculty Directory, and tbe ]994 Hasselback Directory
of Management Faculty-—to obtain a representative
use sucb restricted sample frames would be unwise.
sample of business leaders in academic institutions of
higher
education throughout tbe United States.
BREAKING NEW GROUND
All 224 individuals with e-mail addresses pubWe wanted to expand the investigation of e-mail lished in the sample sources were sent questionas a mode of gathering data for research to include naires via the Internet. From tbe remaining deaus
uot only response rates, but also specific item and divisions cbairs, we randomly selected another
response, response time, cost, item omission, and 224 to receive the mail survey.
tendency to respond to open-ended questions. Tbe
The survey was two typewritten pages in length
overall goal of the study we recently conducted was and dealt with various TQM issues. The questions
to determine if e-mail survey techniques can be were primarily category scaled and related to the posused in place of traditional mail metbods.
sible use of 28 different tools, concepts, or applicaDue to newness of tbis area of study, rather tban tions. One demographic question and tbree semantic
test specific bypotbeses, we attempted to answer a differential questions relating to either tbe value or
expected use of TQM were included in the survey.
series of researcb questions:
Those in the mail sample were also asked if they had
an e-mail address.
• Will e-mail surveys yield results similar to
mailed surveys?
A pretest demonstrated tbat respondents should
bave few problems understanding tbe questions.
However, it became very clear tbat many did not
• Do distinct advantages and disadvantages exist
understand bow to return tbe survey by e-mail, so
for tbe e-mail metbod relative tbe mail method?
the instructions were carefully reworked.
• Do different attitudes exist between e-mail and
An important objective of tbis study was to determail questionnaire respondents and uonremine why tbose who were sent tbe TQM survey either
spondents concerning surveys? Is there a difdid or did not return the questionnaire. Approximately
ference in tbeir likelihood of responding?
five weeks after sending the initial survey, tbose
responding to eitber tbe mail or tbe e-mail survey
Our analysis provided evidence that tbe two received a mail survey designed to measure tbe Ukelimodes are comparable in tbe data gathered and bood of tbe respondent returning: (1) a mail survey on
response rates obtained. Tbe e-mail mode, bowev- sometbing in whicb tbey were interested, (2) a mail
er, bas some distinct advantages and some materi- survey on sometbing in which tbey were not interestal disadvantages. Respondents gave comparable ed, (3) an e-mail survey on sometbing in wbich tbey
responses regardless of tbe mode of delivery. were interested, or (4) an e-mail survey on something
Althougb tbe mail survey response rate was superi- in which they were not interested.
or to the e-mail group, botb rates compared favorTbis follow-up survey also asked respondents to
ably to bistorical results from mail surveys.
report the number of mail and e-mail sitrveys tbey bad
Altbough results do not support tbe widespread received over tbe past year and wbat percentage tbey
32
SUMMER 19%, Vol. 8 No, 2
MARKETING RESEARCH:
bad returned. Additional issues explored were tbe individual's level of knowledge about and use of e-tnail.
Individuals wbo did not respond to eitber the
mail or e-mail TQM sur\'ey received a follow-up
mail questionnaire similar to tbe one sent to respondents. However, tbey also were asked to evaluate tbe
extent to wbicb seven different factors contributed to
tbeir decision not to return tbe TQM survey.
Results
Research question 1: To determine if an e-mail
survey could be used as a surrogate for traditional
mail surveys, we tested for significant differences
in the responses of tbe two groups, T-tests were run
on three Likert scaled questions to determine if a
difference in the means existed, and no differences
were found at tbe ,05 level.
Cbi-square tests were run on 28 yes-or-no-type
questions to test for differences in cell assignments.
One question sbowed a significant difference
between tbe mail and e-mail respondents at tbe ,05
level. Given tbe number of items (28) included in
tbe analysis, a significant difference in one question can be expected due to chance.
Tests of differences were also run for accreditation status, number of faculty, and size of tbe universities for tbe two groups (see Exbibit 1).
Althougb the two groups of respondents appear to
bave come from institutions and business areas of
similar size, the ratio of AACSB-accredited
schools was higher among the e-mail group
(64.6%) than among the mail group (48.3%). Tbis
difference can largely be explained by tbe difference in original samples; 50% of the deans selected for tbe regular mail survey and 63.4% of those
selected for e-mail were from accredited schools.
Survey response rates are an itnportant factor in
judging the quality of a questionnaire-based researcb
project. A strong response rate lessens a researcber's
concern about possible nonresponse bias,
Altbougb tbe mail response rate (65.6%) in our
study was higher tban tbe e-mail rate (52.5%), botb
metbods of delivery resulted in very respectable
response rates for survey researcb. The reported
response rates bave been adjusted for non-deliverable
addresses. Of tbe original 244 surveys sent, one of tbe
tnail surveys was returned because of a bad address,
but 41 of tbe e-mail sur\eys were undeliverable.
Also, considering tbe results of tbe follow-up
survey, a furtber upward adjustment in tbe e-mail
response rate migbt be warranted. Wben nonrespondents were asked wby tbey did not return our
survey. 13 nonresponding e-mail subjects said they
did not remember receiving tbe survey. Althougb
the technology indicated to us that tbose e-mail
questionnaires bad been delivered, apparently tbey
went astray. Assuming tbat tbe 13 subjects would
bave returned the questionnaire (they did return the
follow-up) would bike tbe e-mail response rate to
59.6%, mucb closer to tbe mail response rate.
Tbe above analysis strongly supports our contention tbat, within certain populations, e-mail surveys
can be substituted for mail questionnaires. Witb only
one question of the 31 nondemographic items from tbe
survey having significantly different responses
between the two groups, we conclude tbat virtually no
difference exists in the data gathered by the two metbods. Tbe demograpbic data concerning tbe size of tbe
institution and number of faculty also indicate tbat
respondents fi'om the two groups were similar.
Research question 2: Exbibit 1 also contains survey items tbat indicate advantages of tbe e-mail
mode of data collection. Costs, willingness to
respond to open-ended questions, and response times
were all superior in tbe e-tiiail group. The respondents' tendency to leave an item blank or comment to
questions not requesting a comment were about tbe
same wben comparing tbe two groups.
In assessing the cost of obtaining a response, we
omitted the labor and computer costs. Labor costs
were difficult to define and track accurately. Many
individuals were involved in developing and pretesting tbe questionnaires, assembling tbe samples,
stuffing tbe envelopes, locating e-mail addresses,
etc. Many of these tasks overlapped tbe two groups,
causing cost allocations concerns. The cost of
accessing the Internet is a capital expenditure that
was not passed on to us. Tbe only incremental costs
clearly assignable to a specific sample group were
tbe mailing and stationery costs for tbe mail group.
One open-ended question requesting additional
Exhibit 1
Mean results of selected survey items
Enrollment
Number of faculty
AACSB accredited
Response rates
"Cost/respondent
Responded to openended question
Days to respond
Average items omitted
Commented to multiplechoice questions
Non-deliverable addresses
Have e-mail address
E-mail
Mail
11,641
55.6
64.6%
52.5%
10,631
51.6
48,3%*
65,6%'
$1,53
4,8%-
0
21.9%
4,68 days
3.1%
22.9%
19.1%
100%
11,18 days
.7%
25.2%
,4%'
85,6%
signiticant at the .05 level
' excludes labor costs and capital expenditures for internet access
MARKETING RESEARCH:
SUMMER 1996, Vol. 8 No, 2
33
Exhibit 2
E-mail vs. mail cumulative percent return
% Returned
70
E-mail
Mail
comments was included in the questionnaire. Tbe
e-mail respondents were much more willing to pass
on additional comments tban were mail respondents (21.9% vs. 4.8%). Tbe ability to type a quick
response migbt be an advantage over a mail survey
requiring a band-written response.
Exhibits 1 and 2 clearly indicate tbe responsetime advantage of tbe e-mail survey: 6.5 days faster
tban tbe mail on average. Tbis obviously reflects
the in-transit time differences between tbe two
modes. Exhibit 2 also shows a very rapid rise in tbe
cumulative percent of e-mail responses received.
More than balf of tbe total responses (not cumulative responses) iu tbe e-mail group arrived witbin
two days, and more than 80% witbin one week.
Two areas of concern to survey researcbers are
item nonresponse and additional information
passed on tbrough comments to limited response
questions. Very few items were left blank in eitber
the e-mail or mail surveys, making an inference
from our study difficult relative to item nonresponse. Approximately 25% of tbe respondents in
botb groups added additional comments to at least
one question for wbicb none was requested.
Our project revealed one major disadvantage for
tbe e-mail mode of survey delivery: Non-deliverable
addresses were mucb more common. Of the original
e-mail satnple, 43 out of 224 addresses were not useable. The cuixent technology requires exact addresses for delivery, and tbe smallest error will result in
nondelivery, hnprovements in e-mail software will
overcome this problem to some extent.
Researeh question 3: To determine reasons for
response or nonresponse to eitber mode of data collection, a second survey was mailed to all the sub34
SUMMER 1996, Vol. 8 No, 2
MARKETING RESEARCH:
jects wbo received the TQM que,stionnaire. The follow-up surveys were sent via regular mail, and tbe
overall response rate was 34.5%. The respon.se rates
for e-mail and mail respondents were 56.3% and
44.2%, respectively. The e-mail and tnail nonrespondent rates were low, 18.8% and 6.6%, respectively. Tbis was not unexpected, but we could not
include their responses in our analysis. The problem
witb the nonrespon,se group was further complicated by the fact that only tbree e-mail nonrespondeuts
remembered receiving tbe questionnaire.
The results of botb tbe TQM and follow-up studies support tbe contention that the availability and
use of e-mail is widespread among business scbool
deans and chairpersons. More tban 85% of the mail
sample subjects bad e-mail addresses. Exbibit 3
sbows tbat both sample groups' respondents believe
tbey are knowledgeable about the use of e-mail.
Subjects also are more likely to respond to surveys
concerning topics in wbicb tbey bave interest, a result
tbat agrees witb past survey researcb. However, our
follow-up survey found significant differences
between tbe attitudes of tbe e-mail and mail group
witb respect to tbeir wilbngness to respond to an email survey. The number of e-mail surveys received
(after adjusting for our survey) and the percentage
returned by the two groups were also significandy different. This migbt be due to limited availability of tbe
mail groups' e-mail addresses to researcbers.
STRENGTHS AND WEAKNESSES
Our project supported several potential advantages of tbe e-mail survey, including cost savings
in postage and stationery, reduction iu retum lead
time, and willingness of subjects to respond to
open-ended questions. A major disadvantage was
noted aud measured dealing witb the bigh percentage of non-deliverable surveys iu the e-mail group.
As a result of our experience in tbis e-mail project, we observed but did not measure some additional advantages, such as:
• The ability of subjects to ask questions about
tbe survey easily by using the reply function
available in e-mail.
• Tbe virtually instant feedback on problem email addresses.
• Respondent flexibility in returning the questionnaire, i.e., via e-mail, mail, or fax.
Included in tbe observed disadvantages are
some unanswered questions: Why did 13 of tbe email respondents not remember receiving tbeir surveys? Wby did the mail respondents indicate on the
follow-up survey that tbey were reluctant to retum
Exhibit 3
Eollow-up survey results
E-mail
respondents
Response rates (number)
Likely to respond to mail survey of interest**
Likely to respond to mail survey not of interest
Likely to respond to e-mail survey of interest
Likely to respond to e-maii survey not of interest
Percentage of mail sun/eys completed
E-mail surveys received in past year
Percentage of e-mail survey completed
Knowledgabie regarding e-mail'"
Mail respondents
56,3%(54)
44.2%(65)
1,6
3,4
1.5
3.1
45%
49%
1,54%
49.7%
1,85
,63%*
10,3%'
1.64
1.5
3.3
2.V
3,5*
'significant at ,05 level
" I very likely and 4 very uniikely
' " 1 very knowledgeable and 4 not very knowledgabie
e-mail surveys despite tbeir apparent access to and
knowledge of e-mail? Could tbe data from the email survey be read directly into a database for
immediate presentation or statistical analysis?
LIMITED USE
Despite tbe disadvantages and unanswered questions, e-mail does appear to be a viable alternative
to regular mail in certain circumstances. At tbis
point in time, e-mail surveys would only be representative of groups such as infonnation system professionals, CPAs, engineers, architects, college faculty, or auy select population tbat includes individuals wbo have access to the Internet and whose email addresses are readily available.
E-mail surveys can be especially belpful in
pretesting a survey instrument. In addition to cost
and speed advantages, tbe pretest subject can offer
suggestions and corrections witb ease by using the
reply function. The researcher can subsequently
incorporate recommendations from tbe subjects
and retest tbe questions. E-mail can greatly
enbance tbe dialog during survey construction and,
consequently, strengtben the final product.
Tbe novelty of this area opens tbe door to a variety of additional researcb topics. Tbe feasibility of
using e-mail for consumer research in markets
related to computer hardware and software seems
appropriate. Tbe viability of e-mail surveys to
solicit response from discussion groups without
sending the test instruments to specific addresses
also warrants investigation. Aud, as tbe use of email increases and users become more diversified,
it will be worthwhile to follow its progress by
duplicating tbis type of study periodically. EID
ADDITIONAL READING
Dickson, John P, and Douglas L,
Lewis, Peter H, (1995), '"Home Users
MacLachlan (1992), 'Tax Surveys?"
Biggest Presence on Computer
Markeiing Research: A Magazine of
Networks," A'PH- York Times. 23
Management & AppUcatinn.<i, 4(3),
(October) D-6,
26-30.
Dickson. John P, and Douglas L, MacLachlan (1996). "Fax Surveys:
Retum Patterns and Comparison
with Mail Surveys," Journal of
Marketing Research, forthcoming.
Kiesier, Sara and U e S. Sproull (1986).
"Response EITccts in the Electronic
Survey." Public Opinion Quarterly.
50,402-13,
Lyons, Dan (1994), "The E-Mail
Advantage," Computer Reseller
News, (Septemtwr), 43-52.
Duane Bachmann is a
Professoi" of Marketing at
Central Missouri State
Universitv, Warrensburg, Mo,
John Elfrink is Chairman
of the Accounting
Department at Central
Missouri State tJniversity.
Parker, Loraine (1992), "Collecting Data
the E-Mail Way." Training & Development, (July), 52-54,
Schuldt, Barbara and Jeff E, Totten
(1994). "Electronic Mail vs. Mail
Survey Response Rates." Marketing
Research: A Magazine of Management & Applications. 6(t), 36-9.
Opperman. Martin (1995). "E-mail
Surveys—Potentials and Pitfalls,"
Va77ijna. Garj' A, and Duane Bachmann
Marketing Research: A Magazine of
(1994), "Eax Attracts," Marketing
Management & Appiications, 7
Research: A Magazine of Manage(Summer), 28-33,
ment & Appiications, 6, 19-25.
MARKETING RESEARCH:
Gary Vazzana is an Associate
Professor of Management at
Central Missouri State
University,
SUMMER 1996, Vol. 8 No, 2
35

Download Report