What is Usability?

Measuring Nursing Usability of
Electronic Health Records
February 26, 2014
Frank Lyerla PhD, RN
&
Christine Durbin PhD, JD, RN
DISCLAIMER: The views and opinions expressed in this presentation are those of the author and do not necessarily represent official policy or position of HIMSS.
Conflict of Interest Disclosure
Frank Lyerla PhD, RN and
Christine Durbin PhD, JD, RN
Have no real or apparent conflicts of
interest to report.
© 2014 HIMSS
Learning Objectives
1. Identify the critical components of EHR
usability from a nursing perspective.
2. Categorize nursing EHR usability components.
3. Compare the critical components of nursing
EHR usability with the System Usability Scale.
4. Develop a method for measuring nursing
usability of an EHR.
.
EHR Usability Measurement Protocol:
(Value S.T.E.P.S.)
http://www.himss.org/ValueSuite
EHR Usability Measurement Protocol:
(Value S.T.E.P.S.)
The EHR Nursing Usability Study measures
nursing satisfaction of an Electronic Health Record
(EHR) within assigned individual tasks and as a
part of an overall satisfaction evaluation.
By measuring mouse clicks, keystrokes, errors,
and overall time spent completing specific
assigned nursing documentation tasks, the study
will compare efficiency and effectiveness against
nursing satisfaction with the EHR.
http://www.himss.org/ValueSuite
Background
Missouri Baptist Medical Center (MBMC) in St. Louis MO
One of eight Barnes Jewish Christian (BJC) community
hospitals
Partnership between MBMC and SIUE School of Nursing
Funded by the MBMC Nurse / Faculty Collaborative
Grant
Background
McKesson Horizon Clinical in use for about 5 years
Issues associated with the system
- not fully meeting the needs of the BJC institutions
- system changes affect usability
- user complaints and overtime due to documenting
- administrative decision concerning upgrades or
selecting a different system
Purpose of the Study
Create a protocol that produces clinical (nursing)
information system usability scores
Future studies may include other systems as well as other
system users (medicine, coding, risk management, etc…)
The baseline scores will serve two primary purposes
1.
Determine the impact system modifications have on
usability
2.
System comparisons
EMR Usability:
Who’s working on EMR
Usability?
• HIMSS: Usability Task Force
• National Institute of Standards and
Technology (NIST)
• The Agency for Healthcare Research
and Quality (AHRQ)
• Others
What is Usability?
The extent to which a product can be used
by specified users to achieve specified
goals with effectiveness, efficiency and
satisfaction in a specified context of use.
International Standards Organization, 1998
Ten Principles of Usability
• Simplicity
• Naturalness
• Consistency
• Minimizing cognitive load
• Efficient interactions
• Forgiveness
• Feedback
• Effective use of language
• Effective information presentation
• Preservation of context
HIMSS Task Force, 2009
So we know the definition, goals, and
principles of Usability - but how do
we measure it?
Study design: Two phases
• Phase 1: Nurse focus group sessions to identify
usability concerns & select a satisfaction
survey (subjective measure)
• Phase 2: Develop a protocol that measures
effectiveness & efficiency (objective measures)
and satisfaction (subjective measure)
Definitions
Effectiveness: The accuracy and completeness
with which a user can achieve task goals.
Efficiency: The speed with which a user can
successfully accomplish the task at hand.
Satisfaction: A person’s subjective response to
their interaction with a system.
HIMSS Task Force, 2009
Phase 1: Five BJC Focus Group
Sessions
• Missouri Baptist Medical Center
• Christian Hospital North East
• Boone Medical Center
• Parkland Hospital
• Alton Memorial Hospital
5 RN’s
2 RN’s
5 RN’s
7 RN’s
4 RN’s
Focus Groups
• IRB approval obtained
• Consent forms
• Focus group interview guide
• Audio recordings
• Transcription
• Transcription analysis
Transcripts
• Qualitative data analysis
• Common themes and patterns emerged
• Decision on how to classify / define variables
For example:
“I think downtime is an issue when it is not
scheduled. It happened to us the other morning
and lasted about an hour.” Boone RN
“Having enough working computer work stations
is a huge issue” Boone RN
Transcripts
• It was apparent that many of the concerns
• voiced by the nurses were not just related
• to usability.
• We needed a way to categorize the
• variables.les representing the same variable?
Quality Attributes for Systems
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
Availability
Conceptual Integrity
Interoperability
Maintainability
Manageability
Performance
Reliability
Reusability
Scalability
Security
Supportability
Testability
User Experience / Usability
From: http://msdn.microsoft.com/en-us/library/ee658094.aspx
Examples
Availability (The time a system is functional and working)
“I think downtime is an issue when it is not scheduled. It happened to
us the other morning and lasted about an hour.” Boone RN
“Having enough working computer work stations is a huge issue”
Boone RN
Interoperability (ability for systems to exchange information)
“Nothing pulls from the ER” MoBap RN
“Data can not be exchanged between two in-house systems)” AMH RN
Four Additional Attributes Found
• Lack of knowledge about the system
(Learnability)
• Lack of knowledge about policies / standards
• Other work-related issues that take priority
• Help Desk Support
Data Analysis – System Attributes
Focus on Usability
• While all Quality attributes are important, the
purpose of this study was to concentrate on
determining a method to measure Usability
• Again, a decision was made regarding the
classification/defining of the usability principles.
Note: Improving Usability has been found to
improve Learnability.
Data Analysis – Usability Principles
Selecting a Satisfaction Survey
System Usability Scale (John Brooke, 1986)
- Free
- Simple (10 items)
- Researchers report it to be valid and reliable
- Produces a score (0-100) representing a
composite measure of the overall usability of the
system being studied.
System Usability Scale (SUS)
The SUS and our Qualitative Results
1. (minimizing cognitive load)
2. (simplicity / efficient interactions)
3. (simplicity / efficient interactions)
4. (learnability)
5. (minimizing cognitive load /
interoperability)
The SUS and our Qualitative Results
6. (consistency)
7. (simplicity)
8. (simplicity / naturalness)
9. (forgiveness)
10. (learnability)
Interpreting the SUS
Jeff Sauro examined over 500 research
studies and publicized the following…
Exceptional = 80/100
Average = 68/100
Failing = 50/100
From: http://www.measuringusability.com/sus.php
Phase 2
• Effectiveness
Error rate and completion
• Efficiency
Amount of mouse movement, number of mouse clicks, key
strokes and time it takes to complete a task.
------------------------------------------------------------------------------------• Satisfaction
System Usability Scale (following participant testing)
Efficiency
Effectiveness
NISTIR 7804
• In February of 2012 NIST published
the Technical Evaluation, Testing, and
Validation of the Usability of Electronic
Health Records.
• A guide for conducting usability tests
Usability Software
Morae (TechSmith)
1. Manager
2. Recorder
3. Observer
Three Use Case Scenarios
1.CHF
2. CVA
3. Pneumonia
Eight Tasks for Each Scenario
1. Results Look up
2. Care Organization
3. Assessment
4. Care Plan
5.
6.
7.
8.
Problem List
Medication Administration
Order Entry
Discharge
Creation of Test Patients (IT)
• Training Environment
• Fifteen for each use case
• Provide patient and medication bar
codes for scanning
Location
• Controlled laboratory
setting
• Two testing stations
Lab Facilitators
• Skill set requirements
• Training
• Morae (Observer) to mark errors and check
for completeness (determine effectiveness)
Participant Recruitment
• 15 participants to complete each of the
three scenarios
• Registered Nurses with 2 years
experience using the system
• Gift Cards and allowed to clock in during
the testing sessions
Making Sense of the Data
• Satisfaction:
Overall SUS Score
• Effectiveness:
Error rate /
Completeness (missing data)
• Efficiency:
Baseline data
Correlate efficiency measures with perceived task efficiency
Results
• Due to IT setbacks participant testing
was delayed and therefore only
preliminary results are available
Satisfaction: Average = 60.1 (n=5)
SUS Score Pneumonia
Participant 08-PN
Exceptional
80/100
Participant 06-PN
Average
68/100
Participant 04-PN
Failing
50/100
Participant 02-PN
Participant 01-PN
0
10
20
30
40
50
60
70
80
Effectiveness (n=5)
Errors: 0
Missing data: 0
Efficiency
Average Time on Tasks (Minutes)
Average Perceived Efficiency
12.0
Not
3.00
10.0
8.0
2.00
Moderately
6.0
4.0
1.00
Very
2.0
0.0
0.00
Task 1
Task 2
Task 3
Task 4
Task 5
Task 6
Task 7
Task 8
Task1
Task2
Task3
Task4
Task5
Task6
Task7
Task8
A Review of Benefits Realized
for the Value of Health IT
Establishing a nursing EHR usability
score will be instrumental in helping
healthcare agencies determine the
effects of EHR modifications as well as
assisting in system selection.
.
http://www.himss.org/ValueSuite
Questions?
Thank You!
Frank Lyerla PhD, RN
[email protected]
Christine Durbin PhD, JD, RN
[email protected]
References
• International Standards Organization (ISO) 9241-11. (1998). Available at:
http://www.usabilitynet.org/tools/r_international.htm#9241-11 Accessed
December 15, 2013.
• Belden, J. L., Grayson, R., & Barnes, J. (2009). Defining and testing EMR
usability: Principles and proposed methods of EMR usability evaluation and
rating. Healthcare Information and Management Systems Society (HIMSS).
• Microsoft Application Architecture Guide, 2nd edition, October 2009.
Retrieved May 9, 2013 from: http://msdn.microsoft.com/enus/library/ee658094.aspx
• Measuring Usability with the System Usability Scale by Sauro, J. (2011).
Available at: http://www.measuringusability.com/sus.php Accessed February
2014.

Download Report