Visual GUI Testing: The technique, Problems and the Future

Visual GUI Testing: The technique,
Problems and the Future
Emil Alégroth and Michel Nass
In collaboration with:
Statement 1: Automated testing
lowers testing cost!
TESTING
Maintenance
Manual delta-testing
Development process
Requirements
Development
Engineering
$
$
Total $
Testing
$
Statement 2: The execution speed is
essential for automated tests!
Nightly runs
Statement 3: Regression testing tools
find all defects!
Button 1
Button X
Button 2
Button 3
Click <button 1>
Verify <button1, green>
Click <button 2>
Verify <button 2,green>
Click <button 3>
Verify <button3, green>
Statement 4: One tool to rule them all!
Customer
needs
Acceptance
testing
System
testing
System
reqs.
Integration
testing
FR reqs.
Code
Unit
testing
XUnit
Visual GUI
Testing
Statement 5: Automated testing can
replace manual testing
Coverage
Manual Exploratory Testing
Background to GUI-based testing
Testing of the system
through
Regression
the GUI,
testing
not
testing of the GUI!
GUI
GUI Components
System components
Code
Change code
GUI-based testing techniques
Visual GUI Testing
Manual GUI-based testing
Structural GUI-based testing
Manual GUI-based testing
Costly
Test
step
Input
1
Click on Button x
button x changes
color
2
Click on
button
y
..
N
Tedious
…
Expecte
d output
Button y
changes
color
…
Click on Button z
button z changes
color
Error-prone
Structural GUI-based testing
OK
Color: Gray
Size: 50x100px
Label: “OK”
Okey
Pros and Cons
Independent
of window size
Finds “invisible”
widgets
Complex tools
Automatic <> Manual
Complex component database
Coordinate based for certain Widgets
Requires collaboration between
testers and developers
Evaluate testability
Environment- and Widget
dependent
Visual GUI Testing
Test
step
Input
Expecte
d output
1
Click on Button x
button x changes
color
2
Click on Button y
button y changes
color
..
…
N
Click on Button z
button z changes
color
…
Automate test-scenarios…
…using image recognition
and AI…
…to emulate end-user behavior
for automated System and
Acceptance testing.
Pros and Cons
Implementation independent
Tests existing UI without modifications
Tests in the same way as a human
Programming skills are not required
Easy to understand the test cases
Simpler and more accurate recording
Reliable playback
Uses the cursor
Limited playback speed
Difficult to understand
how the tool “thinks”
JAutomate Demo
Comparing the Techniques (1/2)
Visual GUI Testing /
Manual GUI-based testing
GUI
GUI Components
System components
Code
Structural GUI-based testing
Comparing the Techniques (2/2)
WebDriver driver = new FirefoxDriver();
driver.get("http://en.wikipedia.org/wiki/Main_Pa
ge");
WebElement about=
driver.findElement(By.xpath("//*[@id='naboutsite']/a"));
about.click();
Thread.sleep(5000);
String title=driver.getTitle();
assertEquals(title, "Wikipedia:About - Wikipedia,
the free encyclopedia");
driver.quit();
Maintenance cost
High
Medium
Low
Manual
Structural
Visual
Based on Academic literature and empirical research in industrial practice
Execution speed
High
Medium
Low
Manual
Structural
Visual
Based on Academic literature and empirical research in industrial practice
Flexibility
High
Medium
Low
Manual
Structural
Visual
Based on Academic literature and empirical research in industrial practice
Required technical knowledge
High
Medium
Low
Manual
Structural
Visual
Based on Academic literature and empirical research in industrial practice
Research into VGT
Applicability of VGT
VGT tools:
Initial
support
for VGT
VGT in industry:
VGT in a real
industrial
project
VGT
maintenance:
long-term
feasibility
Saab in Gothenburg
Sikuli and CommercialTool
Saab in Järfälla
Real development project
VGT:
Saab in Gothenburg and Järfälla
Challenges,
Two development projects
problems and
Inceptive
New VGT
limitations
JAutomate, Sikuli and
tools:
New VGT Academic
JAutomate CommercialTool
approaches: Experiment
Random VGT Sikuli
Saab in Växjö
Sikuli
Maintenance of VGT
VGT suite
System X,
v1.0
Development:
100 hours
1 year
VGT suite
System X,
v2.0
Maintenance:
56-62 hours
~1 month
?
Is your Test Suite rotting?
VGT suite
System Y,
v2.0
Maintenance:
12-18 hours
Future research
“Adopting VGT Automated Testing And Research”,
E. Alégroth, M. Nass, H. Olsson, 2014?
?
Conclusion
“Exploratory”
Intuitive
Costly
Error-prone
Tedious
Flexibility
Maintenance
Emulate user
Intuitive
Slower
Uses the
cursor
How does
the tool think
Choice of technique
depends on the
development
context!
Fast
Record/Repla
y
Maintenance
Complexity
!Applicability
Questions!
[email protected]
[email protected]
Statement 4: One tool to rule them all!
Customer
needs
Acceptance
testing
NFR
testing
NFR reqs.
System
reqs.
System
testing
Integration
testing
FR reqs.
Code
Unit
testing
XUnit
VGT

Download Report