Institut für Technik der
Informationsverarbeitung
Tatort Test
„Test is not the Last“
Institut für Technik der Informationsverarbeitung (ITIV)
Prof. Dr.-Ing. Eric Sax
•
Head of „Institut für Technik der Informationsverarbeitung“
•
Director at „Forschungszentrum Informatik, FZI“
KIT – Universität des Landes Baden-Württemberg und
nationales Forschungszentrum in der Helmholtz-Gemeinschaft
www.kit.edu
Contents
1. Review and challenges of the future
2. Process-oriented validation
•
Seamless Process
•
Maturity Level
•
Test depth and coverage indices
3. Test as “fast follower” or rather “late mover”
•
Driver Assistance as an example
4. Summary
TEST TEST
2
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Increasing Functionality
(Automotive)
communication growing faster than the range of functions …
•
•
•
•
•
•
•
•
•
•
• Electric fuel injection
• Cruise control
•
•
•
•
•
Gearbox control
Traction control
ABS
Electric fuel injection
Cruise control
•
•
•
•
•
•
•
•
•
•
•
Airbags
ESP
Active body control
Adaptive gearbox contr.
Adaptive cruise control
Emergency call
Gearbox control
Traction control
ABS
Electric fuel injection
Cruise control
•
•
•
•
•
•
•
•
•
•
•
•
Adaptive low beam
Active steering
Turn warning signal
Stop and go
Lane assistant
Automatic parking
Collision avoidance
Hybridization
Telediagnosis
Vehicle-to-vehicle
communication
Alternative Drivetrains
Airbags
ESP
Active body control
Adaptive gearbox control
Adaptive cruise control
Emergency call
Gearbox control
Traction control
ABS
Electric fuel injection
Cruise control
* Quelle: Vector Informatik
3
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
How it all started …
(Things used to be much better …)
Small,
manageable teams
Chance of
working on
demand
Great depth of
own production
Small quantities
Few variants
No electronics
No software
4
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Complexity as a challenge
Electrics/electronics/software in the vehicle
Control units are spatially distributed
Functions are distributed over several control units
Several functions are integrated in the control units
Safety-related functions with high real-time demands
More than six different bus systems
5
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Complexity as a challenge
Electrics/electronics/software in the vehicle
Control units are spatially distributed
Functions are distributed over several control units
Several functions are integrated in the control units
Safety-related functions with high real-time demands
More than six different bus systems
Devices:
Approximately 140
Thereof 70 control units plus actuators/sensors
28 Body CANs
28 Drive trains
13 Telematics systems
8 Flexray systems
4 MOST
66 LIN
35 million lines of code
6
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Complexity as a challenge
Electrics/electronics/software in the vehicle
Control units are spatially distributed
Functions are distributed over several control units
Several functions are integrated in one control units
Safety-related functions with high real-time demands
More than six different bus systems
Devices:
Approximately 140
Thereof 70 control units plus actuators/sensors
28 Body CANs
28 Drive trains
13 Telematics systems
8 Flexray systems
4 MOST
66 LIN
35 million lines of code
7
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Contents
1. Review and challenges of the future
2. Process-oriented validation
•
Seamless Process
•
Maturity Level
•
Test depth and coverage indices
3. Test as “fast follower” or rather “late mover”
•
Driver Assistance as an example
4. Summary
TEST
12
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Seamless testing
System
Requirements
Analysis
Acceptance test
Transition to
Utilization
test
analysis
Validation
System Design
SW / HW
Requirements
Analysis
Preliminary
Preliminary
Hardware
Software Design
design
System
Integration
Software
Hardware
Integration
Integration
Verification
Detailed H
Detailed
ware Design
Software
Design
implementation
Software
Hardware
Implementation
Implementation
13
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Seamless testing
14
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Principles of testing
Testing is a process
Testing demands discipline - internally and among partners
Testing has something todo with trust
„Tools follow Process“
4-Eye-principle: designing and testing are 2 different roles
Quality can‘t be improved by testing – but will soon get lost
Demands
Clear specifications
Transparent steps and feature/function definition
Complete documentation
Defined (process-)interfaces
Defined roles (including tasks and responsibilities)
This can be assessed!
15
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Issue Tracking
Test-Analysis
Test-Implementation
Test-Specification
Test-Planing
Test-Strategy
Requirements
Test-Process Reference-Model:
Projektmanagement & Controlling
16
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Test-Process Reference-Modell:
Issue Tracking
Test-Analysis
Test-Implementation
Test-Specification
Test-Planing
Test-Strategy
Requirements
Collect all Requirements
Projektmanagement & Controlling
17
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Test-Process Reference-Modell:
Issue Tracking
Test-Analysis
Test-Implementation
Test-Specification
Test-Planing
Test-Strategy
Requirements
Classification
(N,P,S,L,F)
Projektmanagement & Controlling
18
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Measuring Systemtests – the NSPLF-Schematic
Base
Main parts of the device under test (test object)
are known and documented
N
Test Coverage „none" or: „N"
Test object is tested by chance or implicitely
S
Test Coverage „sparsely" oder: „S"
For each feature at least one test case exists
P
L
F
19
Tatort Test
Stuttgart, 07.05.2015
Test Coverage „partly“ or: „P“
For each requirement at least one test case exists
Test Coverage „largely“ or: „L"
For each condition at least one test case exists
Test Coverage „fully or: „F"
Additionally, test cases based on experience are included
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Challenge: Effort vs. Completeness
(Example exterior light  indicator)
Functions
• Early definition (S, P, L, F) per system
• Committment among all partners
Features
Features
N – none
S – sparseley
Constraints
Indicator
∑ Test Cases
No systematic tests
„1 Test per Feature“
Feature
Test
F135
T443
Turn Signal
Indicator left on
48 Features
48 Test Cases
Functions
Depth
Depth
20
P – partly
L – largely
F – fully
Tatort Test
Stuttgart, 07.05.2015
„1 Test per Requirement“
„Combinations“
„Test Cases based on
experience“
Req.
A2F78
Indicator left
reacts on …
Indicator left on
Indicator right
reacts on …
Indicator right on
Test
Req.
T443
A279
Test
T449
Cond.
Cond.
Test
Test
A398
A397
T3506
T3584
KL15/1 on
Emergency flashing
Indicator on
Indicator during
emergency flashing
Test
Example
Function in case of
lost CAN-messages
86 Req.
86 Test Cases
215 Cond.
301 Test Cases
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Test-Process Reference-Modell:
Issue Tracking
Test-Analysis
Test-Implementation
Test-Specification
Test-Planing
Test-Strategy
Requirements
Def. of Responsibilities
detailed planing
(who, when, what)
Projektmanagement & Controlling
21
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Test-Matrix:
Tool for
systematic role
and task
tracking
Catogories for
test cases in
three levels
Master
document for
test process
22
Tatort Test
Stuttgart, 07.05.2015
xxxxxxx
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Test-Process Reference-Modell:
Issue Tracking
Test-Analysis
Test-Implementation
Test-Specification
Test-Planing
Test-Strategy
Requirements
Definition of Test Cases
tool-unspecific & readable
Projektmanagement & Controlling
26
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Durchgängige Testfall-Dokumentation
E.g. in MS-Excel
Interiour Light
Alternatives:
Natural language
Procedural languages
Classification tree
Time Partition Testing
UML structure-diagrams
UML behaviour-diagrams
TTCN3
…
27
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Test-Process Reference-Modell:
Issue Tracking
Test-Analysis
Test-Implementation
Test-Specification
Test-Planing
Test-Strategy
Requirements
Test-Programming,
Test-Execution
Documentation
Projektmanagement & Controlling
28
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Hardware-in-the-loop (HiL)
Design and dimensioning
The control units are allocated via switch matrix to the respective CAN
buses in different vehicle configurations
• Multiplex system,
data gateway
• Instruments
• Lighting devices,
mirror systems
• Rain-light sensor
• Compressed-air
supply
• Energy
management
• Radio/Com.
• Audiosystem
• Telematics,
Bus-FMS
• Vehicle inf.
systems
• Traffic light
override
• Ramps, vehicle
hoists
• Video/DVD/TV
• Navigation
• Mobile phone
29
Tatort Test
Stuttgart, 07.05.2015
• Heating,
ventilation, air
conditioning
• Modular switch
system
• Tire pressure
control
• Door systems,
locking systems
• Flaps
• Level control
• Additional
steering
• Motorsystem
• Instruments
• Fire Alarm
• Real
Antriebstrang
• Level control
rear vehicle
• Articulation angle
control
•
•
•
•
•
Cruise control
Vehicle control
Retarder
Intarder
Transmission
• Automatic
transmission with
converter clutch
Coach specifics
City Bus specifics
• Manual
transmission
• Automated
manual
transmission
• Exhaust
aftertreatment
• Compressed-air
supply
• Tachograph
• Digital
tachograph
• Braking system
• Vehicle
dynamicscruise
• Adaptive
systems
control
• Lane keeping
assist
• Active braking
assist
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Hardware-in-the-loop (HiL) – Vehicle variants
Variably adjustable test stand – Transmission variant examples
HiL-Hardware
Modell
(self made)
Modell (selbst erstellt)
automated
ModellModell
(re-used
components)
(Verwendung
Konzernmodell) transmission
Modell
manual
(self made)
ECU
ECU
ECU
Automated
transmission
transmission
Supplier 1
Supplier 2
Supplier 3
ECU
Automated
transmission
HiL test stand enables alternative designs of vehicle variants from a set of standard components
• Three types of transmissions (manual, automated, and stepped transmission)
• In-house transmissions plus different suppliers
30
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Facts and figures
Hardware-in-the-loop (HiL)
Subdivided into the domains of body chassis (BC) and powertrain
vehicle dynamics (PVD)
Simulation in realtime(1ms)
Real and modeled environment
Real ECUs@HiL: 49
Number of system-tests: 2.300
System-Tests per year: 18.070
Lines of Code: Ca. 3,7 Mio.
Ca. 900 found Issues in 4 years
31
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Hardware-in-the-loop (HiL) – Real Components
Completion of the subsystems of the integration test stand
E.g. compressed-air brake
E.g. lighting devices
Control &
Reporting
Testautomation
e.g. Cluster validation
32
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Hardware-in-the-Loop (HiL): Instrument Cluster
Device under
Test
Image aquisition
HW- interfaces &
Image recognition
Control &
Reporting
FireWire
IEEE1394
Capturing
CAN-Link
Stimulation
Testautomation
e.g. Cluster validation
33
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Hardware-in-the-Loop (HiL): Instrument Cluster
Control &
Reporting
Testautomation
e.g. Cluster validation
34
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
World Wide Approach
Control &
Reporting
Testautomation
35
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
World Wide Approach
Offshore Testing
Remote Access
36
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Test-Process Reference-Modell:
Issue Tracking
Test-Analysis
Test-Implementation
Test-Specification
Test-Planing
Test-Strategy
Requirements
Result Analysis and
Test Statistics
Projektmanagement & Controlling
37
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Test-Process Reference-Modell:
Issue Tracking
Test-Analysis
Test-Implementation
Test-Specification
Test-Planing
Test-Strategy
Requirements
Tracking maturity,
continous improvement
Projektmanagement & Controlling
40
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Existing Maturity Models
CMMI/SPICE
Verification and Validation are
not entirely adressed
CMMI® mainly only reviews the
documents but does not care
about „how“
(Automotive) SPICE demands
strategic and generic approach
Both models do not adress test
equipment and planing
sufficiently
41
Tatort Test
Stuttgart, 07.05.2015
Test Process Improvement
(TPI®)
Combination with existing
assessment critical
TPI® Automotive defines no
reference process
To much open and flexible
issues to make a comparison of
different methods
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Assessments
Preparation
Interviews
Status
3-5 Interviews a 2h
Checkliste
Open discussions
Analysis
Status Quo
0
Key area
01 – Test strategy
02 – Life-cycle model
03 – Moment of involvement
04 – Estimating and planning
05 – Test design techniques
06 – Static test techniques
07 – Metrics
08 – Test automation
09 – Test environment
10 – Office and laboratory environment
11 – Commitment and motivation
12 – Test functions and training
13 – Scope of methodology
14 – Communication
15 – Reporting
16 – Defect management
17 – Testware management
18 – Test process management
19 – Evaluation
20 – Low level testing
21 – Integration testing
42
Tatort Test
Stuttgart, 07.05.2015
1
2
3
A
A
4
5
6
7
8
9
Test Matrix
(who does what)
Control
Implementation
Decisions
10 11 12 13
B
C
D
B
C
B
D
B
A
A
B
A
A
B
A
B
A
A
A
C
C
C
B
B
A
A
B
B
Management
C
D
Operations
C
D
C
Operations
C
B
C
A
A
A
Equipment
C
C
B
B
A
A
Management
Management
B
A
D
C
B
B
A
A
A
Use Case based
C
B
B
B
C
C
C
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Experiences derived from assessments
Main results
Quality and maturity of
automotive ECUs is already on
high level
Lack of transparency of used
tools and methods leads to gaps
and double work in the
OEM/supplier chain.
No common test strategy
committed
43
Tatort Test
Stuttgart, 07.05.2015
Chances for optimization
More focus for planning and
methodological aspects
Strategic testing
Priorities for test activities (i.e.
according to window of time)
Better role definitions
More systematical test definition
linked to requirements (s.
NSPFL-Method)
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Contents
1. Review and challenges of the future
2. Process-oriented validation
•
Seamless Process
•
Maturity Level
•
Test depth and coverage indices
3. Test as “fast follower” or rather “late mover”
•
Driver Assistance as an example
4. Summary
TEST
44
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Increasing Functionality
(Automotive)
communication growing faster than the range of functions …
processes, methods, and tools cannot cope with the growing
“complexity” of the respective systems …
further challenges through
• Adaptive low beam
• Active steering
functional safety …
•
•
•
•
•
•
•
•
• Electric fuel injection
• Cruise control
•
•
•
•
•
Gearbox control
Traction control
ABS
Electric fuel injection
Cruise control
•
•
•
•
•
•
•
•
•
•
•
Airbags
ESP
Active body control
Adaptive gearbox contr.
Adaptive cruise control
Emergency call
Gearbox control
Traction control
ABS
Electric fuel injection
Cruise control
•
•
•
•
•
•
•
•
•
•
•
•
Turn warning signal
Stop and go
Lane assistant
Automatic parking
Collision avoidance
Hybridization
Telediagnosis
Vehicle-to-vehicle
communication
Alternative Drivetrains
Airbags
ESP
Active body control
Adaptive gearbox control
Adaptive cruise control
Emergency call
Gearbox control
Traction control
ABS
Electric fuel injection
Cruise control
* Quelle: Vector Informatik
45
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Example
v2 = 120 km/h
v0 = 80 km/h
v1 = 80 km/h
v3 = 180 km/h
46
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Example
v2 = 120 km/h
v0 = 80 km/h
v1 = 80 km/h
v3 = 180 km/h
47
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Alternatives in scenario
Initial situation
Blue truck and orange car at 80 km/h  orange car changes lane
s1; v1
s2; v2
• v from 0 km/h to 120 km/h in 10 km/h steps → 12 variants
• s from 0 m to 100 m in 10 m steps → 10 variants
• Two pairs → 12*12*10*10* = 14,400 variants for initial situation
48
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Alternatives in scenario
Initial situation
Blue truck and orange car at 80 km/h  orange car changes lane
s1; v1
s2; v2
• v from 0 km/h to 120 km/h in 10 km/h steps → 12 variants
• s from 0 m to 100 m in 10 m steps → 10 variants
• Two pairs → 12*12*10*10* = 14,400 variants for initial situation
49
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Seamless testing
System
Requirements
Analysis
Test
Acceptance test
Transition to
Utilization
test
analysis
Validation
System Design
SW / HW
Requirements
Analysis
Preliminary
Preliminary
Hardware
Software Design
design
System
Integration
Software
Hardware
Integration
Integration
Verification
Detailed H
Detailed
ware Design
Software
Design
implementation
Software
Hardware
Implementation
Implementation
50
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Seamless testing
Test
System
Requirements
Analysis
AcceptanceTransition
test
to
Transition to
Utilization
Utilization
test
analysis
Validation
System
System Design
Integration
System
Integration
SW / HW
Requirements
Analysis
Software
Integration
Verification
Preliminary
Software Design
design
Software
Integration
Detailed
Software Design
implementation
Software
Implementation
51
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Seamless testing
Test
System
Requirements
Analysis
Acceptance test
test
analysis
Validation
System Design
MiL
SW / HW
Requirements
Analysis
Preliminary
Preliminary
Hardware
Software Design
design
Tatort Test
Stuttgart, 07.05.2015
System
Integration
Software
Hardware
Integration
Integration
Verification
I-HiL
Detailed
Software Design
SiL
Software
Hardware
Implementation
Implementation
52
Transition to
Utilization
implementation
K-HiL
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Seamless testing
Test
System
Requirements
Analysis
Acceptance test
test
analysis
Validation
System Design
MiL
SW / HW
Requirements
Analysis
Preliminary
Preliminary
Hardware
Software Design
design
Tatort Test
Stuttgart, 07.05.2015
System
Integration
Software
Hardware
Integration
Integration
Verification
I-HiL
Detailed
Software Design
SiL
Software
Hardware
Implementation
Implementation
53
Transition to
Utilization
implementation
K-HiL
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Seamless testing
Test
System
Requirements
Analysis
Acceptance test
ViL
test
analysis
Validation
System Design
MiL
SW / HW
Requirements
Analysis
Preliminary
Preliminary
Hardware
Software Design
design
Tatort Test
Stuttgart, 07.05.2015
System
Integration
Software
Hardware
Integration
Integration
Verification
I-HiL
Detailed
Software Design
SiL
Software
Hardware
Implementation
Implementation
54
Transition to
Utilization
implementation
K-HiL
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Film von J. Kramer
55
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Concentrate on the functional realization
Isolation of the assisting function from the surrounding components
Split the algorithmical function from the technical realisation
Standardize the interfaces
Sensor-Tests
Sensor
Sensor
Sensor
56
Tatort Test
Stuttgart, 07.05.2015
Fusion-Tests
SensorFusion
Actuator-Tests
Assistance
Function
Execution
Aktuator
Aktuator
Actuator
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Agile Development  Agile Testing
Design Process
User-Stories
Concrete Scenario
Testing Process
Test-Stories
Initial scenario
Parameter variation for starting
situation
Limits, tolerances, steps
Pass- / fail-criteria
check
57
Tatort Test
Stuttgart, 07.05.2015
trust
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Test-Story for Parking Assistent
Parking in parallel to side-strip
1
Vehicle supported
by park assistence
2
4
58
Tatort Test
Stuttgart, 07.05.2015
3
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Anzahl Testfälle für eine Test-Story
Test-Story
Runs after initial position
Variations of parameters for starting point
1
2
3
4
Seize of the parking spot
Distance of vehicle under test relative
to parking spot
Angle relative zu parking spot
Vertical distance of vehicle under test relative to parking spot
Stepwise approach
1
2
3
4
1 to 2 vehicle length in 100 steps
1 to 2 vehicle length in 100 steps
Angle -25 to +25 degrees in 10 steps
0,5 to 2 vehicle width in 100 steps
Sum of all test-cases for this test-story  100 * 100 * 10 * 100 = 10 Mio. cases
59
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Criteria for Pass/Fail
Criteria for successful test case (pass)
Regarding tolerances
No binary decision
E.g.
Parking position in an area
Rotation: +/- 5 degrees
…
Criteria for un-successful test case (fail)
Off tolerance in final position
Collision during maneuver
…
60
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Eventuality and further Test-Stories
Eventualities
No or dynamic reference objects
Additional dynamical objects in parking spot
Additional vehicle searching für parking spot
Incoming human, animal or object
…
Further User-Stories result in futher test-stories
Vertical parking
Parking in garage
Relying on roadmarking
…
61
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Contents
1. Review and challenges of the future
2. Process-oriented validation
•
Seamless Process
•
Maturity Level
•
Test depth and coverage indices
3. Test as “fast follower” or rather “late mover”
•
Driver Assistance as an example
4. Summary
TEST
62
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Motivation for a seamless
Life Cycle Development Model
Weitere Tools:
Anforderung
System
Requirements
Netzwerk-Analysis
AcceptanceTransition
test
to
E/EFeatureliste
Simulation
analysis
GR&AE
Validation
System
System Design
Bauraum,
Verlegewege
Releaseplanung
SW / HW
E/ERequirements
Architektur
Analysis
z.B. FAS-Systeme
Change
Management
Software
Integration
Software Design
(LogiCAD, FUP-Doku)
GBZ
Neue Funktionen
Software
FUP-Liste
Detailed
MUX-Pinning
Software
Design
Stromlaufplan
SAP test
(KEM)
VesuV
BusDoc
Protokoll
Integration
PH von
Diag. Komm. Daten
Lieferant Verification
Preliminary
Test
-spezifikation
FUP-Entwicklung
design
System
Versuch
Integration
Integration
Systemdesign
System, Funktion
Spezifikation, KLH
Transition to
Utilization
Utilization
HiL Test
CANape
ZPRO, ZAEM
SAP
Issuetracking
implementation
NW Test
DPC Test
SG-SW
Komponente
Software
Diag. Test
EDLS
Timinganalyse SLP-Listen
Implementation
CANoe, etc.
CAT
FUP Prüflabor Leitungssatz
Parameter CANdela Flash
VarCode
Online Tester Entwicklung
63
63
Tatort Test
Stuttgart, 07.05.2015
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Motivation for a seamless
Life Cycle Development Model
Weitere Tools:
Test
Anforderung
System
Requirements
Netzwerk-Analysis
AcceptanceTransition
test
to
E/EFeatureliste
Simulation
analysis
GR&AE
Validation
System
System Design
MiL
Bauraum,
Verlegewege
Releaseplanung
SW / HW
E/ERequirements
Architektur
Analysis
Software
MUX-Pinning
SiLDetailed
Software
Design
(LogiCAD, FUP-Doku)
GBZ
Neue Funktionen
z.B. FAS-Systeme
Change
Management
64
Tatort Test
Stuttgart, 07.05.2015
Software
Integration
I-HiL
Software Design
FUP-Liste
Stromlaufplan
SAP test
(KEM)
VesuV
BusDoc
Protokoll
Integration
PH von
Diag. Komm. Daten
Lieferant Verification
Preliminary
Test
-spezifikation
FUP-Entwicklung
design
System
Versuch
Integration
Integration
Systemdesign
System, Funktion
Spezifikation, KLH
Transition to
Utilization
Utilization
ViL
HiL Test
K-HiL
Komponente
CANape
ZPRO, ZAEM
SAP
Issuetracking
implementation
NW Test
DPC Test
SG-SW
Software
Diag. Test
EDLS
Timinganalyse SLP-Listen
Implementation
CANoe, etc.
CAT
FUP Prüflabor Leitungssatz
Parameter CANdela Flash
VarCode
Online Tester Entwicklung
64
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015
Motivation for a seamless
Life Cycle Development Model
Weitere Tools:
Test
Anforderung
System
Requirements
Netzwerk-Analysis
AcceptanceTransition
test
to
E/EFeatureliste
Simulation
analysis
GR&AE
Validation
System
System Design
MiL
Bauraum,
Verlegewege
Releaseplanung
SW / HW
E/ERequirements
Architektur
Analysis
Software
MUX-Pinning
SiLDetailed
Software
Design
(LogiCAD, FUP-Doku)
GBZ
Neue Funktionen
z.B. FAS-Systeme
Change
Management
65
Tatort Test
Stuttgart, 07.05.2015
Software
Integration
I-HiL
Software Design
FUP-Liste
Stromlaufplan
SAP test
(KEM)
VesuV
BusDoc
Protokoll
Integration
PH von
Diag. Komm. Daten
Lieferant Verification
Preliminary
Test
-spezifikation
FUP-Entwicklung
design
System
Versuch
Integration
Integration
Systemdesign
System, Funktion
Spezifikation, KLH
Transition to
Utilization
Utilization
ViL
HiL Test
K-HiL
Komponente
CANape
ZPRO, ZAEM
SAP
Issuetracking
implementation
NW Test
DPC Test
SG-SW
Software
Diag. Test
EDLS
Timinganalyse SLP-Listen
Implementation
CANoe, etc.
CAT
FUP Prüflabor Leitungssatz
Parameter CANdela Flash
VarCode
Online Tester Entwicklung
65
Institut für Technik der Informationsverarbeitung
Prof. Dr.-Ing. Eric Sax
© 2015