ETSI PowerPoint Presentation Template

ETSI Update on TTCN-3 Language Evolution
and Tool Conformance Testing
Nikolay Pakulin
Contributors:
Ina Schieferdecker
Jens Grabowski
Agenda
 STF 430: TTCN3 Evolution
 TTCN3 Overview
 Advanced TRI
 Continuous Signals
 STF 433: TTCN3 Tool Conformance Testing




Motivation
Methodology
Challenges
Requirements-driven Test Development
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 2
TTCN-3 Evolution
TTCN-3 User Conference
2012, Bangelore, India
Nikolay Pakulin
on behalf of STF430
STF administrative data
 STF 430: April 2011 – December 2011 (78 mDays)
 Members:




Gyorgy Rethy, Ericsson
Jens Grabowski, University of Goettingen
Ina Schieferdecker, Fraunhofer FOKUS
Jacob Wieland, Testing Technologies
 STF 430 sessions in 2011




Session 1: 24-27 May
Session 2: 27 June - 1 July
Session 3: 26 - 30 September
Session 4: 28 November - 2 December
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 4
TTCN-3 Technology Overview
Language
mappings
ASN.1
IDL
XSD
Extensions
Documentation
t3doc
Advanced
parameterization
Behaviour
types
Static
configuration
Real-time
support
XML
C#
Continuous
signal
support
Advanced
TRI
TTCN-3 Structuring:
Imports, Groups, Attributes
Core
language
TTCN-3 Behaviour
TTCN-3 Data
TRI/TCI
mapping
Java
TTCN-3 User Conference
2012, Bangelore, India
C
C++
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 5
TTCN-3 language evolution
 Tasks
 Resolution and implementation of CRs
 Prepare the drafts for ETSI publication:
Parts 1 and 4 to 10 of the ES 201 873 series
The 4 extension packages (ES 202 781, ~782, ~784, ~785)
 Cooperation with STF 160 on language requirements of LTE
testing
 Two new language extension packages
 Advanced TRI
 Continuous signal support
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 6
The version forest
 Approved new versions











TTCN-3 Part-1: Core Language
V4.4.1 (44 CRs)
TTCN-3 Part-4: Operational Semantics
V4.4.1 (1 CR)
TTCN-3 Part-5: TTCN-3 Runtime Interface
V4.4.1 (2 CRs)
TTCN-3 Part-6: TTCN-3 Control Interface
V4.4.1 (8 CRs)
TTCN-3 Part-7: Using ASN.1 with TTCN-3
V4.4.1 (1 CR)
TTCN-3 Part-8: Using IDL with TTCN-3
V4.4.1 (2 CR)
TTCN-3 Part-9: Using XML with TTCN-3
V4.4.1 (12 CRs)
TTCN-3 Part-10: TTCN-3 docum. tags
V4.4.1 (1 CRs)
TTCN-3 Extension: Behaviour Types
V1.2.1 (3 CRs)
TTCN-3 Extension: Configuration and deployment V1.2.1 (1 CR)
TTCN-3 Extension: Real-Time and Performance Testing V1.2.2 (2 CRs)
 Not republished:
 TTCN-3 Extension: Advanced Parameterization
V1.3.1
 New:
 TTCN-3 Extension: Advanced TRI
 TTCN-3 Extension: Continuous signal support
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
V1.1.1
V1.1.1
Slide 7
New features in Part 1
 5417: Support of parametrized map/unmap
var MyConfigType cfg:={ option := 1, lock := false};
...
map(mtc:Port4, system:PCO2) param (cfg);
 5262: Partially constrained structured types
 5938: Type restriction by template list
type MyRecord MyRecordSub5 (
{f1 := *, f2 := "user", f3 := pattern
"password|Password" },
{ f1 := (1 .. 10), f2 := "User“, f3 := ? })
 5937: Generalized annotation of attributes to
declarations/members
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 8
New: Advanced TRI 1(3)
 Historically, TTCN has been used to test
communication protocols which typically use
encoded messages. This has been reflected in the TRI
SA and TCI CD design of TTCN-3 by encoding and
decoding messages to/from bitstrings.
 However, TTCN-3 also supports signature-based
communication for which the transformation of
objects into bitstrings and vice versa is cumbersome.
 Therefore, an alternative API is being proposed along
which TTCN-3 values can be directly passed to/from
the SUT.
 Old and new TRI can be used in combination!
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 9
New: Advanced TRI 2(3)
 Redefinition of TRI Data Interface






TriMessageType
use Value
TriAddressType
use Value
TriAddressListType
use ValueList
TriSignatureIdType
unchanged
TriParameterType
unchanged (not being used)
TriParameterPassingModeType
unchanged (not being used)
 TriParameterListType
TciParameterListType
(by that, the TciParameterPassingMode and Values for
parameters will be used)
 TriExceptionType
use Value
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 10
New: Advanced TRI 2(3)
 Changes to Message-based communication operations




triSend  xtriSend
triSendBC xtriSendBC
triSendMC xtriSendMC
triEnqueueMsg xtriEnqueueMsg
 New
 Value xtriConvert(in any value, in Type typeHypothesis)
This operation shall be called whenever the TE has to convert a value. The
TE might convert immediately after reception of the value, or might for
performance considerations postpone the conversion until the actual
access to the value.
 Changes to Procedure-based communication operations
 triCall xtriCall
 etc.
 Changes to Miscellaneous operations
 triExternalFunction  xtriExternalFunction
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 11
TTCN-3 Extension: Continuous signal
support 1(4)
 This new package defines concepts for testing
systems using continuous signals.
 The package introduces
 notions of time and sampling,
 notions of streams, stream ports and stream variables,
and
 definitions allowing an automaton alike control flow
structure supporting the specification of hybrid
behavior (i.e., combined definition of discrete and
continuous behaviors).
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 12
TTCN-3 Extension: Continuous signal
support 2(4)
 The following items shall give an impression about
the contents of the package. Details and further
operations can be found in the package definition.
 Concepts for time and sampling:
 access to global time:
 Setting stepsize for sampling:
now operator
stepsize operator
 Streams
 New port type for streams, e.g.,:
 type port StreamIn stream { in float }
 type port StreamOut stream { out float }
 Operations for accessing data streams:
 retrieving or setting actual values: value operation
 retrieval of time related information: timestamp operation
 Retrieval of actual stepsize of a port: delta operation
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 13
TTCN-3 Extension: Continuous signal
support 3(4)
 Streams (cont.)
 navigation on streams
 previous event:
 event at a certain point in time:
prev operation
at operation
 stream extraction
 Retrieval of (partial) history:
history operation
 Modes
 define states in hybrid automata. Modes may






TTCN-3 User Conference
2012, Bangelore, India
be composed sequential or parallel
have entry and exit conditions
have invariant parts
have a duration
be parameterisable
etc.
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 14
TTCN-3 Extension: Continuous signal
support 4(4)
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
15
Slide 15
A Conformance Test Suite for TTCN-3 Tools
Bogdan Stanca-Kaposta, Testing Technologies
Andras Kovacs, Broadbit
Nikolay Pakulin, ISPRAS
Motivation





Many TTCN-3 compilers (~7 commercial, 2 internal)
High interest in the language
Complex standard: hundreds of pages
No way to check the compliance of TTCN-3 tools
Quality of a test tool is critical
 Relevance:
 For users:
 Is my tool standard compliant?
 Can I switch tools easily / vendor lock-in?
 For tool vendors:
 Where can I improve the standard compliance of my
tool?
 Possibly later: compliance certification
 For standard developers:
 Where is need for clarification?
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 17
STF 433
 Specialist Task Force initiated by ETSI MTS
 Manpower: 3 persons for 100 days total
 Dates: November 2011 – Early 2012
 Work is based on the existing framework.
 Task:
 Update the existing TTCN3 ATS to the newer TTCN-3
revision
 Extend the conformance test suite for TTCN-3 tools.
 Three deliverables:




ETSI TS 102 950-1 V1.2.1 (2012-05) – ICS
ETSI TS 102 950-2 V1.2.1 (2012-05) – TSS & TP
ETSI TS 102 950-3 V1.2.1 (2012-05) – ATS & IXIT
Approved by MTS in May 2012.
 Downloadable from the ETSI download area right now!
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 18
Methodology (1/7)
 Main steps:
1. Validate existing tests against new revision of the
standard.
2. Develop conformance tests.

Follow the priority list discussed with MTS and vendors.
3. Develop adapters for selected tools. Validate and fix
tests.
4. Automate test execution.
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 19
Methodology (2/7)
 Adjusting Conformance ATS from TTCN3 rev. 4.2.1 to
rev. 4.3.1:

Was: 733 tests, 130 pp. of test purposes





Validate that test purposes still hold
Update modified test purposes
Revoke outdated tests
Update ATS folder structure
Approximately 30% of the STF 433 effort
 New tests development
 Clause priority list:


TTCN-3 User Conference
2012, Bangelore, India
Not all clauses can be covered. Selection was made:
Vendor feedback + most basic functionality.
There is no perfect selection!
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 20
Methodology (3/7)
 Development of conformance tests:
 The ATS is not the test automation, but the test input!
 In most cases: 1 module = 1 conformance test.
 Test automation provided for certain vendors
 Elvior and Testing Technologies
 Test suite structure: organized according to clauses.
 Expected output for verdict determination:
 @verdict pass accept/reject [expectedoutput]
 Examples:
 @verdict pass reject
 @verdict pass accept, ttcn3verdict:pass
 @verdict pass accept, noexecution
 Purpose specification:
 @purpose documentreference, description
 Documentreference: part:clause
 Examples:
 @purpose 1:5, Ensure that when the IUT loads a module
containing some definitions before the module declaration then
the module is rejected.
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 21
Methodology (4/7)
 Development of conformance tests (cont.):
 Complete TTCN-3 test input example:
/***************************************************
** @purpose 1:5.2.2, Ensure that the IUT correctly
handles the uniqueness of
variable names in its scope.
** @verdict pass accept, ttcn3verdict:pass
***************************************************/
module Sem_050202_Uniqueness_001 {
import from Sem_050202_Uniqueness_001_import {
const all;
}
type component GeneralComp {}
function f_funcScope() {
var boolean repeatedIdentifier := true;
if(repeatedIdentifier==true) { setverdict(pass);
}
}
testcase TC_Sem_050202_Uniqueness_001()
runs on GeneralComp { f_funcScope(); }
control { execute(TC_Sem_050202_Uniqueness_001()); }
}
TTCN-3 User Conference
2012, Bangelore, India
module Sem_050202_Uniqueness_001_import {
// repeated indentifier
// from imported module
// is allowed
const integer repeatedIdentifier := 0;
}
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 22
Methodology (5/7)
 Creation of documentation:
 Framework already provided by STF393 (Proforma).
 Little time for documentation.
 Working on a single source saves time.
 Solution:
 Write test purposes and expected tool outputs directly
into the test inputs.
 Generate tables for the documentation from the ATS:
 Test purpose tables.
 Test suite structure.
 Much easier to maintain!
 Documentation generation tools developed as part of
STF409.
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 23
Methodology (6/7)
 Validation and fixing:
 Validation and fixing took more than half of the
development time!
 STF internal validation with 2 tools:
 TTworkbench: compile-time and execution.
 TestCast: compile-time and execution.
 Developed necessary adapters
 Loopback, proc. call
 Lots of feedback from TTech and Elvior!
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 24
Methodology (7/7)
 Executing the ATS:
TTCN-3
File
TTCN-3
File
TTCN-3 File
ATS
Sem_060101_TopLevel_001.ttcn
NegSem_060303_component_types_001.ttcn
…
Sem_060101_TopLevel_001.ttcn
** @verdict
pass accept, ttcn3verdict:pass
Test input
TTCN-3 Tool (SUT)
Expected
output
Tool output
Rejection as invalid
Execution with results
(e.g., pass/inconc/fail)
Rejection to execute
Tool output evaluation
pass
TTCN-3 User Conference
2012, Bangelore, India
fail
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 25
Discussion
TTCN-3
Conformance ATS
TTCN-3 Tool
pass
Meaning?
 Compliance does not imply safety from a vendor lock-in!
 Testing is never complete!
 With v1.1.1 of the ATS, about 32% of the core standard clauses
are covered with at least one test.
 Ideally, we should have many tests for each clause.
 Compliance means less likelihood for vendor lock-in.
 With better coverage and higher number of test cases, we
improve the quality of this statement.
 Compliance reduces the likelihood for major tool errors
 Tests range from simple arithmetic operations to
communication operations and verdict handling.
 ATS checks syntax and semantics.
 ATS does not check the correctness of every single tool
message, but of the overall output.
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 26
Challenges (1/2)
 Various standard clarifications were necessary:
 13 CRs for TTCN-3 core languages have been written.
 Unclarified test cases have been excluded from the delivery
("on-hold").
 Tools sometimes handle the standard differently.
 Tests need to be expressed as tool independent as possible.
 Test Automation:
 No standard command line interface for batch run
 Tool-specific adapters
 No common test log format
 TTCN3 semantics
 TTCN3 standard part 1 defines both static and dynamic
semantics
 No definition of well- and ill- formed programs
 What to check at compile time is tool-dependent
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 27
Challenges (2/2)
 Synchronize ATS with new TTCN3 revisions
 Time consuming manual activity
 As ATS grows the effort will grow as well
 To which core language standard version does the
conformance test suite relate to?
 How to maintain compatibility across different language
versions?
 Against which version should the conformance test suite be
developed in the future?
 Which TTCN3 revision to support
 Should we certify newest TTCN3 revision only?
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 28
Conclusion
 The conformance test suite is available now!
 ATS Metrics:
 Total number of conformance tests: 1077 (340 new)
 20 tests fixed
 9 removed since deprecated or duplicates





Number of positive syntactic conformance tests: 138
Number of negative syntactic conformance tests: 82
Number of positive semantic conformance tests: 517
Number of negative semantic conformance tests: 340
Number of clauses (at least partially) covered: 134 (24 new)
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 29
30
TTCN-3 CONFORMANCE
TEST SUITE STF 433
STATUS REPORT
Requirements tracking
case study
Nikolay Pakulin
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 30
Case study in a nutshell
4 chapters: total 7,5 pages




12 Declaring messages,
13 Declaring timers,
23 Timer operations,
24 Verdict operations;
56 requirements elicited,
 7 requirements refined into 34 test purposes;
 covered by 174 test cases
Average 7,5 requirements per page, 3 test
cases per requirement
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 31
Requirements tracking case study
 ISP RAS contributed to the project requirements
management tool Requality
 Hierarchical grouping of requirements
 Requirements elicitation from DOC/HTML documents
 Requirements are connected to certain locations in the
documents
 Requirements locations are highlighted
 Requirements refinement through test purposes
 Test purposes define specific cases to be tested in order to treat
corresponding requirement ‘covered’
 Requirements coverage by test cases report
 Source document change management (beta)
 Update requirements when the text of the standard is modified
 Requality is an open-source plugin to eclipse
 Integrated with version control software (SVN)
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 32
Sample view: Requality in
TTWorkbench
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 33
Requirements Tracking (1/2)
/***************************************************
** @author
STF 433
** @version 0.0.1
** @purpose 1:12, Ensure timer can be declared in components
** @verdict pass accept, ttcn3verdict:pass
***************************************************/
/*
* #reqname /Requirements/12 Declaring timers/Timers can be declared in component type
**/
module Sem_12_toplevel_timer_001 {
type component TComp{
timer t_timer:=10.0;
}
testcase TC_Sem_12_toplevel_timer_001() runs on TComp{
t_timer.start;
if (t_timer.running){
setverdict(pass)
}
else {
setverdict(fail)
}
}
control{
execute(TC_Sem_12_toplevel_timer_001())
}
}
TTCN-3 User Conference
ETSI Update on TTCN-3 Language Evolution and Tool
2012, Bangelore, India
Conformance Testing
Fully qualified name
of the requirement
Slide 34
Requirements Tracking (1/2)
 Requirements
coverage report
 Integrated into
Eclipse
 Evaluates
coverage of
requirements
and test
purposes
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 35
Suggested use case
 Add requirements markup
to TTCN3 standard
TTCN3
standard
updated
 Manual
 Introduce references
between requirements and
TTCN3 test cases
 Manual
 Track requirements
changes between TTCN3
standard revisions
 Automated (beta)
Changes
in reqs.
reviewed
ATS
updated
New
reqs.
added
 Track required changes in
TTCN3 ATS
 Manual, to be automated
TTCN-3 User Conference
2012, Bangelore, India
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 36
Thank You!
Questions?
ETSI Update on TTCN-3 Language Evolution and Tool
Conformance Testing
Slide 37

Download Report