13/10/2014 www.realdolmen.com REPORT ON… WHAT ARE WE REPORTING ON? MIKE MEURS & MAGDALENA STRAS “IF YOU MEASURE THE WRONG THING, AND REWARD THE WRONG THING, DON’T BE SURPRISED IF YOU GET THE WRONG THING” LEE COPELAND OCTOBER 13, 2014 | SLIDE 1 THE SPEAKERS Mike Meurs Senior QA Lead @ Realdolmen 9 Years in the testing business Laboratory technician – Biotechnology Dedicated Parent Passioned about testing Hiking OCTOBER 13, 2014 | SLIDE 2 1 13/10/2014 THE SPEAKERS Magdalena Stras Senior Business Analyst / Process Architect @ bpost Diplomat and starting interior designer Dedicated Parent Devoted hiker and Metallica fan OCTOBER 13, 2014 | SLIDE 3 AIM OF THE PRESENTATION Give an insight in the traditional test report and it’s pitfalls p your controls on test y Demonstrate alternatives to improve management and qualitative project delivery Illustrate that a silver bullet for reporting does NOT exist Answer questions that may rise during this presentation OCTOBER 13, 2014 | SLIDE 4 2 13/10/2014 FIRST TOPIC: REPRESENTATION OF “FACTS” Diff Different perspectives i off reality li How and what we measure in reality to represent “facts” to others OCTOBER 13, 2014 | SLIDE 5 CONCEPTUAL: TRAFFIC OCTOBER 13, 2014 | SLIDE 6 3 13/10/2014 TRAFFIC INFORMATION IN BARS OCTOBER 13, 2014 | SLIDE 7 TRAFFIC INFORMATION PROGRESS OCTOBER 13, 2014 | SLIDE 8 4 13/10/2014 ONE PICTURE THAT SAYS EVERYTING OCTOBER 13, 2014 | SLIDE 9 INTO DETAILS - LEUVEN OCTOBER 13, 2014 | SLIDE 10 5 13/10/2014 INTO DETAILS - BRUSSELS OCTOBER 13, 2014 | SLIDE 11 INTO DETAILS - ANTWERP OCTOBER 13, 2014 | SLIDE 12 6 13/10/2014 CONCRETE: A WEB SHOP… Back end Front end OCTOBER 13, 2014 | SLIDE 13 CURRENT TEST REPORTS 1 2 3 1 2 3 1 2 3 OCTOBER 13, 2014 | SLIDE 14 7 13/10/2014 CURRENT TEST REPORTS OCTOBER 13, 2014 | SLIDE 15 CURRENT TEST REPORTS 1 2 3 1 2 3 1 2 3 OCTOBER 13, 2014 | SLIDE 16 8 13/10/2014 CURRENT TEST REPORTS OCTOBER 13, 2014 | SLIDE 17 SECOND TOPIC: REPORT DRIVEN TESTING Goodhart’s law - “When a measure becomes a target, it ceases to be a good measure.” Testcases written per day VS planned Testcases executed per day VS planned % Test coverage Amount of defects reported Amount of defects re-opened Duration time of defect fixes applied … Easy tasks Surficial approach Quantity Complex tasks Thorough approach Quality OCTOBER 13, 2014 | SLIDE 18 9 13/10/2014 BASIC INGREDIENTS OF A REPORT. A REPORT IS… …brief and to the point …logical and precise …objective (use of safety language) OCTOBER 13, 2014 | SLIDE 19 …created with images and supported by words SPECIFIC INGREDIENTS OF A TEST REPORT Overall Product quality (incl. risks – issues) Test appraoch and efficiency Test related defects (see previous slides) OCTOBER 13, 2014 | SLIDE 20 10 13/10/2014 TEST REPORT – PRODUCT QUALITY Logistics Cross and Up selling Product Search Stock Mgmt Application back end Application front end Shopping basket Log in Transactions No important issues found Testing in progress Proposed Testing done Order Mgmt Client Mgmt BI reports CRM System Critical issues found Issues found Testing blocked Out of scope Testing not started OCTOBER 13, 2014 | SLIDE 21 TEST REPORT – PRODUCT QUALITY (DETAIL) Fill basket from product sea c search Delete items from basket Save Shopping basket Shopping basket Add from whislist Testing in progress Proposed Testing done Continue to payment module Return to Product search Price P i calculation engine No important issues found Restore saved d Shopping Basket Issues found Critcal issues found Testing not started Testing blocked c Out of scope OCTOBER 13, 2014 | SLIDE 22 11 13/10/2014 TEST REPORT – PRODUCT QUALITY (DETAIL) Pay with money transfer Pay with Visa Payment Pay with Pay Pal Continue to order d a ag management No important issues found Testing in progress Proposed Testing done Issues found Critical issues found Testing not started Testing blocked Out of scope OCTOBER 13, 2014 | SLIDE 23 TEST REPORT – PRODUCT QUALITY (DETAIL) Cancel stock order Order new stock Interface Logistics Receive stock Continue to product search No important issues found Testing in progress Proposed Testing done Issues found Critical issues found Testing not started Testing blocked c Out of scope OCTOBER 13, 2014 | SLIDE 24 12 13/10/2014 OTHER WAYS TO VISUALLY REPORT Business processes Data models Functional flows … OCTOBER 13, 2014 | SLIDE 25 TEST REPORT – TEST APPROACH AND EFFICIENCY How was testing conducted? Test approach Setup of the test team Focus points Can we test more efficient? Test data Usage of stubs/ drivers… Automation User roles … What did/will not be tested and are the underlying risks? OCTOBER 13, 2014 | SLIDE 26 13 13/10/2014 A LOOK INTO THE FUTURE The visual dashboard is directly linked to the code A bug g found,, is p placed automatically on the map with an automatic priorisation You can zoom until the code to find the exact location of g the bug Developers and project managers use the map as a daily tool to resolve the bugs in the correct priority. …. Just like Touring Mobilis… OCTOBER 13, 2014 | SLIDE 27 VISUAL REPORTING: TRAFFIC MAP Implementation in pre-UAT and UAT testing project combo, Shop&Deliver, bpost 28 14 13/10/2014 CONTEXT Innovative, high-visibility project Lean startup Small, multi-functional BU team + internal BE development + offshore FE team in India OCTOBER 13, 2014 | SLIDE 29 WE ARE TESTING pre-UAT and UAT obtaining and presenting to different levels (Staff, process managers, BUs and internal testers) info on: QUALITY of software vs requirements compatibility with and usability of PROCESSES AVAILABILITY for testing by wider audience OCTOBER 13, 2014 | SLIDE 30 15 13/10/2014 THE STANDARD REPORTING TELLS US How many & which test cases have been executed By whom & when How many defects have been open related to test cases Have the defects been closed before go-live and also: How many defects were open per test case Average resolution time per defect severity Success rate in fixing defects (reopening ratio) etc. OCTOBER 13, 2014 | SLIDE 31 SO WE TRIED THIS: Overview Announcement Cancellation Merchant Portal Merchant Portal Log in & credentials management Superuser functionality Shipments management Merchant & Shop management Consumer Portal Labels printing Ordering Batch order upload Merchant data B k ffi P Back office Portal l Appointment management Shop overview Order status Consumer data SDL Merchant & Shop registration OCTOBER 13, 2014 | SLIDE 32 16 13/10/2014 THE RESULTS Great overview, but why is this still red??? So which scenarios are testable? What is the forecast? When will it be ready? What are the defects? How severe are they? Can we go live if the road is still yellow? OCTOBER 13, 2014 | SLIDE 33 SO WE DECIDED TO PLAY TRAFFIC INFO… Making a full map of our application landscape OCTOBER 13, 2014 | SLIDE 34 17 13/10/2014 … ALL THE WAY, INCLUDING LIVE EVENT OVERVIEW Functionality level Cancellation Blocking defect detected on intake test (or systest) Non-blocking defect detected on intake test or systest / open tasks Low severityy or no defects detected on intake tests Not sufficiently tested yet Process level Testing not possible for wider BU group Log in Testing possible only according to limited scenarios / parameters Testing possible according to all scenarios / parameters Timeslot picking More testing needs to be performed for assessment Issue level Unknown cause / high severity & high complexity issue / regression issue Offshore development issue (defect assigned to the team in India) Internal development issue (defect assigned to the internal development team) Known low impact issue or fix to be implemented OCTOBER 13, 2014 | SLIDE 35 HOW DOES IT WORK IN PRACTICE? 4. As defects are being resolved (or not) notation is adjusted 1.Scope, initial traffic lights test effort presented on map 2.During testing “roads” areand colour-marked… 3. Issue signs are assigned and traffic lights are changed OCTOBER 13, 2014 | SLIDE 36 18 13/10/2014 WHAT DO WE GAIN? Real-time info about issues and their status / complexity S which So hi h scenarios i are testable? t t bl ? Great overview, but why is this still red??? IIssue indication i di ti & description as indication of testable variants What is the forecast? When will it be ready? Possibility to estimate resolution time based on types of issues Better view on defect What are the defects? How severe are they? location, severity and effort needed to fix Can we go live if the road is still yellow? Better understanding of the impact of defects / open issues OCTOBER 13, 2014 | SLIDE 37 OCTOBER 13, 2014 | SLIDE 38 19
© Copyright 2024 ExpyDoc
