An Approach to Measure
Validation of Agent-Based
Simulations!
Megan M. Olsen!
Mohammad Raunak!
Loyola University Maryland!
!
Swarmfest 2014!
!
Thanks to the
Clare Boothe Luce Program
Model & Simulation Has a Growing Impact!
2!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Must Validate to Trust Results!
§  V&V
evaluates a representation of
a “System Under Study” (SUS)!
§  Verification
- is internally
consistent with no known errors!
§  Validation
- mimics the SUS’s
behavior to a level of confidence
necessary for making the model
useful for its intended application!
3!
!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
© Law. WSC 2008.
Current Simulation Validation!
§  Validating
large scale simulations can be difficult!
§  Many techniques for validation exist!
Informal, static, dynamic, formal (Balci 1998)!
§  Historic data, sensitivity analysis, trace-based… (Sargent
2010)!
§ 
Analysis & Modeling
!
System
Under Study
Realization
Conceptual
Model
CM Validation
Simulation
Model
Verification
Operational Validation
4!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Construction
V&V
Experimentation
and Updating
Next Step: Quantifying the Level of Validation!
§  How
do we know enough validation has been
performed?!
§  How
can we compare validation between two different
simulation models?!
§  GOAL:
A validation coverage metric to quantify the
level of validation on a simulation model!
5!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Validation Coverage Overview!
§  Goal:
Systematize a quantization of validation on
simulations!
6!
Identify
Relevant
Aspects!
Assign Validation!
Weights!
Identify elements
to validate related
to an aspect!
Determine potential
validation
techniques!
Attempt to validate
with validation
techniques!
Compute
Validation
Coverage!
Determine
validatable
elements
Prepare
Validation
Validate
Calculate
coverage
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
A Guide to Elements to Validate!
§  Aspects
are comprised of !
input data (id)!
§  observable emergent information (oei)!
§ 
7!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Determine
validatable
elements!
Prepare!
Validation!
Validate
Calculate
coverage
Tasmanian Devils Simulation!
G. Fay, M. Olsen, J. Gran, O. Carja, J. Granka, A. Johnson, V. Weinberger. Agent-based model
of Tasmanian Devils examines spread of Devil Facial Tumor Disease due to road construction.
Proceedings of the International Conference on Complex Systems. 2011.
8!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Determine
validatable
elements!
Prepare!
Validation!
Validate
Calculate
coverage
Tasmanian Devils Simulation!
G. Fay, M. Olsen, J. Gran, O. Carja, J. Granka, A. Johnson, V. Weinberger. Agent-based model
of Tasmanian Devils examines spread of Devil Facial Tumor Disease due to road construction.
Proceedings of the International Conference on Complex Systems. 2011.
9!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Validation Coverage Overview!
§  Goal:
Systematize a quantization of validation on
simulations!
10!
Identify
Relevant
Aspects!
Assign Validation!
Weights!
Identify elements
to validate related
to an aspect!
Determine potential
validation
techniques!
Attempt to validate
with validation
techniques!
Compute
Validation
Coverage!
Determine
validatable
elements
Prepare
Validation
Validate
Calculate
coverage
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Determine
validatable
elements
Prepare!
Validation!
Validate
Calculate
coverage
Prepare to Validate – Assign Element Weights!
!§ Importance of elements may vary!
r11
r1 2
r1 3
r2 1 r2 2 r2 3 r2 4
!
11!
7/24/14!
Olsen, Raunak. An Approach to Measure Validation of Agent-Based
Simulations!
Determine
validatable
elements
Prepare!
Validation!
Validate
Calculate
coverage
Tasmanian Devils – Element Weights!
Element!
12!
7/24/14!
Label!
Weight!
Scent diffusion!
e11,1!
2!
Prey decay!
e11,2!
1!
Prey size!
e11,3!
2!
Rate of prey appearance!
e11,4!
1!
Following scent!
e21,1!
2!
Agent movement!
e21,2!
1!
World size!
e23,1!
1!
Rate of hunger!
e41,1!
2!
Prey decrease by agent eating!
e52,1!
1!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Determine
validatable
elements
Prepare!
Validation!
Validate
Calculate
coverage
Prepare to Validate – Validation Techniques!
§  Determine
weights for each validation technique!
!
13!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Determine
validatable
elements
Prepare!
Validation!
Calculate
coverage
Validate
Prepare to Validate – Validation Techniques!
§  Determine
appropriate validation techniques for
elements (oeis/ids)!
!
14!
Element!
Label!
Weight!
Applicable VTs!
Scent diffusion!
oei11,1!
2!
AN, RV, SA!
Prey decay!
id11,2!
1!
FV-id, GoF!
Prey size!
id11,3!
2!
FV-id, GoF!
Rate of prey appearance!
id11,4!
1!
FV-id, GoF!
Following scent!
oei21,1!
2!
AN, RV, SA!
Agent movement!
oei21,2!
1!
AN, RV, FV-oei!
World size!
id23,1!
1!
FV-id, GoF!
Rate of hunger!
oei41,1!
2!
AN, RV, FV-oei, SA!
Prey decrease by agent eating!
id52,1!
1!
FV-id, GoF!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Validation Coverage Overview!
§  Goal:
Systematize a quantization of validation on
simulations!
15!
Identify
Relevant
Aspects!
Assign Validation!
Weights!
Identify elements
to validate related
to an aspect!
Determine potential
validation
techniques!
Attempt to validate
with validation
techniques!
Compute
Validation
Coverage!
Determine
validatable
elements
Prepare
Validation
Validate
Calculate
coverage
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Determine
validatable
elements
Prepare!
Validation!
Validate!
Calculate
coverage
Validate Using Appropriate Techniques!
§  Track
which of the potential techniques have been
used and succeeded!
16!
Element!
Label! Weight!
Applicable VTs!
Applied VTs!
Scent diffusion!
oei11,1!
2!
AN, RV, SA!
AN, RV, SA!
Prey decay!
id11,2!
1!
FV-id, GoF!
FV-id, GoF!
Prey size!
id11,3!
2!
FV-id, GoF!
FV-id, GoF!
Rate of prey appearance!
id11,4!
1!
FV-id, GoF!
FV-id!
Following scent!
oei21,1!
2!
AN, RV, SA!
AN, RV!
Agent movement!
oei21,2!
1!
AN, RV, FV-oei!
AN, RV, FV-oei!
World size!
id23,1!
1!
FV-id, GoF!
FV-id, GoF!
Rate of hunger!
oei41,1!
2!
AN, RV, FV-oei,
SA!
AN, RV, FV-oei!
Prey decrease by agent eating!
id52,1!
1!
FV-id, GoF!
FV-id, GoF!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Validation Coverage Overview!
§  Goal:
Systematize a quantization of validation on
simulations!
17!
Identify
Relevant
Aspects!
Assign Validation!
Weights!
Identify elements
to validate related
to an aspect!
Determine potential
validation
techniques!
Attempt to validate
with validation
techniques!
Compute
Validation
Coverage!
Determine
validatable
elements
Prepare
Validation
Validate
Calculate
coverage
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Coverage Calculation!
§  Element
types (et) and elements (e) per aspect (va):!
vak={etk1,etk2,…,etki}!
§  etki={eki,1,eki,2 ,…,eki,j}!
§  Each eki,j is a specific id or oei with weight rkj,i!
§ 
§  Applicable
§ 
a(vkijl )=1 for each vkijl that has been successfully applied !
§  Validation
7/24/14!
validation techniques (v) with weights:!
Coverage:!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
18!
Determine
validatable
elements
Prepare!
Validation!
Validate!
Calculate
coverage!
Calculate Coverage!
Element!
Label! Weight!
Applicable VTs!
Applied VTs!
VT score!
Rate of prey!
id11,4!
1!
FV-id(2), GoF(3)!
FV-id!
2/5 = 0.4!
Following scent!
oei21,1!
2!
AN(1), RV(4), SA(2)!
AN, RV!
5/7= 0.71!
Rate of hunger!
oei41,1!
2!
AN, RV, FV-oei(2), SA! AN, RV, FV-oei!
(all other VT scores are 1)
𝑣𝑐=
0.4∗1+0.71∗2+0.78∗2+8/8+1+2+2 𝑣𝑐=11.38/13 =0.875=87.5%
19!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
7/9=0.78!
Validation Coverage Could Have Huge Impact !
§  Simulation
validation is crucial!
§  Coverage calculation quantifies simulation validation !
§  What if !
We always expected this much detail?!
§  We had a standard for comparing validation between models?!
§ 
20!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
The Tricky Parts!
§  How
do we !
Guarantee that weights are chosen well?!
§  Determine all elements have been found?!
§  Standardize applicable validation techniques?!
§  Institutionalize a validation coverage metric as a standard?!
§  Validate the validation metric?!
§ 
21!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!
Summary!
Thank You!
[email protected]
[email protected]
www.cs.loyola.edu
§  Simulation
validation is important!
§  We propose a validation coverage metric and
framework for quantifying simulation validation!
§  Metric could improve scientific quality of simulations!
! Identify
Relevant
Aspects!
Identify elements
to validate related
to an aspect!
Assign Validation!
Weights!
Determine potential
validation
techniques!
Attempt to validate
with validation
techniques!
Compute
Validation
Coverage!
Two publications: Winter Simulation Conference 2013, 2014 (forthcoming)
22!
7/24/14!
Olsen, Raunak. An Approach to Measure
Validation of Agent-Based Simulations!