PowerPoint

Local Bias and its Impacts on the
Performance of Parametric Estimation
Models
Accepted by PROMISE2011 (Best paper award)
Ye Yang, Lang Xie, Zhimin He (iTechs)
Qi Li, Vu Nguyen, Barry Boehm (USC)
Ricardo Valerdi (MIT)
Agenda






Background
Research questions
Measuring local bias
Measuring the impacts of local bias
Handling Local Bias
Conclusions and future work
2
Background
 COCOMO II model
 Proposed by Dr. Barry Boehm; one of the most accurate cost
estimation models; widely adopted by industry.
 Typical parametric estimation model, need tune parameters
against local data (local calibration)
Effort  A  Size
B  0.01
5
 SFi
i 1
17
  EM j
j 1
Organization 1
General Model
Organization 2
3
Background (Cont.)
 Model usage circle
 Local calibration relies on local historical data and domain
knowledge, i.e. with local assumptions.
 In most cases, such local assumptions vary from the general
model assumptions. It is possible that the mismatches between
“general assumptions” and “local assumptions” will result in
surprising calibration results.
 E.g., counter-intuitive
Model
Local data
Local
Localization
calibration results: negative
assumptions
values of regression
Underlying
model
coefficients for level of
General
programmer capability
assumptions
Model
Model
(PCAP), indicating higher
Building
Usage
Historical data
PCAP leads to higher effort.
Model
updates
Model
Calibration
Calibration
data
4
Research questions
 Research questions:
 Is there a way to measure the local bias introduced in the model
localization (local calibration) stage?
 As the historical data accumulates from multiple companies, how
will the associated local bias impact the performance of the
general parametric estimation model?
 Are there any correlation patterns between local bias and model
performance variation after incorporating local dataset into the
calibration dataset?
 Assumptions:
 The general parametric model follows a similar structure as the
COCOMO II.
 In model localization stage, constant A and constant B are tuned
with local data.
 In model usage stage, locally calibrated A and B are used for
project estimation.
5
Measuring local bias
 Definition of local bias:
Effort '
A'
localbias | ln(
) || ln( )  ( B ' B) ln( Size) |
Effort
A
 where A’and B’are model parameters calibrated from local data
of each organization, A and B are default constant values of
COCOMO II model (A=2.94, B=0.91), and in our study we set
Size=100KLOC.
6
Measuring local bias (cont.)
 Data sets
 CII 2010 data set; contains two subsets: the CII2000 subset (161
data points from 16 organizations) and the After2000 subset (92
additional data points newly collected from 10 different
organizations since year 2000)
Characteristics
# data points
# organizations
min
Size(KSLOC) max
median
min
Effort(PM) max
median
CII2000
Subset
161
16
2.6
1292.8
46.92
6
11400
192.5
CII 2010
Dataset
253
23
1.68
2505.2
45.28
3.5
11400
170
7
Measuring local bias (cont.)
 Analysis procedure
 Divide After2000 subset into 10 groups according to their
corresponding organization.
 For each group, we conduct a representative local calibration
using data in that group only and produce its local A’ and B’.
 Calculate the corresponding local bias value of each group.
 Compare local bias values among all groups.
CII 2010
Dataset
After2000
Subset
CII 2000 Subset
Group by Organization_ID
Subset
1
Subset
2
A1’, B1’
A2’, B2’
…
local_bias1 local_bias2
Default
Constants:
A, B
Subset
n
An’, Bn’
local_biasn
A, B
8
Measuring local bias (cont.)
Parameters of local models:
Local bias of each group:
 Different local A and B in each group, indicating local bias introduced when
adopting local calibration;
 Local bias varies in different group, ranging from 0.06 to 2.25; the local bias
measures how much relative error the corresponding local model will produce.
9
Measuring the impacts of local bias
 Analysis procedure
 First, for each group ssi in the After2000 subset:
1. combine ssi with CII 2000 data set to produce a new data set
dsi ;
2. Assessing model performance on data set dsi , record values
of performance indicators;
 Then conduct correlation analysis between local bias and model
performance
CII 2000 subsetI
SS1
Performance
Local bias
CII 2000 subsetI
SS2
Performance
Local bias
……
……
……
Correlation analysis
10
Measuring the impacts of local bias
 Performance assessment
 Basic performance indicators: MMRE (mean MRE), stdMRE (the
variance of MRE)
 Assessment procedure:
Spliting data set
into training set
and test set
Tuning model
parameters on
training set
Repeat the above
steps for 2000 times
2000 (MMRE,
stdMRE) pairs
Evaluating model
performance on
test set
MMRE, stdMRE
Average MMRE
Range of MMRE
Average stdMRE
Range of stdMRE
 In our study, we employ Average MMRE, Range of MMRE, Average
stdMRE, and Range of stdMRE to assess the performance of an
estimation model.
11
Measuring the impacts of local bias(cont.)
 Model performance
12
Measuring the impacts of local bias(cont.)
 Spearman correlation coefficients between local bias
and model performance:
Local bias
Local bias
*num
Range of
stdMRE
Average
stdMRE
Range of
MMRE
Average
MMRE
Correlation
Coefficient
0.7787
0.1677
0.4731
0.1671
p-value
0.0080
0.6435
0.1673
0.6455
Correlation
Coefficient
0.6120
0.8085
0.4731
0.6777
p-value
0.0508
0.0046
0.1673
0.0313
 At the significant level of p-value less than 0.05, the range of
stdMRE is significantly positive correlated with local bias and
local_bias*num. Both the average stdMRE and the average MMRE
are significantly positive correlated with local_bias*num.
 Range of stdMRE reflects the uncertainty of model performance.
Hence, the bigger the local bias is, the weaker the performance is.
13
Handling Local Bias
 Motivation
 Performance of the general COCOMO II model seriously decrease
on the After2000 subset!
 Need to calibrate a new version of COCOMO II model on the CII
2010 data set.
/通用格式
Pred(20)
CII2000
After2010
CII2010
0.6211
0.2393
0.4822
Pred(25)
0.6957
0.3152
0.5573
Pred(30)
0.7516
0.3696
0.6126
/通用格式
/通用格式
/通用格式
CII2000
/通用格式
After2010
/通用格式
CII2010
/通用格式
/通用格式
/通用格式
Pred(20)
Pred(25)
Pred(30)
14
Handling Local Bias (cont.)
 Local bias handling approach
 Assumption： local historical data set with higher local bias
presents more different pattern for cost estimation, and it should
be assigned a lower weight when being used for model
calibration.
 Constraints for weight distribution function Weight=F ( LocalBias )
 IF LocalBias =0, THEN Weight =1;
 IF LocalBias → +∞, THEN Weight → 0;
 The F should be a decreasing function on interval [0, +∞).
 Three functions
/通用格式
/通用格式
/通用格式
/通用格式
/通用格式
/通用格式
/通用格式
/通用格式
/通用格式
/通用格式
/通用格式
1
e LocalBias
/通用格式
/通用格式
F 3：Weight 
/通用格式
/通用格式
1
F 2：Weight 
1ln( LocalBias )
1/(X+1)
1/ln(X+1)+1
1/E^x
/通用格式
/通用格式
1
F 1：Weight 
LocalBias 1
15
Handling Local Bias (cont.)
 Weight assigned to each organization
OID
LocalBias
46
0.03952465
595
F1
F2
F3
/通用格式
0.96197815 0.962682998 0.961246259
0.20628288 0.828992947 0.842074323 0.813602892
/通用格式
179 0.276339409 0.783490655 0.803861012 0.758555426
/通用格式
14 0.302222798 0.767917749 0.791093772
LocalBias
0.73917336
1/(X+1)
106 0.302948518 0.767490032 0.790745252 0.738637122
1/[ln(X+1)+1]
/通用格式
1/e^X
99 0.568446596 0.637573509 0.689614414 0.566404611
93 0.726123018 0.579332985 0.646881635 0.483780969
590 1.009427633
/通用格式
0.49765415 0.588980208 0.364427506
599 1.820976691 0.354487154 0.490897974 0.161867579
/通用格式
1
2
3
4
5
6
7
8
9
10
597 2.190999501 0.313381434 0.462891345 0.111804944
16
Handling Local Bias (cont.)
 Model performance on the CII2000 subset
Equal
COCOMO Weights
F1
F2
F3
Pred(20) 0.6211
0.4907
0.5093
0.5031
0.5342
Pred(25) 0.6957
0.5963
0.6149
0.6149
0.6273
Pred(30) 0.7516
0.677
0.7205
0.7143
0.7081
/通用格式
/通用格式
/通用格式
COCOMO II
/通用格式
Equal weights
/通用格式
Function-1
/通用格式
 Model calibrated with equal
weights performs worst on the
CII2000 subset;
 The general COCOMO II model
performs best;
Function-2
Function-3
/通用格式
/通用格式
/通用格式
Pred(20)
Pred(25)
Pred(30)
17
Handling Local Bias (cont.)
 Model performance on the After2000 subset
COCOMO
Equal
Weights
F1
F2
F3
Pred(20)
0.2393
0.25
0.2609
0.2717
0.2609
Pred(25)
0.3152
0.3261
0.3261
0.3261
0.3152
Pred(30)
0.3696
0.3804
0.4022
0.4022
0.4022
/通用格式
/通用格式
/通用格式
/通用格式
COCOMO II
/通用格式
Equal weights
Function-1
/通用格式
Function-2
/通用格式
 The general COCOMO II model
performans worst on the After
2000 subset
 Models calibrated with weights
exhibit better performance
than models calibrated without
weights.
Function-3
/通用格式
/通用格式
/通用格式
Pred(20)
Pred(25)
Pred(30)
18
Handling Local Bias (cont.)
 Model performance on the whole CII 2010 data set
COCOMO
Equal
Weights
F1
F2
F3
Pred(20)
0.4822
0.4032
0.419
0.419
0.4348
Pred(25)
0.5573
0.498
0.5099
0.5099
0.5138
Pred(30)
0.6126
0.5692
0.6047
0.6008
0.5968
/通用格式
/通用格式
/通用格式
COCOMO II
/通用格式
Equal weights
Function-1
/通用格式
Function-2
Function-3
/通用格式
/通用格式
 The general COCOMO II model
works better on the whole CII
2010 data set than calibrated
models;
 Models calibrated with weights
exhibit better performance than
models calibrated without
weights.
/通用格式
Pred(20)
Pred(25)
Pred(30)
19
Conclusions
 The proposed LocalBias measure can be used to
quantitatively measure and analyze potential local
bias associated with individual organization data
subset in the overall dataset.
 As historical data accumulates from multiple
companies, the associated local bias will cause the
range of stdMRE increase.
 The correlation analysis verifies that the model
performance is significantly correlated by the
degree of local bias and the number of data points
associated with each additional group.
 Weight calibration helps to reduce impact of local
bias and thus improve the usability of crosscompany data for model calibration.
20
Future work
 More empirical studies on other public dataset to
future validate and refine results.
 Develop more effective methods for reducing local
bias and improving general calibration outcomes.
21
Thanks!
Q&A

Download Report