ソフトウェア開発プロジェクトのリスク分析

Effort Estimation Based on
Collaborative Filtering
Profes 2004
-
Wed. 7 April 2004
Naoki Ohsugi, Masateru Tsunoda,
Akito Monden, and Ken-ichi Matsumoto
Graduate School of Information Science,
Nara Institute of Science and Technology
Software Development Effort Estimation
• There are methods for estimating required efforts to
complete ongoing software development projects.
• We can conduct the estimation based on past
projects’ data.
Cow
Wed. 7 April 2004
Profes 2004
2 of 19
Problems in Estimating Effort
• Past project’s data usually contain many Missing
Values (MVs).
–
Briand, L., Basili, V., and Thomas, W.: A Pattern Recognition Approach for Software Engineering Data
Analysis. IEEE Trans. on Software Eng., vol.18, no.11, pp.931-942 (1992)
• MVs give bad influences to accuracy of estimation.
–
Kromrey, J., and Hines, C.: Nonrandomly Missing Data in Multiple Regression: An Empirical Comparison
of Common Missing-Data Treatments. Educational and Psychological Measurement, vo.54, no.3,
pp.573-593 (1994)
Cow? Horse?
?
Wed. 7 April 2004
Profes 2004
?
3 of 19
Goal and Approach
• Goal: to achieve accurate estimation using data with
many MVs.
• Approach: to employ Collaborative Filtering (CF).
– Technique for estimating user preferences using data with
many MVs (e.g. Amazon.com)
Wed. 7 April 2004
Profes 2004
4 of 19
CF based User Preference Estimation
• Evaluating similarities between the target user and
the other users.
• Estimating the target preference using the other
users’ preferences.
Book 1
Book 2
Book 3
Book 4
Book 5
5
5
1
3
5
?
User A
User B
User C
User D
(prefer)
(prefer)
(not prefer)
(so so)
(target)
(prefer)
5
5
?
3
5
Similar User
(prefer)
(prefer)
(MV)
(so so)
(prefer)
?
5
1
?
5
Similar User
Dissimilar User
(MV)
(prefer)
(not prefer)
(MV)
(prefer)
1
?
?
5
1
(not prefer)
Wed. 7 April 2004
Estimate
(MV)
(MV)
(prefer)
Profes 2004
(not prefer)
5 of 19
CF based Effort Estimation
• Evaluating similarities between the target project and
the past projects.
• Estimating the target effort using the other projects’
efforts.
Project A
Project B
Project
type
1
# of
faults
Design
cost
Coding
cost
60
100
50
1
40
?
40
20
Similar Project
?
50
80
?
30
Similar Project
0
?
?
200
80
Dissimilar Project
(new develop)
(new develop)
Project C
Project D
(MV)
(maintain)
Wed. 7 April 2004
(MV)
(MV)
(MV)
Profes 2004
(MV)
Testing
cost
?
25
(target)
Estimate
6 of 19
Step1. Evaluating Similarities
# of faults
• Each project is
represented as a vector
of normalized metrics.
• Smaller angle between
2 vectors denotes
higher similarity
between 2 projects. Coding cost
Project B
0
Project A
# of
faults
60
Design
cost
100
(1.0)
(0.0625)
(target)
Project B
1
40
?
40
20
Wed. 7 April 2004
(1.0)
(1.0)
(0.0)
Coding
cost
50
Project type
Project
type
1
(1.0)
(MV)
Profes 2004
Project A
(0.0)
Testing
cost
?
(0.0)
Similarity: 0.71
7 of 19
Step2. Calculating Estimated Value
• Choosing similar k-projects.
– k is called Neighborhood Size.
• Calculating the estimated value from weighted sum of
the observed values on the similar k-projects.
Project A
Project B
Project
type
1
# of
faults
Design
cost
Coding
cost
60
100
50
1
40
?
?
50
80
0
?
?
(new develop)
(new develop)
Project C
Project D
(MV)
(maintain)
Wed. 7 April 2004
(MV)
(MV)
(MV)
Profes 2004
Testing
cost
?
25
Estimate (k=2)
40
20
Similarity: 0.71
?
30
Similarity: 0.71
200
80
Similarity: 0.062
(MV)
(target)
8 of 19
Case Study
• We evaluated the proposed method, using data
collected from large software development company
(over 7,000 employees).
– The data were collected from 1,081 projects in a decade.
• 13% projects for developing new products.
• 36% projects for customizing ready-made products.
• 51% projects were unknown.
– The data contained 14 kinds of metrics.
• Design cost, coding cost, testing cost, # of faults, etc., ...
Wed. 7 April 2004
Profes 2004
9 of 19
Unevenly Distributed Missing Values
Metrics
Rate of MVs
Mainframe or not
75.76%
New development or not
7.49%
Total design cost (DC)
0.00%
Total coding cost (CC)
0.00%
DC for regular staffs of a company
86.68%
DC for dispatched staffs from other companies
86.68%
DC for subcontract companies
86.59%
CC for regular staffs
86.68%
CC for dispatched staffs
86.68%
CC for subcontract companies
86.59%
# of faults found in the review of conceptual design
83.53%
# of faults found in the review of functional design
70.77%
# of faults found in the review of program design
80.20%
Testing cost
Wed. 7 April 2004
0.00%
Total
Profes 2004
10 of 19
59.83%
Evaluation Procedure
1. We divided the data into 50-50 two datasets
randomly; Fit Dataset and Test Dataset
2. We estimated Testing Costs in the Test Dataset using
the Fit Dataset.
3. We compared the estimated Costs and the actual
Costs.
Original Data
1081
projects
Wed. 7 April 2004
Fit Dataset
541
projects
used
Estimated
Testing Costs
divided
compared
Test Dataset
Actual
540
Testing Costs
extracted
projects
Profes 2004
11 of 19
Regression Model We Used
• We employed stepwise metrics selection.
• We employed the following Missing Data Treatments.
– Listwise Deletion
– Pairwise Deletion
– Mean Imputation
Wed. 7 April 2004
Profes 2004
12 of 19
Relationships Between the Estimated Costs
and the Actual Costs
Regression
(Listwise Deletion)
100
100
100
100
10
10
10
10
11
11
Costs
Costs
ActualActual
Costs
Costs
ActualActual
CF (k = 22)
0.1
0.1
0.01
0.01
0.001
0.001
0.0001
0.0001
0.0001 0.001
0.001
0.0001
0.01
0.01
0.001
0.001
0.01
0.01
0.1
0.1
11
10
10
100
100
0.0001
0.0001
0.0001 0.001
0.001
0.0001
0.01
0.01
0.1
0.1
11
10
10
100
100
Estimated Costs
Estimated Costs
Estimated Costs
Wed. 7 April 2004
0.1
0.1
Estimated Costs
Profes 2004
13 of 19
Evaluation Criteria of Accuracy
• MAE: Mean Absolute Error
• VAE: Variance of AE
Absolute Error =|Estimated Cost – Actual Cost |
• MRE: Mean Relative Error
• VRE: Variance of RE
|Estimated Cost – Actual Cost |
Relative Error =
Actual Cost
• Pred25
– Ratio of the projects whose Relative Errors are under 0.25.
Wed. 7 April 2004
Profes 2004
14 of 19
Accuracy of Each Neighborhood Size
• The most accurate estimation was observed at k = 22.
MRE = 0.82 (k = 22)
0.89
Mean Relative Error
0.88
0.87
0.86
0.85
0.84
0.83
0.82
0.81
1
5
10
15
20
25
30
35
40
45
50
Neighborhood Size
Wed. 7 April 2004
Profes 2004
15 of 19
Accuracies of CF and Regression Models
• All evaluation criteria indicated CF (k=22) was the
most effective for our data.
MAE
VAE
MRE
VRE
Pred25
CF (k = 22)
0.21
2.2
0.82
3.45
36%
Regression
(Listwise Deletion)
0.7
16.45
30.22
287581.18
10%
52.75 97171.84 6344.27 18937623098
12%
Regression
(Pairwise Deletion)
Regression
(Mean Imputation)
Wed. 7 April 2004
1.33
5.07
Profes 2004
331.69 24208218.49
4%
16 of 19
Related Work
• Analogy-based Estimation
– It estimates effort using values of the similar projects.
• Shepperd, M., and Schofield, C.: Estimating Software Project Effort
Using Analogies. IEEE Trans. on Software Eng., vol.23, no.12, pp.76743 (1997)
– They had another approach to evaluate similarities between
projects.
– They never mentioned missing values.
Wed. 7 April 2004
Profes 2004
17 of 19
Summary
• We proposed a method for estimating software
development efforts using Collaborative Filtering.
• We evaluated the proposed method.
– The results suggest the proposed method has possibility for
making good estimation using data including many MVs.
Wed. 7 April 2004
Profes 2004
18 of 19
Future Work
• Designing the method to find appropriate
neighborhood size automatically.
• Improving accuracy of estimation by other similarity
evaluation algorithms.
• Comparing accuracies to another methods (e.g.
analogy-based estimation).
Wed. 7 April 2004
Profes 2004
19 of 19

Download Report