19_chapter 9

EXPERIMENTAL WORK PART II
Chapter 9
Model implementation and
evaluation in a passenger car
The final machine learning models derived at the concluding part of
Chapter 8 are implemented on to a passenger vehicle. Misfire
detection in passenger car is analysed in Section 9.2 followed by
model extension to accommodate VFD in Section 9.3 and Section 9.4
presents the concluding remarks on model performance in a
passenger car.
169
CHAPTER 9 MODEL IMPLEMENTATION AND EVALUATION IN A PASSENGER CAR
9.1 INTRODUCTION
The models DT-CFS-KD and RF-CFS-KD developed and extensively analysed in part I
was designed for implementation in a passenger car. The misfire detection model has to be
implemented in a passenger car operating on actual road conditions and the results have to
be thoroughly analysed before adapting the system for real world applications. The
capabilities of the model to accommodate model extension for detecting simultaneous
misfire and performance at varied speed and load conditions were encouraging, referring to
the results presented in Chapter 8. However, the interaction of multiple systems in an
automobile running on actual road conditions could induce vibration components that have
not been encountered in the stand-alone test bench based misfire detection model. These
new components in the signal could interfere with the learning process of the developed
model and cause errors since they were developed using signals acquired from the engine
test bed operating at no load, a condition in which misfire detection is very difficult. But
while implementing in a passenger car, this condition is not feasible since the dead weight
of the vehicle is always available as a constant load and is multiple times higher than the
payload in a passenger car. All these system changes might alter the model sensitivity and
could lead to necessary retuning of the model.
The addition of new class (or event) to be detected is also evaluated by incorporating air
filter chocking, gear knock, engine over speed (indicating a need to change to a higher
gear) and low tyre inflation pressure in addition to misfire. This will enable the vehicle
management system to be developed over the existing misfire detection model. The
experimental set up in the passenger car for misfire simulation and other fault simulations
are as described in Section 5.3.2. The model implementation results are discussed in the
following section.
The two distinct models developed using statistical and histogram based features need to
be analysed for their adaptability to work under diversified conditions when incorporating
additional classes. The performance of histogram based model was observed to be a little
170
lower than the expected value and it could be reassessed for inclusion or exclusion based
on the performance in the real time situation.
9.2 MODEL IMPLEMENTATION FOR MISFIRE DETECTION
The misfire simulation is as mentioned in Section 5.3.2 and was simulated at 40
Kilometers per hour (Kmph). The statistical and histogram features were extracted from
the time series vibration signature of the engine block measured on top of the engine block.
The model implementation results are analysed in the following sections. A vibration
signature plot of no-misfire and misfire conditions recorded from the Suzuki passenger car
Vibration amplitude
operating at 40 Kmph is presented in Figures 9.1 and 9.2 respectively.
3.E-01
2.E-01
1.E-01
0.E+00
-1.E-01
-2.E-01
-3.E-01
-4.E-01
-5.E-01
Good (No misfire)
0
1000
2000
3000
4000
5000
6000
7000
8000
7000
8000
Sample number
Figure 9.1 No misfire plot for Suzuki passenger car
Vibration amplitude
2.E-01
Misfire
2.E-01
1.E-01
5.E-02
0.E+00
-5.E-02
-1.E-01
-2.E-01
-2.E-01
0
1000
2000
3000
4000
5000
6000
Sample number
Figure 9.2 Misfire in cylinder 3 of Suzuki passenger car
A visual inspection of the plots above indicates that they are not comparable to the plots
obtained from the engine test rig using Ambassador engine. The disturbance due to misfire
171
in one cylinder is noticeable as a change in pattern in Figure 9.2 when compared to that of
the no misfire signal in Figure 9.1. The ability of the developed models to be sensitive to
these patterns of signals has to be evaluated.
9.2.1 Model analysis using statistical features
The evolved model chosen for implementation from Chapter 6 was CFS based FSS using
decision tree for secondary optimisation in which standard deviation, standard error,
skewness and range were selected as prominent features, followed by Konenenko
discretisation of data. But this decision was revised, when the results in Chapter 8
indicated that the secondary optimisation of the FSS was not effective in all cases. Hence
the DT-CFS-KD model based on statistical features using CFS based FSS followed by
Konenenko discretisation of data becomes the model of choice. Though decision tree was
identified as the first choice from the results presented in Chapter 6, it was replaced with
random forest, after reviewing the results presented in Chapter 8, where RF-CFS-KD using
random forest established itself as the most significant model of choice followed by
decision tree. Hence both the algorithms are taken for implementation study in the
passenger car.
Table 9.1 Classifier parameters for model in the passenger car
Parameters for evaluation
Decision tree
Random forest
Model performance evaluation
10-fold stratified
cross-validation
10-fold stratified
cross-validation
Model building time
0.01 s
0.01 s
Total Number of Instances
400
400
Correctly Classified Instances
400
400
Incorrectly Classified Instances
0
0
Classification accuracy
100 %
100 %
Mean absolute error
0
0
Root mean squared error
0
0
MDL correction
Incorporated
Incorporated
Number of leaves
2
-
Size of the tree
3 levels
10 trees
172
The performance parameters of the models using random forest and decision tree presented
in Table 9.1. The results are impressive for both the classifiers, reaching 100%
classification accuracy. A plot of statistical features using the two class data presented in
Figure 9.3 below depicts that the data are easily distinguishable in to two classes, most
often with a wide margin between the values of the two classes.
0.0004
Standard deviation
0.0003
0.0004
0.0002
0.0002
0.0001
0.0001
0
0
0
100
200
300
0
400
0.4
0.2
500
0
0
100
200
300
0.8
0.2
0
0
200
300
100
200
400
300
400
Median
0.001
0.0005
100
300
0.0015
0.4
0
200
Kurtosis
0
400
Range
0.6
100
1000
Maximum
0
Variance
0.0003
-0.0005
400
0
10
100
200
300
400
0.002
0
0
-10
-20
Skewness
-30
0
100
200
300
-0.002
Mode
-0.004
400
0
173
100
200
300
400
4.00E-04
0
2.00E-04
-0.2
0.00E+00
-0.4
-2.00E-04
Mean
-4.00E-04
0
100
200
300
Minimum
-0.6
0
400
100
200
300
400
Figure 9.3 Plot of statistical features (Passenger car)
The features, standard deviation, variance, kurtosis, range, median, skewness and
minimum show a sharp change in values after 200 signals indicating easy classification
between the two classes. This also indicates that any one of these features will be sufficient
to detect misfire. The CFS algorithm did not offer any subset; instead it recommended the
use of all features. This is due to the fact that cross correlation was not established within
the features considered. The feature plot revealed that a single feature was sufficient. The
challenge arises when multiple classes or instances have to be detected using the same
signal feature and when the difference in vibration pattern between cases closely match.
Both random forest and decision tree, with any one of the features discussed above or all
the features, returns a classification accuracy of 100%. A representative confusion matrix
is presented in Table 9.2.
Table 9.2 Confusion matrix for statistical features
Good
Misfire
200
0
Good
0
200
Misfire
These results prove that the developed model is capable of identifying misfire in a
passenger car. However the model has to be tested for its capabilities as a multi-class
vehicle management system using the single accelerometer input.
174
9.2.2 Model analysis using histogram features
The histogram models validated in Chapter 7 could not hold the test of diversification and
had to be removed from active consideration. But from the feature plots presented in
Figure 9.3, it is clearly evident that histogram could also perform satisfactorily for
detecting misfire in two class modes in a passenger car. The evaluation result proves that
DT-EWB with 10 bins record a peak performance of 100% classification accuracy using
decision tree and random forest. But DT-EFB5, EFB with five bins returned 94.5% and
95% classification accuracy for decision tree and random forest respectively. The model
using EWB with 10 bins could be considered for further evaluation. The parameters and
confusion matrix are not presented since they do not contain additional information for
evaluating the model.
9.3 ANALYSIS ON MODEL EXTENSION IN A PASSENGER CAR
The misfire model is extended to accommodate various other fault conditions and its
performance using statistical and histogram features are evaluated. The focus here is to
evaluate a single sensor based vehicle management system capable of identifying the
following faults:

Misfire (Mis1)

Air filter chocking (Choke)

Gear knock (GnoK)

Engine over speed (indicating a change to higher gear) (GrHiRpm)

Low tyre inflation pressure (20Psi)
3.E-01
2.E-01
1.E-01
0.E+00
-1.E-01
-2.E-01
-3.E-01
-4.E-01
-5.E-01
-6.E-01
Good
Vibration amplitude
Vibration amplitude
The vibration signature of all the simulated faults are presented in Figure 9.4
0
2000
4000
6000
2.E-01
2.E-01
1.E-01
5.E-02
0.E+00
-5.E-02
-1.E-01
-2.E-01
-2.E-01
Misfire
0
8000
2000
4000
6000
Sample number
Sample number
175
8000
Gear knock
1.E-01
Vibration amplitude
Vibration amplitude
2.E-01
0.E+00
-1.E-01
-2.E-01
-3.E-01
-4.E-01
-5.E-01
0
2000
4000
6000
4.E-01
3.E-01
2.E-01
1.E-01
0.E+00
-1.E-01
-2.E-01
-3.E-01
-4.E-01
-5.E-01
-6.E-01
8000
High engine rpm
0
Sample number
Choking
2.E-01
1.E-01
0.E+00
-1.E-01
-2.E-01
-3.E-01
-4.E-01
0
2000
4000
4000
6000
8000
Sample number
Vibration amplitude
Vibration amplitude
3.E-01
2000
6000
8000
3.E-01
2.E-01
1.E-01
0.E+00
-1.E-01
-2.E-01
-3.E-01
-4.E-01
-5.E-01
Low tyre pressure
0
Sample number
2000
4000
6000
8000
Sample number
Figure 9.4 Signal plots for various simulated faults in the Suzuki passenger car
9.3.1 Model extension analysis in a passenger car using statistical features
The performance of the model, when all the above mentioned conditions or classes are
implemented into the model, need to be thoroughly evaluated with a focus to maintain
100% misfire detection followed by maximum possible classification accuracy in all the
other classes, 100% being the ideal target. The statistical features coupled with CFS based
FSS followed by Konenenko discretisation when applied to misfire detection was capable
of identifying the misfire with a single feature as mentioned in Section 9.2. The model
extension study is performed at a vehicle speed of 40 Kmph and the conditions are
simulated as mentioned in Section 4.5.2.
The results presented in Table 9.3 are encouraging since an overall classification accuracy
of 92.8% and 94.6% were recorded for the model using decision tree and random forest as
a base classifier respectively. However, DT-CFS-KD based on decision tree records 99%
176
for misfire detection as compared to random forest model RF-CFS-KD which outperforms
with 100% classification accuracy. Based on this criterion random forest has an edge over
decision tree.
Analysing the results presented in Table 9.4 for random forest based RF-CFS-KD, it is
clearly evident that misclassification among good and misfire is zero considering first two
rows and first two columns. The misclassification of 17 good condition as choking and 8
as low tyre pressure is undesirable, but not critical. Misfire and engine running at low gear
high speed (GrHiRpm) are classified with an accuracy of 100% followed by gear knock
with 99% and low tyre pressure at 96.5% accuracy. The choking of air filter gets classified
with 84.5%, which is the least among all conditions. An important observation made here
is that none of these classes are misclassified as misfire or vice versa which increases the
confidence on the model capability.
Table 9.3 Classifier performance parameters for model extension in Suzuki car
Parameters for evaluation
Decision tree
Random forest
10-fold stratified
10-fold stratified
cross-validation
cross-validation
Model building time
0.2 s
0.3 s
Total Number of Instances
1200
1200
Correctly Classified Instances
1114
1135
86
65
Classification accuracy - Multi class
92.8 %
94.6 %
Classification accuracy - Two class
99 %
100 %
Mean absolute error
0.0328
0.0251
Root mean squared error
0.1401
0.1168
MDL correction
Incorporated
Incorporated
Number of leaves
139
-
155 levels
10 trees
Model performance evaluation
Incorrectly Classified Instances
Size of the tree
177
Table 9.4 Random forest confusion matrix for model extension in Suzuki car
Good
Mis1
GnoK
Choke
GrHiRpm
20Psi
175
0
0
17
0
8
Good
0
200
0
0
0
0
Mis1
0
1
198
0
0
1
GnoK
22
0
0
169
0
9
Choke
0
0
0
0
200
0
GrHiRpm
3
0
2
2
0
193
20Psi
The confusion matrix of DT-CFS-KD using decision tree as a base classifier, presented in
Table 9.5 performs a notch below that of the random forest with an additional challenge of
not being able to classify misfire with 100% accuracy, a mandatory requirement for the
design to be acceptable for real time implementation. Hence DT-CFS-KD could only be
recommended as a second choice.
Table 9.5 Decision tree confusion matrix for model extension in Suzuki car
Good
Mis1
GnoK
Choke
GrHiRpm
20Psi
168
0
1
21
0
10
Good
0
198
2
0
0
0
Mis1
0
1
197
1
0
1
GnoK
25
0
1
159
0
15
Choke
0
0
0
0
200
0
GrHiRpm
4
0
2
2
0
192
20Psi
All other inferences stated for RF-CFS-KD results presented in Table 9.4 holds good for
decision tree with corresponding change in values. The performances in all classes except
low gear high speed (GrHiRpm) is a little lower than the performance of random forest.
178
9.3.2 Model extension analysis in a passenger car using histogram features
The model using histogram based on 10 bin EWB was found to perform acceptably in two
class mode of the Suzuki car misfire detection, whereas the 5 bin EFB is not considered
due to poor performance. The result of the model using random forest is presented in Table
9.6. From the evaluation, it is evident that the misfire detection is done with an accuracy of
99.5%, but the results of predicting other classes are far from encouraging.
Table 9.6 Random forest confusion matrix with EWB for model extension in Suzuki car
Good
Mis1
GnoK
Choke
GrHiRpm
20Psi
142
0
18
29
5
6
Good
0
199
0
0
1
0
Mis1
19
0
83
27
4
67
GnoK
31
0
22
96
30
21
Choke
7
2
6
37
147
1
GrHiRpm
21
0
71
32
0
76
20Psi
Based on these results, it can be safely concluded that histogram based models are not
suitable for misfire and multiple fault detection in an automobile.
9.4 CONCLUSION
The analyses of implementing the developed models in a Suzuki passenger car, using
statistical and histogram features were done. The decision tree and random forest were
used as the base classifiers. The results can be summarised as follows:
a) Two-class mode:
The RF-CFS-KD model using statistical features performs misfire detection at 100%
accuracy with both the classifiers, proving that the developed model is capable of
identifying misfire in a passenger car.
179
The model using DT- EWB10 histogram features record a peak performance of 100% with
both decision tree and random forest, whereas DT-EFB5 returned around 95% for both the
classifiers. Hence the model using EWB with 10 bins alone is considered for further
evaluation.
b) Multi-class mode:
The decision tree performance was not comparable with random forest and hence it is not
considered. The histogram features returned very poor performance and hence, histogram
in any format is not recommended.
The model extension analysis reveals that RF-CFS-KD i.e. statistical feature based model
using random forest classifier, performs with 100% misfire detection and was also able to
classify most of the conditions with a good level of accuracy ranging between 100% and
96.5%, except for choking of air filter which gets classified with 84.5% accuracy. A major
advantage observed is that none of these classes are misclassified as misfire or vice versa,
which increases the reliability of the model.
180

Download Report