You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
495 lines
17 KiB
Markdown
495 lines
17 KiB
Markdown
# 앙상블(Enemble)이란?
|
|
앙상블(Ensemble) 학습은 여러 개의 머신러닝 모델을 결합하여 단일 모델보다 더 나은 성능을 얻기 위한 기법을 말한다
|
|
앙상블 기법은 여러 개의 약한 학습기(weak learner)를 결합하여 강한 학습기(strong learner)를 생성하는 아이디어에 기반한다.
|
|
앙상블 기법은 다양한 머신러닝 문제에서 높은 성능을 달성하는 데 매우 효과적이다.
|
|
다양한 모델의 특징과 장점을 결합하므로, 오버피팅(과적합)을 줄이고 일반화 성능을 향상시킬 수 있다.
|
|
|
|
|
|
# 앙상블 학습의 주요 기법:
|
|
|
|
1. **배깅(Bagging, Bootstrap Aggregating)**:
|
|
- 동일한 알고리즘에 대해 훈련 데이터의 서로 다른 부분 집합(subset)을 사용하여 여러 모델을 훈련한다.
|
|
- 모든 모델의 예측을 집계하여 최종 예측을 생성한다.
|
|
- ex) **랜덤 포레스트(Random Forest)**.
|
|
|
|
2. **부스팅(Boosting)**:
|
|
- 연속적으로 모델을 훈련시키면서, 이전 모델의 오류를 다음 모델이 보정한다.
|
|
- 모든 모델의 예측을 조합하여 최종 예측을 생성한다.
|
|
- ex) **AdaBoost, Gradient Boosting, XGBoost, LightGBM, CatBoost**
|
|
|
|
3. **스태킹(Stacking)**:
|
|
- 여러 다른 모델로부터의 예측을 취합하여, 그 예측들을 입력으로 사용하는 새로운 모델(메타 모델)을 훈련시킨다.
|
|
- 이 메타 모델이 최종 예측을 생성한다.
|
|
|
|
|
|
---
|
|
|
|
### 배깅(Bagging) 예제 코드[¶]()
|
|
|
|
<details>
|
|
<summary>Code View</summary>
|
|
|
|
<summary>Bootstrap Aggregating</summary>
|
|
<div markdown="1">
|
|
|
|
````python
|
|
import numpy as np
|
|
import matplotlib as mpl
|
|
import matplotlib.pyplot as plt
|
|
from sklearn.datasets import load_iris
|
|
from sklearn.tree import DecisionTreeClassifier
|
|
from sklearn.ensemble import BaggingClassifier
|
|
|
|
iris = load_iris()
|
|
X, y = iris.data[:, [0,2]], iris.target
|
|
|
|
model1 = DecisionTreeClassifier(max_depth =10, random_state=0).fit(X, y)
|
|
model2 = BaggingClassifier(DecisionTreeClassifier(max_depth=4), n_estimators=50, random_state=0).fit(X, y)
|
|
|
|
x_min, x_max = X[:,0].min() - 1, X[:,0].max() + 1
|
|
y_min, y_max = X[:,1].min() - 1, X[:,1].max() + 1
|
|
xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1), np.arange(y_min, y_max, 0.1))
|
|
|
|
plt.subplot(121)
|
|
Z1 = model1.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
|
|
plt.contourf(xx, yy, Z1, alpha=0.6, cmap=mpl.cm.jet)
|
|
plt.scatter(X[:,0], X[:,1], c=y, alpha=1, s=50, cmap=mpl.cm.jet, edgecolors="k")
|
|
plt.title("Decision tree")
|
|
plt.subplot(122)
|
|
|
|
Z2 = model2.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
|
|
plt.contourf(xx, yy, Z2, alpha=0.6, cmap=mpl.cm.jet)
|
|
plt.scatter(X[:,0], X[:,1],c=y,alpha=1,s=50,cmap=mpl.cm.jet,edgecolors="k")
|
|
plt.title("Bagging of decision trees")
|
|
plt.tight_layout()
|
|
plt.show()
|
|
````
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
<summary>랜덤 포리스트 (Random Forest)</summary>
|
|
<div markdown="1">
|
|
|
|
````python
|
|
import pandas as pd
|
|
from sklearn import datasets
|
|
from sklearn.model_selection import train_test_split
|
|
from sklearn.ensemble import RandomForestClassifier
|
|
from sklearn import metrics
|
|
|
|
iris = datasets.load_iris()
|
|
print('Class names :', iris.target_names)
|
|
print('target : [0:setosa, 1:versicolor, 2:virginical]')
|
|
print('No. of Data :', len(iris.data))
|
|
print('Featrue names :', iris.feature_names)
|
|
|
|
data = pd.DataFrame({
|
|
'sepal length': iris.data[:,0], 'sepal width': iris.data[:,1], 'petal length': iris.data[:,2],
|
|
'petal width':iris.data[:,3], 'species':iris.target
|
|
})
|
|
print(data.head()) # 일부 데이터 출력
|
|
|
|
x = data[['sepal length', 'sepal width', 'petal length', 'petal width']] # 입력
|
|
y = data['species'] # 출력
|
|
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3) # 테스트 데이터 30%
|
|
print('No. of traing data: ', len(x_train))
|
|
print('No. of test data:', len(y_test))
|
|
|
|
forest = RandomForestClassifier(n_estimators=100) # 모델 생성
|
|
forest.fit(x_train, y_train)
|
|
|
|
y_pred = forest.predict(x_test) # 추론 (예측)
|
|
print('Accuracy :', metrics.accuracy_score(y_test, y_pred))
|
|
````
|
|
|
|
</div>
|
|
|
|
<summary>Result</summary>
|
|
<div markdown="1">
|
|
|
|
````planetext
|
|
Class names : ['setosa' 'versicolor' 'virginica']
|
|
target : [0:setosa, 1:versicolor, 2:virginical]
|
|
No. of Data : 150
|
|
Featrue names : ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
|
|
sepal length sepal width petal length petal width species
|
|
0 5.1 3.5 1.4 0.2 0
|
|
1 4.9 3.0 1.4 0.2 0
|
|
2 4.7 3.2 1.3 0.2 0
|
|
3 4.6 3.1 1.5 0.2 0
|
|
4 5.0 3.6 1.4 0.2 0
|
|
No. of traing data: 105
|
|
No. of test data: 45
|
|
Accuracy : 0.9333333333333333
|
|
````
|
|
|
|
</div>
|
|
|
|
|
|
<summary>배깅 회귀 (Bagging Regression)</summary>
|
|
<div markdown="1">
|
|
|
|
````python
|
|
import numpy as np
|
|
import pandas as pd
|
|
from sklearn.datasets import load_boston # scikit-leanr < 1.2
|
|
# from sklearn.datasets import fetch_california_housing # replace dataset
|
|
from sklearn.metrics import mean_squared_error
|
|
from sklearn.model_selection import train_test_split
|
|
from sklearn.ensemble import BaggingRegressor
|
|
from sklearn.tree import DecisionTreeRegressor
|
|
import matplotlib.pyplot as plt
|
|
|
|
boston = load_boston() # < 1.2
|
|
data = pd.DataFrame(boston.data)
|
|
data.columns = boston.feature_names
|
|
data['PRICE'] = boston.target
|
|
print(data.head())
|
|
|
|
# replace dataset
|
|
# california = fetch_california_housing()
|
|
# data = pd.DataFrame(california.data)
|
|
# data.columns = california.feature_names
|
|
# data['PRICE'] = california.target
|
|
# print(data.head())
|
|
|
|
X, y = data.iloc[:,:-1],data.iloc[:,-1]
|
|
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123)
|
|
bag = BaggingRegressor(base_estimator = DecisionTreeRegressor( ), n_estimators = 10,
|
|
max_features=1.0, bootstrap_features=False, random_state=0)
|
|
bag.fit(X_train,y_train)
|
|
preds = bag.predict(X_test)
|
|
rmse = np.sqrt(mean_squared_error(y_test, preds))
|
|
print("RMSE: %f" % (rmse))
|
|
|
|
````
|
|
|
|
</div>
|
|
|
|
<summary>Result</summary>
|
|
<div markdown="1">
|
|
|
|
````planetext
|
|
CRIM ZN INDUS CHAS NOX RM AGE DIS RAD TAX
|
|
0 0.00632 18.0 2.31 0.0 0.538 6.575 65.2 4.0900 1.0 296.0
|
|
1 0.02731 0.0 7.07 0.0 0.469 6.421 78.9 4.9671 2.0 242.0
|
|
2 0.02729 0.0 7.07 0.0 0.469 7.185 61.1 4.9671 2.0 242.0
|
|
3 0.03237 0.0 2.18 0.0 0.458 6.998 45.8 6.0622 3.0 222.0
|
|
4 0.06905 0.0 2.18 0.0 0.458 7.147 54.2 6.0622 3.0 222.0
|
|
|
|
PTRATIO B LSTAT PRICE
|
|
0 15.3 396.90 4.98 24.0
|
|
1 17.8 396.90 9.14 21.6
|
|
2 17.8 392.83 4.03 34.7
|
|
3 18.7 394.63 2.94 33.4
|
|
4 18.7 396.90 5.33 36.2
|
|
RMSE: 4.594919
|
|
````
|
|
|
|
</div>
|
|
</details>
|
|
|
|
---
|
|
|
|
### 부스팅(Boosting) 예제 코드[¶]()
|
|
|
|
<details>
|
|
<summary>Code View</summary>
|
|
|
|
<summary>AdaBoost - Regression</summary>
|
|
<div markdown="1">
|
|
|
|
````python
|
|
import numpy as np
|
|
import matplotlib.pyplot as plt
|
|
from sklearn.tree import DecisionTreeRegressor
|
|
from sklearn.ensemble import AdaBoostRegressor
|
|
|
|
rng = np.random.RandomState(1)
|
|
X = np.linspace(0, 6, 100)[:, np.newaxis]
|
|
y = np.sin(X).ravel() + np.sin(6*X).ravel() + rng.normal(0, 0.1, X.shape[0])
|
|
|
|
regr_1 = DecisionTreeRegressor(max_depth=4)
|
|
regr_2 = AdaBoostRegressor(DecisionTreeRegressor(max_depth=4), n_estimators=100, random_state=rng)
|
|
|
|
regr_1.fit(X, y)
|
|
regr_2.fit(X, y)
|
|
y_1 = regr_1.predict(X)
|
|
y_2 = regr_2.predict(X)
|
|
|
|
plt.figure()
|
|
plt.scatter(X, y, c="k", label="training samples")
|
|
plt.plot(X, y_1, c="g", label="n_estimators=1", linewidth=2)
|
|
plt.plot(X, y_2, c="r", label="n_estimators=100", linewidth=2)
|
|
plt.xlabel("data")
|
|
plt.ylabel("target")
|
|
plt.title("AdaBoost Regression")
|
|
plt.legend()
|
|
plt.show()
|
|
````
|
|
|
|
|
|
|
|
</div>
|
|
|
|
<summary>Gradient Boosting - Regression</summary>
|
|
<div markdown="1">
|
|
|
|
````python
|
|
import numpy as np
|
|
import pandas as pd
|
|
from sklearn import datasets
|
|
import matplotlib.pyplot as plt
|
|
from sklearn.model_selection import train_test_split
|
|
from sklearn.metrics import mean_squared_error
|
|
from sklearn import ensemble
|
|
from sklearn.metrics import mean_squared_error, r2_score
|
|
from sklearn.model_selection import cross_val_predict
|
|
|
|
boston = datasets.load_boston() # Boston 집값 데이터, 13개 속성, 마지막 중간값 정보
|
|
print(boston.data.shape, boston.target.shape)
|
|
print(boston.feature_names)
|
|
|
|
data = pd.DataFrame(boston.data, columns=boston.feature_names)
|
|
data = pd.concat([data, pd.Series(boston.target, name='MEDV')], axis=1)
|
|
print(data.head())
|
|
X = data.iloc[:,:-1]
|
|
y = data.iloc[:,-1]
|
|
x_training_set, x_test_set, y_training_set, y_test_set = train_test_split(X, y, test_size=0.10, random_state=42, shuffle=True)
|
|
````
|
|
|
|
</div>
|
|
|
|
<summary>Result</summary>
|
|
<div markdown="1">
|
|
|
|
````planetext
|
|
(506, 13) (506,)
|
|
['CRIM' 'ZN' 'INDUS' 'CHAS' 'NOX' 'RM' 'AGE' 'DIS' 'RAD' 'TAX' 'PTRATIO'
|
|
'B' 'LSTAT']
|
|
CRIM ZN INDUS CHAS NOX RM AGE DIS RAD TAX
|
|
0 0.00632 18.0 2.31 0.0 0.538 6.575 65.2 4.0900 1.0 296.0
|
|
1 0.02731 0.0 7.07 0.0 0.469 6.421 78.9 4.9671 2.0 242.0
|
|
2 0.02729 0.0 7.07 0.0 0.469 7.185 61.1 4.9671 2.0 242.0
|
|
3 0.03237 0.0 2.18 0.0 0.458 6.998 45.8 6.0622 3.0 222.0
|
|
4 0.06905 0.0 2.18 0.0 0.458 7.147 54.2 6.0622 3.0 222.0
|
|
|
|
PTRATIO B LSTAT MEDV
|
|
0 15.3 396.90 4.98 24.0
|
|
1 17.8 396.90 9.14 21.6
|
|
2 17.8 392.83 4.03 34.7
|
|
3 18.7 394.63 2.94 33.4
|
|
4 18.7 396.90 5.33 36.2
|
|
````
|
|
|
|
</div>
|
|
|
|
<div markdown="1">
|
|
|
|
````python
|
|
params = {'n_estimators':500, 'max_depth':4, 'min_samples_split':2, 'learning_rate':0.01, 'loss':'ls'}
|
|
model = ensemble.GradientBoostingRegressor(**params)
|
|
model.fit(x_training_set, y_training_set)
|
|
model_score = model.score(x_training_set, y_training_set)
|
|
print('R2 sq: ', model_score)
|
|
|
|
y_predicted = model.predict(x_test_set)
|
|
print('Mean squared error: %.2f'% mean_squared_error(y_test_set, y_predicted))
|
|
print('Test Variance score: %.2f' % r2_score(y_test_set, y_predicted))
|
|
|
|
fig, ax = plt.subplots()
|
|
ax.scatter(y_test_set, y_predicted, edgecolors=(0,0,0))
|
|
ax.plot([y_test_set.min(), y_test_set.max()], [y_test_set.min(), y_test_set.max()], 'k--', lw=4)
|
|
ax.set_xlabel('Actual')
|
|
ax.set_ylabel('Predicted')
|
|
ax.set_title('Ground Truth vs Predicted')
|
|
plt.show()
|
|
````
|
|
|
|
|
|
|
|
</div>
|
|
|
|
<summary>Gradient Boosting - Classification</summary>
|
|
<div markdown="1">
|
|
|
|
````python
|
|
from sklearn.datasets import make_hastie_10_2
|
|
from sklearn.ensemble import GradientBoostingClassifier
|
|
import matplotlib.pyplot as plt
|
|
|
|
X, y = make_hastie_10_2(random_state=0)
|
|
X_train, X_test = X[:2000], X[2000:]
|
|
y_train, y_test = y[:2000], y[2000:]
|
|
print(X.shape, y.shape)
|
|
print(X[0:5,:])
|
|
print(y[0:5])
|
|
|
|
clf = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, max_depth=1, random_state=0)
|
|
clf.fit(X_train, y_train)
|
|
print('Accuracy score (training): {0:.3f}'.format(clf.score(X_train, y_train)))
|
|
print('Accuracy score (testing): {0:.3f}'.format(clf.score(X_test, y_test)))
|
|
````
|
|
|
|
</div>
|
|
|
|
<summary>Result</summary>
|
|
<div markdown="1">
|
|
|
|
````planetext
|
|
(12000, 10) (12000,)
|
|
[[ 1.76405235 0.40015721 0.97873798 2.2408932 1.86755799 -0.97727788
|
|
0.95008842 -0.15135721 -0.10321885 0.4105985 ]
|
|
[ 0.14404357 1.45427351 0.76103773 0.12167502 0.44386323 0.33367433
|
|
1.49407907 -0.20515826 0.3130677 -0.85409574]
|
|
[-2.55298982 0.6536186 0.8644362 -0.74216502 2.26975462 -1.45436567
|
|
0.04575852 -0.18718385 1.53277921 1.46935877]
|
|
[ 0.15494743 0.37816252 -0.88778575 -1.98079647 -0.34791215 0.15634897
|
|
1.23029068 1.20237985 -0.38732682 -0.30230275]
|
|
[-1.04855297 -1.42001794 -1.70627019 1.9507754 -0.50965218 -0.4380743
|
|
-1.25279536 0.77749036 -1.61389785 -0.21274028]]
|
|
[ 1. -1. 1. -1. 1.]
|
|
Accuracy score (training): 0.879
|
|
Accuracy score (testing): 0.819
|
|
````
|
|
|
|
</div>
|
|
|
|
<summary>XGBoosting - Regression</summary>
|
|
<div markdown="1">
|
|
|
|
````python
|
|
import numpy as np
|
|
import pandas as pd
|
|
from sklearn.datasets import load_boston
|
|
from sklearn.metrics import mean_squared_error
|
|
from sklearn.model_selection import train_test_split
|
|
import xgboost as xgb
|
|
|
|
boston = load_boston()
|
|
data = pd.DataFrame(boston.data)
|
|
data.columns = boston.feature_names
|
|
data['PRICE'] = boston.target
|
|
print(data.head())
|
|
X, y = data.iloc[:,:-1], data.iloc[:,-1]
|
|
|
|
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123)
|
|
xg_reg = xgb.XGBRegressor(objective='reg:squarederror', colsample_bytree=0.3, learning_rate=0.1, max_depth=5, alpha=10, n_estimators=10)
|
|
xg_reg.fit(X_train, y_train)
|
|
preds = xg_reg.predict(X_test)
|
|
rmse = np.sqrt(mean_squared_error(y_test, preds))
|
|
print('RMSE: %f' % (rmse))
|
|
````
|
|
|
|
</div>
|
|
|
|
<summary>Result</summary>
|
|
<div markdown="1">
|
|
|
|
````planetext
|
|
CRIM ZN INDUS CHAS NOX RM AGE DIS RAD TAX
|
|
0 0.00632 18.0 2.31 0.0 0.538 6.575 65.2 4.0900 1.0 296.0
|
|
1 0.02731 0.0 7.07 0.0 0.469 6.421 78.9 4.9671 2.0 242.0
|
|
2 0.02729 0.0 7.07 0.0 0.469 7.185 61.1 4.9671 2.0 242.0
|
|
3 0.03237 0.0 2.18 0.0 0.458 6.998 45.8 6.0622 3.0 222.0
|
|
4 0.06905 0.0 2.18 0.0 0.458 7.147 54.2 6.0622 3.0 222.0
|
|
|
|
PTRATIO B LSTAT PRICE
|
|
0 15.3 396.90 4.98 24.0
|
|
1 17.8 396.90 9.14 21.6
|
|
2 17.8 392.83 4.03 34.7
|
|
3 18.7 394.63 2.94 33.4
|
|
4 18.7 396.90 5.33 36.2
|
|
RMSE: 10.423243
|
|
````
|
|
|
|
</div>
|
|
|
|
|
|
<summary>LightGBM</summary>
|
|
<div markdown="1">
|
|
|
|
````python
|
|
from lightgbm import LGBMClassifier, LGBMRegressor
|
|
from lightgbm import plot_importance, plot_metric, plot_tree
|
|
from sklearn.datasets import load_iris
|
|
from sklearn.model_selection import train_test_split
|
|
from sklearn.model_selection import cross_validate
|
|
|
|
iris = load_iris()
|
|
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.2, random_state=123)
|
|
lgbmc = LGBMClassifier(n_estimators=400)
|
|
evals = [(X_test, y_test)]
|
|
lgbmc.fit(X_train, y_train, early_stopping_rounds=100, eval_metric='logloss', eval_set=evals, verbose=True)
|
|
preds = lgbmc.predict(X_test)
|
|
|
|
cross_val = cross_validate(
|
|
estimator=lgbmc,
|
|
X=iris.data, y=iris.target,
|
|
cv=5
|
|
)
|
|
|
|
print('avg fit time: {} (+/- {})'.format(cross_val['fit_time'].mean(), cross_val['fit_time'].std()))
|
|
print('avg fit time: {} (+/- {})'.format(cross_val['score_time'].mean(), cross_val['score_time'].std()))
|
|
print('avg fit time: {} (+/- {})'.format(cross_val['test_score'].mean(), cross_val['test_score'].std()))
|
|
|
|
plot_metric(lgbmc)
|
|
plot_importance(lgbmc, figsize=(10,12))
|
|
plot_tree(lgbmc, figsize=(28,14))
|
|
````
|
|
|
|
</div>
|
|
|
|
<summary>Result</summary>
|
|
<div markdown="1">
|
|
|
|
````planetext
|
|
[1] valid_0's multi_logloss: 0.95847
|
|
[2] valid_0's multi_logloss: 0.832184
|
|
[3] valid_0's multi_logloss: 0.731164
|
|
[4] valid_0's multi_logloss: 0.641056
|
|
[5] valid_0's multi_logloss: 0.571726
|
|
[6] valid_0's multi_logloss: 0.507286
|
|
[7] valid_0's multi_logloss: 0.454933
|
|
[8] valid_0's multi_logloss: 0.410205
|
|
[9] valid_0's multi_logloss: 0.372194
|
|
[10] valid_0's multi_logloss: 0.333919
|
|
[11] valid_0's multi_logloss: 0.310212
|
|
[12] valid_0's multi_logloss: 0.282326
|
|
[13] valid_0's multi_logloss: 0.257165
|
|
[14] valid_0's multi_logloss: 0.240836
|
|
[15] valid_0's multi_logloss: 0.225383
|
|
[16] valid_0's multi_logloss: 0.211583
|
|
[17] valid_0's multi_logloss: 0.199289
|
|
[18] valid_0's multi_logloss: 0.186269
|
|
[19] valid_0's multi_logloss: 0.171556
|
|
[20] valid_0's multi_logloss: 0.168245
|
|
[21] valid_0's multi_logloss: 0.161065
|
|
[22] valid_0's multi_logloss: 0.151371
|
|
[23] valid_0's multi_logloss: 0.148081
|
|
[24] valid_0's multi_logloss: 0.143843
|
|
[25] valid_0's multi_logloss: 0.140169
|
|
...
|
|
[137] valid_0's multi_logloss: 0.376748
|
|
avg fit time: 0.5514350891113281 (+/- 0.3701610138582717)
|
|
avg fit time: 0.010002517700195312 (+/- 0.009552237668971902)
|
|
avg fit time: 0.9600000000000002 (+/- 0.04898979485566355)
|
|
````
|
|
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
</details>
|
|
|
|
|
|
|
|
### 참고[¶]()
|
|
|
|
- 산업인공지능개론 과목, 이건명 교수
|
|
- ChatGPT |