You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
318 lines
11 KiB
Markdown
318 lines
11 KiB
Markdown
# 앙상블(Enemble)이란?
|
|
앙상블(Ensemble) 학습은 여러 개의 머신러닝 모델을 결합하여 단일 모델보다 더 나은 성능을 얻기 위한 기법을 말한다
|
|
앙상블 기법은 여러 개의 약한 학습기(weak learner)를 결합하여 강한 학습기(strong learner)를 생성하는 아이디어에 기반한다.
|
|
앙상블 기법은 다양한 머신러닝 문제에서 높은 성능을 달성하는 데 매우 효과적이다.
|
|
다양한 모델의 특징과 장점을 결합하므로, 오버피팅(과적합)을 줄이고 일반화 성능을 향상시킬 수 있다.
|
|
|
|
|
|
# 앙상블 학습의 주요 기법:
|
|
|
|
1. **배깅(Bagging, Bootstrap Aggregating)**:
|
|
- 동일한 알고리즘에 대해 훈련 데이터의 서로 다른 부분 집합(subset)을 사용하여 여러 모델을 훈련한다.
|
|
- 모든 모델의 예측을 집계하여 최종 예측을 생성한다.
|
|
- ex) **랜덤 포레스트(Random Forest)**.
|
|
|
|
2. **부스팅(Boosting)**:
|
|
- 연속적으로 모델을 훈련시키면서, 이전 모델의 오류를 다음 모델이 보정한다.
|
|
- 모든 모델의 예측을 조합하여 최종 예측을 생성한다.
|
|
- ex) **AdaBoost, Gradient Boosting, XGBoost, LightGBM, CatBoost**
|
|
|
|
3. **스태킹(Stacking)**:
|
|
- 여러 다른 모델로부터의 예측을 취합하여, 그 예측들을 입력으로 사용하는 새로운 모델(메타 모델)을 훈련시킨다.
|
|
- 이 메타 모델이 최종 예측을 생성한다.
|
|
|
|
|
|
---
|
|
|
|
### 배깅(Bagging) 예제 코드[¶]()
|
|
|
|
<details>
|
|
<summary>Code View</summary>
|
|
|
|
<summary>Bootstrap Aggregating</summary>
|
|
<div markdown="1">
|
|
|
|
````python
|
|
import numpy as np
|
|
import matplotlib as mpl
|
|
import matplotlib.pyplot as plt
|
|
from sklearn.datasets import load_iris
|
|
from sklearn.tree import DecisionTreeClassifier
|
|
from sklearn.ensemble import BaggingClassifier
|
|
|
|
iris = load_iris()
|
|
X, y = iris.data[:, [0,2]], iris.target
|
|
|
|
model1 = DecisionTreeClassifier(max_depth =10, random_state=0).fit(X, y)
|
|
model2 = BaggingClassifier(DecisionTreeClassifier(max_depth=4), n_estimators=50, random_state=0).fit(X, y)
|
|
|
|
x_min, x_max = X[:,0].min() - 1, X[:,0].max() + 1
|
|
y_min, y_max = X[:,1].min() - 1, X[:,1].max() + 1
|
|
xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1), np.arange(y_min, y_max, 0.1))
|
|
|
|
plt.subplot(121)
|
|
Z1 = model1.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
|
|
plt.contourf(xx, yy, Z1, alpha=0.6, cmap=mpl.cm.jet)
|
|
plt.scatter(X[:,0], X[:,1], c=y, alpha=1, s=50, cmap=mpl.cm.jet, edgecolors="k")
|
|
plt.title("Decision tree")
|
|
plt.subplot(122)
|
|
|
|
Z2 = model2.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
|
|
plt.contourf(xx, yy, Z2, alpha=0.6, cmap=mpl.cm.jet)
|
|
plt.scatter(X[:,0], X[:,1],c=y,alpha=1,s=50,cmap=mpl.cm.jet,edgecolors="k")
|
|
plt.title("Bagging of decision trees")
|
|
plt.tight_layout()
|
|
plt.show()
|
|
````
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
<summary>랜덤 포리스트 (Random Forest)</summary>
|
|
<div markdown="1">
|
|
|
|
````python
|
|
import pandas as pd
|
|
from sklearn import datasets
|
|
from sklearn.model_selection import train_test_split
|
|
from sklearn.ensemble import RandomForestClassifier
|
|
from sklearn import metrics
|
|
|
|
iris = datasets.load_iris()
|
|
print('Class names :', iris.target_names)
|
|
print('target : [0:setosa, 1:versicolor, 2:virginical]')
|
|
print('No. of Data :', len(iris.data))
|
|
print('Featrue names :', iris.feature_names)
|
|
|
|
data = pd.DataFrame({
|
|
'sepal length': iris.data[:,0], 'sepal width': iris.data[:,1], 'petal length': iris.data[:,2],
|
|
'petal width':iris.data[:,3], 'species':iris.target
|
|
})
|
|
print(data.head()) # 일부 데이터 출력
|
|
|
|
x = data[['sepal length', 'sepal width', 'petal length', 'petal width']] # 입력
|
|
y = data['species'] # 출력
|
|
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3) # 테스트 데이터 30%
|
|
print('No. of traing data: ', len(x_train))
|
|
print('No. of test data:', len(y_test))
|
|
|
|
forest = RandomForestClassifier(n_estimators=100) # 모델 생성
|
|
forest.fit(x_train, y_train)
|
|
|
|
y_pred = forest.predict(x_test) # 추론 (예측)
|
|
print('Accuracy :', metrics.accuracy_score(y_test, y_pred))
|
|
````
|
|
|
|
</div>
|
|
|
|
<summary>Result</summary>
|
|
<div markdown="1">
|
|
|
|
````planetext
|
|
Class names : ['setosa' 'versicolor' 'virginica']
|
|
target : [0:setosa, 1:versicolor, 2:virginical]
|
|
No. of Data : 150
|
|
Featrue names : ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
|
|
sepal length sepal width petal length petal width species
|
|
0 5.1 3.5 1.4 0.2 0
|
|
1 4.9 3.0 1.4 0.2 0
|
|
2 4.7 3.2 1.3 0.2 0
|
|
3 4.6 3.1 1.5 0.2 0
|
|
4 5.0 3.6 1.4 0.2 0
|
|
No. of traing data: 105
|
|
No. of test data: 45
|
|
Accuracy : 0.9333333333333333
|
|
````
|
|
|
|
</div>
|
|
|
|
|
|
<summary>배깅 회귀 (Bagging Regression)</summary>
|
|
<div markdown="1">
|
|
|
|
````python
|
|
import numpy as np
|
|
import pandas as pd
|
|
from sklearn.datasets import load_boston # scikit-leanr < 1.2
|
|
# from sklearn.datasets import fetch_california_housing # replace dataset
|
|
from sklearn.metrics import mean_squared_error
|
|
from sklearn.model_selection import train_test_split
|
|
from sklearn.ensemble import BaggingRegressor
|
|
from sklearn.tree import DecisionTreeRegressor
|
|
import matplotlib.pyplot as plt
|
|
|
|
boston = load_boston() # < 1.2
|
|
data = pd.DataFrame(boston.data)
|
|
data.columns = boston.feature_names
|
|
data['PRICE'] = boston.target
|
|
print(data.head())
|
|
|
|
# replace dataset
|
|
# california = fetch_california_housing()
|
|
# data = pd.DataFrame(california.data)
|
|
# data.columns = california.feature_names
|
|
# data['PRICE'] = california.target
|
|
# print(data.head())
|
|
|
|
X, y = data.iloc[:,:-1],data.iloc[:,-1]
|
|
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123)
|
|
bag = BaggingRegressor(base_estimator = DecisionTreeRegressor( ), n_estimators = 10,
|
|
max_features=1.0, bootstrap_features=False, random_state=0)
|
|
bag.fit(X_train,y_train)
|
|
preds = bag.predict(X_test)
|
|
rmse = np.sqrt(mean_squared_error(y_test, preds))
|
|
print("RMSE: %f" % (rmse))
|
|
|
|
````
|
|
|
|
</div>
|
|
|
|
<summary>Result</summary>
|
|
<div markdown="1">
|
|
|
|
````planetext
|
|
CRIM ZN INDUS CHAS NOX RM AGE DIS RAD TAX
|
|
0 0.00632 18.0 2.31 0.0 0.538 6.575 65.2 4.0900 1.0 296.0
|
|
1 0.02731 0.0 7.07 0.0 0.469 6.421 78.9 4.9671 2.0 242.0
|
|
2 0.02729 0.0 7.07 0.0 0.469 7.185 61.1 4.9671 2.0 242.0
|
|
3 0.03237 0.0 2.18 0.0 0.458 6.998 45.8 6.0622 3.0 222.0
|
|
4 0.06905 0.0 2.18 0.0 0.458 7.147 54.2 6.0622 3.0 222.0
|
|
|
|
PTRATIO B LSTAT PRICE
|
|
0 15.3 396.90 4.98 24.0
|
|
1 17.8 396.90 9.14 21.6
|
|
2 17.8 392.83 4.03 34.7
|
|
3 18.7 394.63 2.94 33.4
|
|
4 18.7 396.90 5.33 36.2
|
|
RMSE: 4.594919
|
|
````
|
|
|
|
</div>
|
|
</details>
|
|
|
|
---
|
|
|
|
### 부스팅(Boosting) 예제 코드[¶]()
|
|
|
|
<details>
|
|
<summary>Code View</summary>
|
|
|
|
<summary>AdaBoost - Regression</summary>
|
|
<div markdown="1">
|
|
|
|
````python
|
|
import numpy as np
|
|
import matplotlib.pyplot as plt
|
|
from sklearn.tree import DecisionTreeRegressor
|
|
from sklearn.ensemble import AdaBoostRegressor
|
|
|
|
rng = np.random.RandomState(1)
|
|
X = np.linspace(0, 6, 100)[:, np.newaxis]
|
|
y = np.sin(X).ravel() + np.sin(6*X).ravel() + rng.normal(0, 0.1, X.shape[0])
|
|
|
|
regr_1 = DecisionTreeRegressor(max_depth=4)
|
|
regr_2 = AdaBoostRegressor(DecisionTreeRegressor(max_depth=4), n_estimators=100, random_state=rng)
|
|
|
|
regr_1.fit(X, y)
|
|
regr_2.fit(X, y)
|
|
y_1 = regr_1.predict(X)
|
|
y_2 = regr_2.predict(X)
|
|
|
|
plt.figure()
|
|
plt.scatter(X, y, c="k", label="training samples")
|
|
plt.plot(X, y_1, c="g", label="n_estimators=1", linewidth=2)
|
|
plt.plot(X, y_2, c="r", label="n_estimators=100", linewidth=2)
|
|
plt.xlabel("data")
|
|
plt.ylabel("target")
|
|
plt.title("AdaBoost Regression")
|
|
plt.legend()
|
|
plt.show()
|
|
````
|
|
|
|
|
|
|
|
</div>
|
|
|
|
<summary>Gradient Boosting - Regression</summary>
|
|
<div markdown="1">
|
|
|
|
````python
|
|
import numpy as np
|
|
import pandas as pd
|
|
from sklearn import datasets
|
|
import matplotlib.pyplot as plt
|
|
from sklearn.model_selection import train_test_split
|
|
from sklearn.metrics import mean_squared_error
|
|
from sklearn import ensemble
|
|
from sklearn.metrics import mean_squared_error, r2_score
|
|
from sklearn.model_selection import cross_val_predict
|
|
|
|
boston = datasets.load_boston() # Boston 집값 데이터, 13개 속성, 마지막 중간값 정보
|
|
print(boston.data.shape, boston.target.shape)
|
|
print(boston.feature_names)
|
|
|
|
data = pd.DataFrame(boston.data, columns=boston.feature_names)
|
|
data = pd.concat([data, pd.Series(boston.target, name='MEDV')], axis=1)
|
|
print(data.head())
|
|
X = data.iloc[:,:-1]
|
|
y = data.iloc[:,-1]
|
|
x_training_set, x_test_set, y_training_set, y_test_set = train_test_split(X, y, test_size=0.10, random_state=42, shuffle=True)
|
|
````
|
|
|
|
</div>
|
|
|
|
<summary>Result</summary>
|
|
<div markdown="1">
|
|
|
|
````planetext
|
|
(506, 13) (506,)
|
|
['CRIM' 'ZN' 'INDUS' 'CHAS' 'NOX' 'RM' 'AGE' 'DIS' 'RAD' 'TAX' 'PTRATIO'
|
|
'B' 'LSTAT']
|
|
CRIM ZN INDUS CHAS NOX RM AGE DIS RAD TAX
|
|
0 0.00632 18.0 2.31 0.0 0.538 6.575 65.2 4.0900 1.0 296.0
|
|
1 0.02731 0.0 7.07 0.0 0.469 6.421 78.9 4.9671 2.0 242.0
|
|
2 0.02729 0.0 7.07 0.0 0.469 7.185 61.1 4.9671 2.0 242.0
|
|
3 0.03237 0.0 2.18 0.0 0.458 6.998 45.8 6.0622 3.0 222.0
|
|
4 0.06905 0.0 2.18 0.0 0.458 7.147 54.2 6.0622 3.0 222.0
|
|
|
|
PTRATIO B LSTAT MEDV
|
|
0 15.3 396.90 4.98 24.0
|
|
1 17.8 396.90 9.14 21.6
|
|
2 17.8 392.83 4.03 34.7
|
|
3 18.7 394.63 2.94 33.4
|
|
4 18.7 396.90 5.33 36.2
|
|
````
|
|
|
|
</div>
|
|
|
|
<div markdown="1">
|
|
|
|
````python
|
|
params = {'n_estimators':500, 'max_depth':4, 'min_samples_split':2, 'learning_rate':0.01, 'loss':'ls'}
|
|
model = ensemble.GradientBoostingRegressor(**params)
|
|
model.fit(x_training_set, y_training_set)
|
|
model_score = model.score(x_training_set, y_training_set)
|
|
print('R2 sq: ', model_score)
|
|
|
|
y_predicted = model.predict(x_test_set)
|
|
print('Mean squared error: %.2f'% mean_squared_error(y_test_set, y_predicted))
|
|
print('Test Variance score: %.2f' % r2_score(y_test_set, y_predicted))
|
|
|
|
fig, ax = plt.subplots()
|
|
ax.scatter(y_test_set, y_predicted, edgecolors=(0,0,0))
|
|
ax.plot([y_test_set.min(), y_test_set.max()], [y_test_set.min(), y_test_set.max()], 'k--', lw=4)
|
|
ax.set_xlabel('Actual')
|
|
ax.set_ylabel('Predicted')
|
|
ax.set_title('Ground Truth vs Predicted')
|
|
plt.show()
|
|
````
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
</details>
|