Bayesian Optimization is a popular searching algorithm for hyper-parameters in the machine learning area.
There are also two popular Python libraries for this algorithm: Hyperopt and Optuna. So I have done some experiments on these two libraries. The trial is using LightGBM to classify tabular data, and the hyper-parameters and their ranges are:
- n_estimators (10~100000)
- learning_rate (0.0001~1.0)
- num_leaves (2, 2000)
- max_depth (2, 200)
- min_child_samples (1, 10000)
- reg_alpha (0.001, 10.0)
- reg_lambda (0.001, 10.0)
I run 10 trials each time for 10 times, by using different algorithms in Hyperopt and Optuna. The result for Hyperopt is:
Clik here to view.
And the result for Optuna is:
Clik here to view.
Seems that for my tabular data, TPE and Annealing works best, and Hyperopt works better than Optuna
