They did just make a GitHub repo public which has some of the data, some of which seems newly public:
Thank you!!!
I previously wrote about their confusion with hyperparameters and this confirms it. They claim they trained models with identical hyperparameters... But there's a whole bunch of hyperparameters that are automatically set to default values unless you change them. From their repo:
xg = XGBClassifier(
n_estimators=xgb_cfg['n_estimators'], max_depth=xgb_cfg['max_depth'],
learning_rate=xgb_cfg['learning_rate'],
min_child_weight=xgb_cfg.get('min_child_weight', 3),
scale_pos_weight=weight_ratio, random_state=seed + fold_offset,
use_label_encoder=False, eval_metric='logloss', verbosity=0)
lg = LGBMClassifier(
n_estimators=lgbm_cfg['n_estimators'], max_depth=lgbm_cfg['max_depth'],
learning_rate=lgbm_cfg['learning_rate'],
min_child_samples=lgbm_cfg.get('min_child_samples', 3),
scale_pos_weight=weight_ratio, random_state=seed + fold_offset,
verbosity=-1)
As I mentioned previously, all of these implementations have unique hyperparameters. LGBMClassifier, for example, has `num_leaves` that defaults to 31 if you don't change it. XGBoost has `max_leaves` (which is only equivalent if multiple other hyperparameters are set to specific values) with a default of `None`, meaning it's not used.
So the claim in the paper that identical hyperparameters is false because they didn't know about default values.
In any case, it still leaves the issue that the methodology, specifically averaging 3 GBDT and 1 Random Forest, itself makes no sense for this use case. If the 3 GBDT had identical hyperparameters, you would get identical predictions, with some minor random variation based on processing. That gives no additional information or predictive power. Since their entire set of hyperparameters were not identical anyway, we'll get some variance in predictions across models, but the predictions are likely extremely highly correlated. Model ensembling is useful for predictions where there is disagreement or uncertainty between the models. It's not useful if you have models that generate the same prediction. Since all they did here was take the straight average of the 4 model predictions, final_pred = (mod1+mod2+mod3+mod4)/4, it's really little more than a weighted average of the GBDT (weighted .75) and the RF (weighted .25), since mod1≈mod2≈mod3.
The methodology, the largely AI generated code, the labeling process, and all the other issues I've described previously make it clear that none of the authors are experienced with data science or statistical modeling. I also suspect that none of the peer reviewers were experts in statistical modeling.
I still need to confirm this by looking at the code some more, but it appears that they calculated Shapley values only on the random forest model. Shapley values represent the contribution for a given feature, conditional on the values of the other features. Put simply, they answer the question: how did feature f with value x contribute to the prediction for observation i? Since they used black box models instead of logistic regression, we don't get simple and easily interpretable coefficients for features.
This is from a brief look at the code just on my phone. I'll take a better look later.
I'm sure it sounds… Wunderbar, and all. So proud. But Germany is OT, here. Maybe you should start a German thread.
