I have some critiques of Villarroel's paper
Use of machine learning to enhance detection of transient astronomical phenomena in historical observatory images. (For convenience, I will refer to Dr. Villarroel as BV, the convention she uses in the paper.)
I've said this many times but I'll repeat it again, I am
NOT accusing BV of deliberate fraud or deception. Every critique I am making here can be the result of honest mistakes and/or subconscious biases. I am not making any claims on motivation.
The first critique actually applies to most of her papers - a very high rate of self-citation. For example, 7 of the first 10 citations of this machine learning paper are to her previous work.
Earth-Projected Clustering of Historical Optical Transients in the Palomar Observatory Sky Survey-I (POSS-I) has 12 self-citations in the first 15 sources. This is not inherently a bad thing and is not too surprising given the niche nature of the topic. However, the lack of replication by other,
independent researchers and the extreme reliance on her own work creates the risk of mistakes and biases propagating throughout her research.
Back to the ML paper specifically. I am a data scientist/actuary so I know the models used extremely well. Their training set was 250 samples, which is quite small. Only 250 samples when using 23 features makes that sample size even more problematic. That's a small number of samples in general and very small relative to the number of features. The next issue is the target, i.e. what the model is trying to predict. The model is a probabilistic classification model, meaning it outputs the probability that a transient is "real". How were samples determined to be real or defects?
This training set was manually inspected by an astronomer with expertise in the transient phenomenon (BV); 134 were labelled likely real transients and 116 as plate defects.
Dr. Villarroel herself decided! But there was a second reviewer...
To enhance classification consistency over time, a second reviewer (SB) who was trained by the primary evaluator periodically reviewed a subset of the images assigned to the primary reviewer, withany disagreements discussed with the primary reviewer and resolved.
The second reviewer was trained by Dr. Villarroel herself! Any disagreement was discussed with BV. Why does this matter? The goal of this paper was to
External Quote:
enhance transient identification accuracy and validate the phenomenon.
But since BV determined the training data, this model isn't enhancing accuracy or validating the phenomenon. It is a model that just predicts BV's opinion. This exercise can be summarized as "the model I built on data I classified based on my own assumptions validates my other set of claims that are also based on the same assumptions." Since the original analysis and the ML analysis are all predicated on the same assumptions, none of the potential sources of bias are actually addressed. All she's really testing is whether the data used to build the ML model is similar to the data used for the transient analysis, which is of course true. It comes from the same source!
The modeling procedure and its description is quite odd. Saying "model" is somewhat inaccurate since BV actually trained 4 models and a final meta model (aka ensemble models, model stacking, etc). She trained 4 models and then the final prediction is just the average of those 4. That makes the 250 sample size even more insufficient. It's also just completely unnecessary and leads to a mountain of possible issues, such as overfitting, despite the use of cross-validation. The modeling procedure also suggests a lack of experience and expertise on machine learning. See for example:
External Quote:
The ensemble ML classifier detailed in this study combined four tree-based models(XGBoost, Random Forest, Gradient Boosting, LightGBM), each trained with 300 trees and identical hyperparameters, with final classification predictions based on the unweighted mean ofthe four models' predicted probabilities.
The above sentence doesn't actually make sense to someone who uses these models (such as me). She says the four models are XGBoost, Random Forest, Gradient Boosting, and LightGBM. XGBoost and LightGBM are both gradient boosting decision tree models (hence the GB in both names). It's not even clear what the third model is referring to. "Gradient boosting" is a specific modeling technique, not a particular model itself. It might be the implementation of gradient boosted trees from the python package she used (the paper says sci-kit learn which includes
GradientBoostingClassifier). Random Forest, while not gradient boosted, is another type of ensemble of decision trees. Which random forest implementation was used was also not specified. Again, I'll just assume sci-kit learn
RandomForestClassifier.
So 3 of the four models are just different implementations of the same model type and the other model is extremely similar. There is no prima facie reason to use an ensemble of four extremely similar models. It actually overcomplicates the analysis, significantly increasing variance while likely having no significant decrease in model bias (see
Bias–variance tradeoff. Model stacking is a useful technique when the models aren't highly correlated. Think of a case where you have 2 models that give the exact same predictions. Averaging them gives you no benefit. Model stacking is useful when you have say, model 1 that performs on some subset of data and poorly on the rest, and another model 2 that does the opposite. Stacking those models is useful because the meta model will learn when to use model 1 and when to use model 2.
Saying all 4 were trained with "identical hyperparameters" is also simply impossible. "Hyperparameters" are essentially the settings determining how the model will learn from the data. Despite the similarity between XGBoost, LightGBM, and Gradient Boosting, each of these implementations have different hyperparameters. You can view their hyperparameters at the following links:
XGBoost,
LightGBM,
GradientBoostingClassifier,
RandomForestClassifier. Not just different default values for hyperparameters, they all have some hyperparameters that are completely
unique from each other, due to the difference in how they build their trees. Random Forest is a different algorithm altogether so it simply cannot have identical hyperparameters to the others. XGBoost and LightGBM can be explained simply as combining the predictions of a bunch of
decision trees. However, XGBoost builds its trees "depth-wise" and LightGBM builds its trees "leaf-wise". The meaning of those terms is out of scope here, but the point is that since they build their trees using different algorithms, it is not possible to have identical hyperparameters.
The features themselves used in the model also may not be what many would have expected. My immediate assumption when seeing the title of the paper was that BV used a computer vision model of some sort. That is, a model that actually "looks" at the images on the plate (using the pixel values). The most common CV model type has been
convolutional neural networks (or at least traditionally most common until transformers arrived). These models actually look at the images by taking their pixel values as inputs. From there, the basic explanation is they learn relationships and structures of things in the image. BV's models took a different approach:
External Quote:
The ML model included 23 predictors extracted from the red FITS images and the VASCO v4 catalog. Seven catalog-level morphometric features were included: signal-to-noise ratio (SNR), point spread function (PSF) ratio, elongation, compactness, sharpness, number of comparison stars, and candidate score (described in Solano et al.1). The ML model also included 6 plate-aggregate features: a plate quality indicator, plate-level SNR fraction, SNR standard deviation, mean elongation, and high-SNR source count. Finally, the model included 10 morphometric features identified by the ML model in the red FITS images themselves: PSF Full Width at Half Maximum (PSF FWHM), ellipticity, sharpness, connected pixel count, aperture flux, distance to plate edge, symmetry score, gradient magnitude, proximity to bright star, and FITS-measured SNR. The ML model did not include any spectral features or red-blue plate comparison.
Fewer than half of the features actually relate to any given transient itself. 13 of the 23 features are either catalog-level (7) or plate-level (6).
We can also take a look at Figure 2 to see how these features contribute to the overall prediction. The figure visualizes SHAP values, which are a measure of how a feature contributed to the overall prediction (the name deriving from Game Theory's Shapley Values). We can see which features were the most influential, on average. Here are the top 6 features (some formatting added in by me):
External Quote:
- plate_low_snr_frac = Plate- level: fraction of candidates on the same plate with SNR < 5;
- plate_quality_GOOD = Plate-level: binary, plate rated as GOOD (one-hot encoded);
- plate_quality_MODERATE = Plate-level: binary, plate rated as MODERATE (one-hot encoded);
- plate_elongation_mean = Plate-level: mean elongation of all candidates on the same plate;
- plate_snr_std = Plate-level: standard deviation of SNR across all candidates on the same plate;
- candidate_score = Candidate-level: original pipeline quality score from Solano et al.
The top 5 features are all
plate-level. The 6th most important feature is "candidate score", which is a value derived in another paper authored by, you guessed it, BV.
External Quote:
Solano, E., Villarroel, B., & Rodrigo, C. Discovering vanishing objects in POSS I red images using the Virtual Observatory, Monthly Notices Royal Astron Soc. 515, 1380–1391 (2022).
This is just another source of bias being introduced into the model. The most important features being plate-level means that very little of a prediction for a given transient is actually a characteristic of that specific transiet.
Lastly, one question I have is "why not start with a simple logistic regression generalized linear model?" It is best practice in ML to start with the simplest models first (kind of analogous to Occam's razor). Basically, start simple, and then look for evidence of interactions and non-linear effects that would be handled by more complex models. Getting even more specific, catalogs and plates should likely be treated as "
random effects" because multiple transients on a plate violate the independence assumption of most statistical analyses. That topic is again out of scope for this comment.
Ok, so what's the point of all of the above? Returning to the abstract, the goal of the paper was improving identification and validating the phenomenon.
External Quote:
These findings remain debated; some argue transients identified via existing automated pipelines are simply plate defects. Machine learning (ML) was used to enhance transient identification accuracy and validate the phenomenon.
This ML paper does neither. Due to the fact that every step of the ML process was determined by BV, the ML approach hasn't done anything to actually remove that bias. The common saying is "garbage in, garbage out", but I don't consider the work to be "garbage", so a more fair saying would be "biased or invalid data in, biased or invalid data out".
There's quite a lot of jargon in this comment, and I tried to balance providing sufficient detail without being too dense. Please feel free to ask for any clarifications!
One last thing I forgot to mention. Despite the paper referring to a GitHub repo for the code, it doesn't appear to exist, so all of the above is based off the paper itself. I would have preferred the code itself, but I'm working with what I can.
