Transients in the Palomar Observatory Sky Survey

[JMartJr]

It's similar to the "why do alien spaceships have flashing lights all over them?" question that always leads to some "who can know the motivations of beings beyond our ken" type answer.

Best answer I ever heard for this was along the lines of "Maybe the lights we're seeing are in a wavelength THEY can;t see, they don;t even know how eye catching they are!" I don't buy it (they've abducted and tested enough people by now to know what wavelengths WE can see!) but it gets points for creative thinking, I guess...

 

[FatPhil]

Best answer I ever heard for this was along the lines of "Maybe the lights we're seeing are in a wavelength THEY can;t see, they don;t even know how eye catching they are!" I don't buy it (they've abducted and tested enough people by now to know what wavelengths WE can see!) but it gets points for creative thinking, I guess...

So these hyper-intelligent beings (with technological capabilities unimaginably more advanced than ours) are deliberately (as it's are part of the design of the craft) expending energy on something they can't make use of? I think that's ascribing a certain level of dumbness to the beings, which isn't what I would classify as "creative thinking", it's more like projection.

 

[jarlrmai]

Yes because aliens are can build amazing spacecraft but haunt figured out what Herchel etc discovered in the 1800's

 

[Mendel]

Best answer I ever heard for this was along the lines of "Maybe the lights we're seeing are in a wavelength THEY can;t see, they don;t even know how eye catching they are!" I don't buy it (they've abducted and tested enough people by now to know what wavelengths WE can see!) but it gets points for creative thinking, I guess...

Especially when the lights just so happen to meet aircraft regulations...

So these hyper-intelligent beings (with technological capabilities unimaginably more advanced than ours) are deliberately (as it's are part of the design of the craft) expending energy on something they can't make use of? I think that's ascribing a certain level of dumbness to the beings, which isn't what I would classify as "creative thinking", it's more like projection.

Well, so do we. A fair amount of gear gives off IR light—precisely because we can't see it. (e.g. security cameras, TV remotes)

 

[FatPhil]

Well, so do we. A fair amount of gear gives off IR light—precisely because we can't see it. (e.g. security cameras, TV remotes)

Yes, but we actively try to reduce it. For example, the transition away from incandescent lightbulbs in the EU and much of the world over the last couple of decades.

 

[GM4AJK]

expending energy on something they can't make use of?

Twitter/X would like a word

 

[jdog]

So these hyper-intelligent beings (with technological capabilities unimaginably more advanced than ours) are deliberately (as it's are part of the design of the craft) expending energy on something they can't make use of? I think that's ascribing a certain level of dumbness to the beings, which isn't what I would classify as "creative thinking", it's more like projection.

Veering slightly more off-topic, but in his new UAP-hunting memoir Out of the Shadows, Jay Stratton apparently contends that the white froth reported on the ocean surface under the Nimitz Tic-Tac was the seawater boiling from the unknown energy of the Tic-Tac's propulsion system -- which is a boggling assumption without any evidence, and suggests there should be a very hot column of air underneath every Tic-Tac and a heat trail wherever they go.

 

[HoaxEye]

They did just make a GitHub repo public which has some of the data, some of which seems newly public:

https://github.com/dca-doherty/VASCO-ML

MIT License, so I've attached a .zip here:

Thanks Mick. Worth adding what is and isn't in that release, because it changes what anyone can do with it.


The 107,875-row file is the VASCO 2025 sample. The detection output behind Solano et al. (2022). It carries RA, Dec and observation date plus some catalogue-level columns. What it doesn't carry is the per-candidate ML scores behind the paper's figures: the ml_probability column takes 287 distinct values across 107,875 rows, and 99.7% of rows share one identical value, while the committed decile tables span a continuous range in ten equal bins. The scored catalogue, the 250 training labels and the image-derived features aren't in the repo either.

I tried the documented reproduction in their own pinned environment. reproduce.py --verify-only passes 22/22, but it only compares committed outputs against committed expectations - nothing is recomputed. The full run stops in half a second at stage one of four. Working around each obstacle in turn: the config points at data/raw/ while the file ships at data/; requirements.txt omits seaborn, which the training stage imports; feature extraction then reports OK in under a second while producing a cache containing none of the 16 image-derived features (the config points at a D: path); and training stops on "No label files loaded", which can't be worked around.

To be clear about what that does and doesn't mean: the published results can't currently be regenerated from the published artifacts. It is not evidence they're wrong. Any one of the labels, the feature matrix or the scored catalogue would change the situation.

Two other things in the write-up: 1) the catalogue can't be derived from the method the 2022 paper describes, 2) and why my catalogue isn't the same list (stated by stage, because raw-detection agreement and candidate-level agreement are very different numbers and mine has been misquoted before):
https://github.com/jannefi/poss1-plate-slice/blob/main/docs/REFERENCE_CATALOGUE_REPRODUCIBILITY.md

MNRAS 515(1):1380 describes tessellating the sky in circular regions of 30 arcmin radius, fed by an image cutout service. No tessellation of that kind can produce this catalogue, and the reason is measurable without any access to it.

...The ceiling above implies a negative that is worth stating precisely: the catalogue was not produced by a position-blind tessellation of the sky, grid or circular, because no such tessellation reaches it.
...
What the released file cannot settle at all. It carries ra, dec and obs_date and no plate identifier or exposure time. A single observing date commonly covers several plates, so for most rows the exposure a row came from is not determined by the released columns. Any account of the construction that depends on knowing the plate per row is therefore not checkable against the public file, and we make none here.

POSS-II red plates

Separately, and I think more interesting: a control I've wanted for a long time. Run the identical pipeline on POSS-II red plates covering the same sky as POSS-I plates I've already processed. If the candidate rate were a property of my filters rather than of POSS-I, it should reproduce. It doesn't: 0.1243 ± 0.0152 over 30 plate pairs, about an eighth, and no stratum departs from it when split by declination, galactic latitude or plate grade.

The caveat matters as much as the number: POSS-II red is a different emulsion at a different depth and epoch, sampled 1.68× finer, so "fewer candidates on a cleaner, deeper plate" is an ordinary explanation this test can't exclude. It's same-sky, not same-instrument. Busko (2026, arXiv:2603.20407) is the comparison that does control for era and instrument class.

https://github.com/jannefi/poss1-plate-slice/blob/main/docs/POSS2_RED_CONTROL.md

The short answer: POSS-II yields about an eighth of POSS-I's candidates on the same sky — per-plate ratio 0.1243 ± 0.0152 over 30 plate pairs, pooled 644 / 5,939 = 0.1084. The suppression does not come from any one part of the sky or any one class of plate: split by declination, by galactic latitude and by plate grade, no stratum departs from it (Kruskal–Wallis p = 0.45, 0.38, 0.23 respectively). Whatever drives POSS-I's rate, this pipeline does not produce it from photographic plates in general.

Additional notes

Four notes on choices I made and used on my github repo:

I used "the VASCO 2025 sample", not the letter. I know the letter notation is Watters' and has been used in that thread before, including me. But it's now a locked convention in the repo and gate-enforced, so using it in a post while the repo bans it would be inconsistent the first time someone checks.

I put the "not evidence they're wrong" line in the body, not a footnote. In a thread started by the site's most prominent VASCO critic, that sentence is what keeps the post a reproducibility report rather than a pile-on, and it's the thing that preserves your ability to talk to the VASCO team afterwards.

I built the POSS-II caveat into the same breath as the number. A skeptic-leaning forum will read "an eighth" as "POSS-I candidates are mostly artifacts", which is exactly what it does not show.

On the zip: I'd avoid referencing or re-hosting it. The data in that repo isn't MIT. The licence covers the code, and the README grants the datasets only "for reproducibility of the associated publication".

 

[HoaxEye]

To be clear about what that does and doesn't mean: the published results can't currently be regenerated from the published artifacts. It is not evidence they're wrong. Any one of the labels, the feature matrix or the scored catalogue would change the situation.

I have made a lot of analysis and datasets concerning the VASCO 2024 sample (107K rows) and related source materials during this year. I will publish the relevant ones as reports, code etc. at some point.

I want to continue analyzing other photographic plate collections, but that is a significant effort. Suggestions/ideas are welcome.

 

[yoshy]

the published results can't currently be regenerated from the published artifacts

Incredibly frustrating and disappointing.

Excellent work as usual by you. Some of the best I've seen on this entire site

 

[yoshy]

Project Lost Light invites professional scientists, citizen scientists, and students from around the world to participate in a challenge to replicate -- or challenge -- pre-Sputnik transient studies using historical (pre-Sputnik) photographic plate collections from around the world.

First prize: $75,000
Second prize: $50,000
Third prize: $25,000

The competition accepts data from all pre-Sputnik photographic plate collections, except those from Palomar Observatory.

@HoaxEye you should enter this contest and get that bag!!!

Direct link to the contest: https://www.researchhub.com/grant/32931/project-lost-lights

Project Lost Light

Project Lost Light is a new open challenge to find out whether these flashes are real. We're inviting observatories, university teams and citizen scientists anywhere in the world to search old plate archives - from any observatory except Palomar - and test the findings for themselves[

Timeline

October 15th 2026 to January 15th 2027:

From January 15th 2027 (9 months):

Apply with a pre-registration on ResearchHub, detailing your exact experimental plans and analysis details.

Our team will invite expert reviewers to give you suggested feedback and support on your pre-registration.

Execute your pre-registered plan and submit your report, transient lists and code within 9 months (extra time allowed if digitizing a new collection).

Publish your results on an open access repository (e.g. arXiv) or journal (e.g. Nature).

Within two weeks after the submission deadline:

Judges will convene to choose award recipients based on criteria below.

 

[FatPhil]

Judges will convene to choose award recipients based on criteria below.

I'm reminded of the various "it only matters who counts the votes"-related quotes[*].

Who are the judges?

[* Quote Investigator has an interesting and fully-referenced deep dive into those here, there are more than you might expect, almost all pre-Stalin: https://quoteinvestigator.com/2021/01/28/count-votes/ if you want an OT side branch to deviate into over afternoon coffee.]

 

[yoshy]

I'm reminded of the various "it only matters who counts the votes"-related quotes[*].

Who are the judges?

[* Quote Investigator has an interesting and fully-referenced deep dive into those here, there are more than you might expect, almost all pre-Stalin: https://quoteinvestigator.com/2021/01/28/count-votes/ if you want an OT side branch to deviate into over afternoon coffee.]

"Based on criteria below". I kept scrolling and never actually found the criteria lol. I did find it on their other page Lost Light Project

Criteria
Transient searches should be based on pairs of images of the same field obtained during a single night. At later stages of the project, comparisons between images of the same field obtained with the same filter at different epochs are strongly recommended.
The required sky coverage depends on the scientific question being addressed. Investigations of the Earth shadow deficit require homogeneous sky coverage, whereas studies of the reported correlations with historical nuclear tests and Unidentified Aerial Phenomena do not.
The fraction of transients located within the Earth's shadow at a geosynchronous altitude of approximately 42,000 km should be investigated. Earth shadow analyses should account for the empirical sky coverage of the observations and compare the observed transient distribution with randomly generated control samples, following the methodology of Bruehl et al. (2026), Section 8. Control samples based on stars are not recommended, as their spatial distribution is affected by Galactic extinction and other astrophysical selection effects. Detailed methodological recommendations will be published alongside the code before the challenge opens.
All observations should have accurately documented timestamps and exposure times. The transient sample should provide sufficient statistical power for the scientific question being addressed. Initial searches should be restricted to optical images with exposure times of 30 minutes or longer, preferably obtained with telescopes of at least 0.8 m aperture.
Detection rates may vary between observatories owing to differences in geographical location, instrumentation, observing strategies, and other factors that have yet to be characterized.

I'm confused. This doesn't seem to establish any measurable criteria at all.

 

[yoshy]

Some more thoughts/questions on the ML paper. Figure 2 shows the SHAP values (Shapley values) from their model. SHAP values are used to quantify the contribution from a feature to the prediction, similar to the betas in a linear regression or GLM (generalized linear model). In simple regression, the betas represent the marginal contribution for a feature. That contribution is constant. For example, if you have two features, x_1 and x_2, beta_1 is the marginal contribution from x_1 and is always the same value regardless of the value of x_2.

\[ y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \epsilon \]

For tree ensemble models, such as random forest and GBDT, they model interactions so there is no single number as there is in linear regression. The contribution from x_1 will depend on the value of x_2, and vice versa. SHAP values are the contribution from a feature for an individual prediction, taking into account the values of all other features. Figure 2 in the paper shows the distribution of SHAP values for the features along with the feature value indicated by the color of the dot.

First, the paper is again misleading and written incorrectly.

Shapley (SHAP) valuesquantifying relative contributions of each predictor to the final ML model are displayed in Fig.

Recall that the final model is the average of 4 separate models. However, looking at the code, we can see that the SHAP values are calculated only on the random forest model, i.e. they represent the contributions for only 1 of the 4 models. See the following snippet from pipeline_train.py.

def compute_shap_values(models: dict, X: np.ndarray, feature_names: List[str],
                        output_dir: str) -> np.ndarray:
    """Compute SHAP values for the RF model and save summary plot."""
    try:
        import shap
        Xs = models['scaler'].transform(X)
        explainer = shap.TreeExplainer(models['rf'])

As well as this snippet from pipeline_score.py.

def compute_per_candidate_shap(models: dict, X: np.ndarray,
                                feature_names: List[str],
                                top_k: int = 3) -> pd.DataFrame:
    """Compute per-candidate SHAP top-k features using RF TreeExplainer."""
    try:
        import shap
        Xs = models['scaler'].transform(X)
        explainer = shap.TreeExplainer(models['rf'])

This is a mistake that should be caught in a competent peer review.

This reply is long and dense already, so I am going to write a followup discussing the results we see in Figure 2.

 

[yoshy]

tldr; the model is just a prediction of BV's opinion. "BV in, BV out"

The current study addresses several limitations of our prior work. Arguments that all transients are platedefects12,13 have been addressed by developing and validating an ML approach permitting quantitative discrimination between real transients and plate defects via ML-derived probability scores.

Before even getting to the data below, this is simply a baseless assertion. Even without the many issues, this methodology simply does not address all of the critiques. And I'll reiterate, the "real" transients that were used to train the model were designated as real by BV herself. Training a model on data that assumes you are correct will just give you a model that assumes you are correct on similar data.

On to interpreting the results of Figure 2. The model in paper is predicting if there are "real" transients in the image (with real being determined by BV). A positive SHAP value means the feature increased the probability of there being a real transient. In post #940, I mentioned how most of the features are either catalog- or plate-level features, and not features of the transients themselves. Figure 2 shows the contributions from each feature, ordered by the mean of their absolute value. I.e., the most influential feature on average is plate_low_snr_frac. The top 5 most influential features are all plate-level features.

Feature	Definition	SHAP value interpretation
plate_low_snr_frac	fraction of candidates on the same plate with SNR<5	Low feature values (blue dots) greatly increase the probability of "real" transient. Feature = # low SNR candidates / total # candidates on plate, so the better the SNR of the plate, the higher the probability of "real".
plate_quality_GOOD	Plate-level: binary, plate rated as GOOD (one-hot encoded)	One-hot encoded means the feature is either 1 (plate is rated "good") or 0 (plate is not rated "good"). SHAP indicates good quality plates decrease the probability of "real"
plate_quality_MODERATE	Plate-level: binary, plate rated as MODERATE (one-hot encoded)	MODERATE quality is nearly the exact opposite contribution of GOOD. Nearly identical magnitudes just different sign. Moderate plates => higher probability of "real"
plate_elongation_mean	Plate-level: mean elongation of all candidates on the same plate. Not defined any further in this paper.	Lots (all?) blue on right whereas left side has mix of red and blue. Low elongation can contribute in either direction while higher elongation decreases probability of "real". The average contribution is close to 0 from visual estimate.
plate_snr_std	Plate-level: standard deviation of SNR across all candidates on the same plate	Similar to elongation. Low standard deviation of SNR goes either way but high SNR decreases probability of "real"

SNR (signal-to-noise ratio) isn't described other than "FITS-measured SNR". The actual calculation of that SNR is not described, not even something like "FITS-measured SNR as calculated by software X". If it is the usual meaning of SNR, the human eye can perceive low vs high SNR. E.g., grainy or noisy images have lower SNR.

Plate quality is one of 'GOOD', 'MODERATE', 'POOR', 'NO_PLATE'. The determination of quality was not described in the paper. Note that POOR and NO_PLATE do not show up in the figure. This isn't explained anywhere, but from the code, those features were chosen to be excluded. The behavior of the GOOD and MODERATE features is extremely strange. Why would a good plate consistently decrease the probability of "real" while a moderate plate would increase the probability? What about cases where the plate is POOR (i.e. GOOD and MODERATE both 0, the blue dots)? Without a definition to confirm, I can only speculate, but I assume plate quality is again something evident to the naked eye.

Elongation and SNR standard deviation are harder to summarize due to the mixed contributions. candidate_score as mentioned before is another value calculated by BV in a previous work. Red gradient magnitude contribution appears pretty small average, with a cluster of low values that have positive contributions.

What can we conclude from the above? First, the peer review was a formality at best. There is a great deal of missing information, even beyond things that someone with better domain knowledge would know. Second, SHAP values are a great tool for model interpretation, but they don't explain everything. Partial dependence plots and plots of SHAP contribution vs feature value (feature value on the x axis, SHAP on y) would show a much clearer picture of their effects.

What are the important conclusions from this? The most important features for predicting "real" transients are visible to the naked eye. BV designated what is and what isn't real for the training data. The second reviewer was trained by BV, and in any case they disagreed, they discussed it together, meaning every single label is biased toward BV's opinion. Remember, this is being portrayed as a way to validate BV's earlier determinations with an "objective approach". Except it's not. Every step of the process has been determined by BV. It's data leakage all the way down. The model has simply learned how to predict what BV is going to say is a real transient.

@HoaxEye (since you know the data so well) or anyone else, please feel free to correct or add additional information where lacking.

 

[Mendel]

"Based on criteria below". I kept scrolling and never actually found the criteria lol. I did find it on their other page Lost Light Project

Criteria
Transient searches should be based on pairs of images of the same field obtained during a single night.

I think that's an entirely new method?
Unless we're back to the red vs. blue comparison?
That I think Villarroel never described in her papers? Or do I remember that wrong?

 

[yoshy]

Data is public now.


Link goes here https://www.researchhub.com/grant/32931/project-lost-lights

Table from the page with all the relevant links.

Recommendations for future replication efforts (B. Villarroel)	Read
Earth shadow computation code (G. Nir)	GitHub
ML-filtered POSS-I transient sample (vanish-bruehl26, SVO)	Catalog
How to cite the vanish-bruehl26 sample	Credits
Reference transient detection pipeline	https://github.com/dca-doherty/VASCO-challenge
Control sample construction code and methodological notes	Coming soon
Set of original Palomar plates, could be used for exploration purpose (STSCI)	https://archive.stsci.edu/missions/dss/FitsArchive/
Challenge website	https://lostlightproject.netlify.app/

 

[HoaxEye]

@HoaxEye you should enter this contest and get that bag!!!

Thanks, but I'll pass, and for a reason that has nothing to do with the rules: I've never taken money or donations for any of this and I'd rather keep it that way. Anything with a prize attached creates a conflict of interest or the appearance of one.

On the criteria, I think the criticism needs splitting in two, because the method requirements and the award criteria differ a lot in quality. The method requirements are specific enough to act on. Same-night image pairs, random control samples rather than star-based ones, exposures of 30 minutes or more, preferably 0.8 m or larger. Sounds fine.

The award criteria are another matter. The only stated basis for choosing among submissions is that "an independent prize committee judges how well the pre-registered plan was carried out". There's no rubric, no weighting, no score, no threshold, no benchmark. Correctness of the result explicitly isn't a criterion - "null and refuting results are equally eligible", which is defensible in itself, but it means the entire award turns on process fidelity as judged by an unnamed body.

Which is @FatPhil 's question, still unanswered, and it's worse than it looks. The complete public description of the judging body is the phrase "an independent prize committee". No members, no appointment process, no affiliations, and no conflict-of-interest policy of any kind. They clearly did think about conflicts for entrants, since there's an exclusion for researchers who've already published on this. The same consideration doesn't appear anywhere for the people choosing winners, in a competition to replicate specific authors' work.

One more thing worth flagging. The rule that data from Palomar is excluded doesn't appear on the challenge site or the grant page as far as I can find. I've only seen it in a post on X by BV. Meanwhile the challenge's own resource table offers entrants POSS-I full plate scans and an ML-filtered POSS-I sample. An eligibility rule that contradicts the published resource list and lives only in a tweet is going to catch people out.

One last thing, and it's the reason I think the criteria matter more than they might seem to. The challenge is being promoted with "are we alone on Earth, or are we witnessing the presence of another civilisation?" while the team's own response to Watters et al. says of this catalogue that "up to two thirds may be false positives, including plate defects", and the replication guidance says it "contains a mixed population, particularly near the plate edges". Those are their words, not mine, and I don't think the papers are wrong to hedge. But entrants are being recruited under the first framing and judged on how faithfully they follow a pre-registration, by a committee nobody has named. Pre-registration exists precisely to stop people finding what they were told to expect, and it only works if the people assessing it are visible.

 

[HoaxEye]

On the criteria, I think the criticism needs splitting in two, because the method requirements and the award criteria differ a lot in quality. The method requirements are specific enough to act on. Same-night image pairs, random control samples rather than star-based ones, exposures of 30 minutes or more, preferably 0.8 m or larger. Sounds fine.

Separately, on the "Recommendations for future replication efforts" that the challenge points entrants to. I've tried to follow this path for a while now, so here's what someone attempting it actually hits.

Where do you start? The advice is to read Solano et al. (2022) and the Commentary, then begin at the ~298,000-object stage in Solano et al. (2022) and continue through the later filtering to reach the ~107,000 catalogue. The difficulty is that there's no published route to 298,000. The method the 2022 paper describes: tessellating the sky in 30-arcmin-radius circular regions fed by a cutout service cannot produce that stage. I've measured the ceiling: plate selection in a cutout service is deterministic in position and changes on arcminute scales, so no position-blind tessellation gets past about 85% recall whatever its pitch, and a hundredfold increase in downloads buys about 8 points. So the instruction is precise and unexecutable at the same time: begin at a stage for which no published method exists.

Which scans? The recommendation is entire plate scans rather than cutouts, which I agree with, and the link given is to archive.stsci.edu/missions/dss/FitsArchive/XEsurvey. That's the one part I'd push back on, because it's the specific archive I ended up documenting a problem with. The STScI copy carries a structural pixel defect in a band near the plate edge, where its values run strongly negative and IRSA's don't, and on the astrometry the two archives disagree on about a fifth of plates. Using STScI's own 1994 CD-ROM headers as a third reference, IRSA is the one that matches and STScI is the archive that changed. Details and the comparison are here: https://github.com/jannefi/poss1-plate-slice/blob/main/docs/SCAN_SOURCE_SENSITIVITY.md Written for exactly this case, someone reproducing from STScI pixels. The recommendation gives no reason for that source beyond "to replicate the methodology in Solano et al. (2022)".

What settings? This is the gap that matters most. The recommendations contain nothing about SExtractor or PSFEx: no configuration, no detection threshold, no parameter names, no software versions, no detection code. The catalogue being public doesn't close that: you can now see which positions they selected, but not how they were selected from pixels, and those parameters are what a detection stage is. Combined with the handpicked training examples also being unpublished, the ML work can't be rebuilt from the plates by anyone, however carefully they follow this guidance. A reference pipeline is promised, but a reference pipeline written later isn't the one that produced the catalogue.

And one genuine disagreement rather than a gap. The recommendations say to normalise by total transient and control counts, and name per-plate normalisation as an error that "would effectively cancel the effect". My own work went the other way: I use a paired per-plate baseline, because a uniform random-sky baseline manufactures apparent enrichment that disappears when you pair properly. Both positions have a stated rationale and they can't both be right. That one's worth settling before anyone builds a shadow analysis on either.

 

[HoaxEye]

@HoaxEye (since you know the data so well) or anyone else, please feel free to correct or add additional information where lacking.

Checked. You're right on the parts that can be verified, with two corrections and two additions.

Confirmed: SHAP really is computed on one model of four - shap.TreeExplainer(models['rf']) at pipeline_train.py:135 and pipeline_score.py:40,44, against a build_ensemble docstring reading "the four-model ensemble". Your feature definitions match the paper's own table too: it defines plate_low_snr_frac as "fraction of candidates on the same plate with SNR < 5" in those words.

Two corrections. plate_quality_GOOD and plate_quality_MODERATE aren't two features behaving oddly: they're one binary variable encoded twice. The column takes exactly two values in the released file, GOOD 97,837 and MODERATE 10,038, summing to all 107,875 rows, so GOOD = 1 − MODERATE exactly. Mirror-image SHAP with equal magnitude and opposite sign is what perfect anticorrelation does to a redundant one-hot. That also means the blue dots aren't POOR plates: there are no POOR rows, so blue on plate_quality_GOOD is MODERATE. And POOR/NO_PLATE weren't quietly dropped: they're all-zero columns removed by a documented prevalence filter at 1%. The fair criticism stays. The paper doesn't mention it.

Two additions. The "plate-level" features aren't plate-level: the paper says "same plate", but the code groups by obs_date at all three call sites, and the GitHub file has no plate column at all, so grouping by plate wasn't possible with that input. On the SVO release, which does carry the plate, 187 of 310 dates cover more than one plate - 88,067 of 107,875 rows.

And plate_low_snr_frac mostly encodes missing data. snr is filled with 0 and then tested < 5, so every blank counts as low-SNR - and snr is populated on 319 of 107,875 rows, 0.30%. Reproducing the computation on the released file, the top-ranked feature comes out exactly 1.0 for 206 of 310 groups and above 0.99 for 78%. plate_snr_std is exactly 0 for 66% of groups, plate_n_high_snr for 69%.

One limit on that: Figure 2's SHAP is computed on the 250 training rows, and what I measured is the population being scored. Without the labels there's no telling whether the training rows sit in that 0.30% with real SNR. If they do, the model learned on a distribution that barely exists in what it scores. If they don't, the top feature is near-constant in training and its SHAP is hard to read. Both are problems, different ones, and I'd rather not guess.

So your conclusion holds and gets sharper: it isn't only that the model keys on plate properties rather than the transients. Part of what it keys on is whether the row has any measurements at all, grouped by night rather than by plate.