Transients in the Palomar Observatory Sky Survey

[JMartJr]

It's similar to the "why do alien spaceships have flashing lights all over them?" question that always leads to some "who can know the motivations of beings beyond our ken" type answer.

Best answer I ever heard for this was along the lines of "Maybe the lights we're seeing are in a wavelength THEY can;t see, they don;t even know how eye catching they are!" I don't buy it (they've abducted and tested enough people by now to know what wavelengths WE can see!) but it gets points for creative thinking, I guess...

 

[FatPhil]

Best answer I ever heard for this was along the lines of "Maybe the lights we're seeing are in a wavelength THEY can;t see, they don;t even know how eye catching they are!" I don't buy it (they've abducted and tested enough people by now to know what wavelengths WE can see!) but it gets points for creative thinking, I guess...

So these hyper-intelligent beings (with technological capabilities unimaginably more advanced than ours) are deliberately (as it's are part of the design of the craft) expending energy on something they can't make use of? I think that's ascribing a certain level of dumbness to the beings, which isn't what I would classify as "creative thinking", it's more like projection.

 

[jarlrmai]

Yes because aliens are can build amazing spacecraft but haunt figured out what Herchel etc discovered in the 1800's

 

[Mendel]

Best answer I ever heard for this was along the lines of "Maybe the lights we're seeing are in a wavelength THEY can;t see, they don;t even know how eye catching they are!" I don't buy it (they've abducted and tested enough people by now to know what wavelengths WE can see!) but it gets points for creative thinking, I guess...

Especially when the lights just so happen to meet aircraft regulations...

So these hyper-intelligent beings (with technological capabilities unimaginably more advanced than ours) are deliberately (as it's are part of the design of the craft) expending energy on something they can't make use of? I think that's ascribing a certain level of dumbness to the beings, which isn't what I would classify as "creative thinking", it's more like projection.

Well, so do we. A fair amount of gear gives off IR light—precisely because we can't see it. (e.g. security cameras, TV remotes)

 

[FatPhil]

Well, so do we. A fair amount of gear gives off IR light—precisely because we can't see it. (e.g. security cameras, TV remotes)

Yes, but we actively try to reduce it. For example, the transition away from incandescent lightbulbs in the EU and much of the world over the last couple of decades.

 

[GM4AJK]

expending energy on something they can't make use of?

Twitter/X would like a word

 

[jdog]

So these hyper-intelligent beings (with technological capabilities unimaginably more advanced than ours) are deliberately (as it's are part of the design of the craft) expending energy on something they can't make use of? I think that's ascribing a certain level of dumbness to the beings, which isn't what I would classify as "creative thinking", it's more like projection.

Veering slightly more off-topic, but in his new UAP-hunting memoir Out of the Shadows, Jay Stratton apparently contends that the white froth reported on the ocean surface under the Nimitz Tic-Tac was the seawater boiling from the unknown energy of the Tic-Tac's propulsion system -- which is a boggling assumption without any evidence, and suggests there should be a very hot column of air underneath every Tic-Tac and a heat trail wherever they go.

 

[HoaxEye]

They did just make a GitHub repo public which has some of the data, some of which seems newly public:

https://github.com/dca-doherty/VASCO-ML

MIT License, so I've attached a .zip here:

Thanks Mick. Worth adding what is and isn't in that release, because it changes what anyone can do with it.


The 107,875-row file is the VASCO 2025 sample. The detection output behind Solano et al. (2022). It carries RA, Dec and observation date plus some catalogue-level columns. What it doesn't carry is the per-candidate ML scores behind the paper's figures: the ml_probability column takes 287 distinct values across 107,875 rows, and 99.7% of rows share one identical value, while the committed decile tables span a continuous range in ten equal bins. The scored catalogue, the 250 training labels and the image-derived features aren't in the repo either.

I tried the documented reproduction in their own pinned environment. reproduce.py --verify-only passes 22/22, but it only compares committed outputs against committed expectations - nothing is recomputed. The full run stops in half a second at stage one of four. Working around each obstacle in turn: the config points at data/raw/ while the file ships at data/; requirements.txt omits seaborn, which the training stage imports; feature extraction then reports OK in under a second while producing a cache containing none of the 16 image-derived features (the config points at a D: path); and training stops on "No label files loaded", which can't be worked around.

To be clear about what that does and doesn't mean: the published results can't currently be regenerated from the published artifacts. It is not evidence they're wrong. Any one of the labels, the feature matrix or the scored catalogue would change the situation.

Two other things in the write-up: 1) the catalogue can't be derived from the method the 2022 paper describes, 2) and why my catalogue isn't the same list (stated by stage, because raw-detection agreement and candidate-level agreement are very different numbers and mine has been misquoted before):
https://github.com/jannefi/poss1-plate-slice/blob/main/docs/REFERENCE_CATALOGUE_REPRODUCIBILITY.md

MNRAS 515(1):1380 describes tessellating the sky in circular regions of 30 arcmin radius, fed by an image cutout service. No tessellation of that kind can produce this catalogue, and the reason is measurable without any access to it.

...The ceiling above implies a negative that is worth stating precisely: the catalogue was not produced by a position-blind tessellation of the sky, grid or circular, because no such tessellation reaches it.
...
What the released file cannot settle at all. It carries ra, dec and obs_date and no plate identifier or exposure time. A single observing date commonly covers several plates, so for most rows the exposure a row came from is not determined by the released columns. Any account of the construction that depends on knowing the plate per row is therefore not checkable against the public file, and we make none here.

POSS-II red plates

Separately, and I think more interesting: a control I've wanted for a long time. Run the identical pipeline on POSS-II red plates covering the same sky as POSS-I plates I've already processed. If the candidate rate were a property of my filters rather than of POSS-I, it should reproduce. It doesn't: 0.1243 ± 0.0152 over 30 plate pairs, about an eighth, and no stratum departs from it when split by declination, galactic latitude or plate grade.

The caveat matters as much as the number: POSS-II red is a different emulsion at a different depth and epoch, sampled 1.68× finer, so "fewer candidates on a cleaner, deeper plate" is an ordinary explanation this test can't exclude. It's same-sky, not same-instrument. Busko (2026, arXiv:2603.20407) is the comparison that does control for era and instrument class.

https://github.com/jannefi/poss1-plate-slice/blob/main/docs/POSS2_RED_CONTROL.md

The short answer: POSS-II yields about an eighth of POSS-I's candidates on the same sky — per-plate ratio 0.1243 ± 0.0152 over 30 plate pairs, pooled 644 / 5,939 = 0.1084. The suppression does not come from any one part of the sky or any one class of plate: split by declination, by galactic latitude and by plate grade, no stratum departs from it (Kruskal–Wallis p = 0.45, 0.38, 0.23 respectively). Whatever drives POSS-I's rate, this pipeline does not produce it from photographic plates in general.

Additional notes

Four notes on choices I made and used on my github repo:

I used "the VASCO 2025 sample", not the letter. I know the letter notation is Watters' and has been used in that thread before, including me. But it's now a locked convention in the repo and gate-enforced, so using it in a post while the repo bans it would be inconsistent the first time someone checks.

I put the "not evidence they're wrong" line in the body, not a footnote. In a thread started by the site's most prominent VASCO critic, that sentence is what keeps the post a reproducibility report rather than a pile-on, and it's the thing that preserves your ability to talk to the VASCO team afterwards.

I built the POSS-II caveat into the same breath as the number. A skeptic-leaning forum will read "an eighth" as "POSS-I candidates are mostly artifacts", which is exactly what it does not show.

On the zip: I'd avoid referencing or re-hosting it. The data in that repo isn't MIT. The licence covers the code, and the README grants the datasets only "for reproducibility of the associated publication".

 

[HoaxEye]

To be clear about what that does and doesn't mean: the published results can't currently be regenerated from the published artifacts. It is not evidence they're wrong. Any one of the labels, the feature matrix or the scored catalogue would change the situation.

I have made a lot of analysis and datasets concerning the VASCO 2024 sample (107K rows) and related source materials during this year. I will publish the relevant ones as reports, code etc. at some point.

I want to continue analyzing other photographic plate collections, but that is a significant effort. Suggestions/ideas are welcome.

 

[yoshy]

the published results can't currently be regenerated from the published artifacts

Incredibly frustrating and disappointing.

Excellent work as usual by you. Some of the best I've seen on this entire site

 

[yoshy]

Project Lost Light invites professional scientists, citizen scientists, and students from around the world to participate in a challenge to replicate -- or challenge -- pre-Sputnik transient studies using historical (pre-Sputnik) photographic plate collections from around the world.

First prize: $75,000
Second prize: $50,000
Third prize: $25,000

The competition accepts data from all pre-Sputnik photographic plate collections, except those from Palomar Observatory.

@HoaxEye you should enter this contest and get that bag!!!

Direct link to the contest: https://www.researchhub.com/grant/32931/project-lost-lights

Project Lost Light

Project Lost Light is a new open challenge to find out whether these flashes are real. We're inviting observatories, university teams and citizen scientists anywhere in the world to search old plate archives - from any observatory except Palomar - and test the findings for themselves[

Timeline

October 15th 2026 to January 15th 2027:

From January 15th 2027 (9 months):

Apply with a pre-registration on ResearchHub, detailing your exact experimental plans and analysis details.

Our team will invite expert reviewers to give you suggested feedback and support on your pre-registration.

Execute your pre-registered plan and submit your report, transient lists and code within 9 months (extra time allowed if digitizing a new collection).

Publish your results on an open access repository (e.g. arXiv) or journal (e.g. Nature).

Within two weeks after the submission deadline:

Judges will convene to choose award recipients based on criteria below.

 

[FatPhil]

Judges will convene to choose award recipients based on criteria below.

I'm reminded of the various "it only matters who counts the votes"-related quotes[*].

Who are the judges?

[* Quote Investigator has an interesting and fully-referenced deep dive into those here, there are more than you might expect, almost all pre-Stalin: https://quoteinvestigator.com/2021/01/28/count-votes/ if you want an OT side branch to deviate into over afternoon coffee.]

 

[yoshy]

I'm reminded of the various "it only matters who counts the votes"-related quotes[*].

Who are the judges?

[* Quote Investigator has an interesting and fully-referenced deep dive into those here, there are more than you might expect, almost all pre-Stalin: https://quoteinvestigator.com/2021/01/28/count-votes/ if you want an OT side branch to deviate into over afternoon coffee.]

"Based on criteria below". I kept scrolling and never actually found the criteria lol. I did find it on their other page Lost Light Project

Criteria
Transient searches should be based on pairs of images of the same field obtained during a single night. At later stages of the project, comparisons between images of the same field obtained with the same filter at different epochs are strongly recommended.
The required sky coverage depends on the scientific question being addressed. Investigations of the Earth shadow deficit require homogeneous sky coverage, whereas studies of the reported correlations with historical nuclear tests and Unidentified Aerial Phenomena do not.
The fraction of transients located within the Earth's shadow at a geosynchronous altitude of approximately 42,000 km should be investigated. Earth shadow analyses should account for the empirical sky coverage of the observations and compare the observed transient distribution with randomly generated control samples, following the methodology of Bruehl et al. (2026), Section 8. Control samples based on stars are not recommended, as their spatial distribution is affected by Galactic extinction and other astrophysical selection effects. Detailed methodological recommendations will be published alongside the code before the challenge opens.
All observations should have accurately documented timestamps and exposure times. The transient sample should provide sufficient statistical power for the scientific question being addressed. Initial searches should be restricted to optical images with exposure times of 30 minutes or longer, preferably obtained with telescopes of at least 0.8 m aperture.
Detection rates may vary between observatories owing to differences in geographical location, instrumentation, observing strategies, and other factors that have yet to be characterized.

I'm confused. This doesn't seem to establish any measurable criteria at all.

 

[yoshy]

Some more thoughts/questions on the ML paper. Figure 2 shows the SHAP values (Shapley values) from their model. SHAP values are used to quantify the contribution from a feature to the prediction, similar to the betas in a linear regression or GLM (generalized linear model). In simple regression, the betas represent the marginal contribution for a feature. That contribution is constant. For example, if you have two features, x_1 and x_2, beta_1 is the marginal contribution from x_1 and is always the same value regardless of the value of x_2.

\[ y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \epsilon \]

For tree ensemble models, such as random forest and GBDT, they model interactions so there is no single number as there is in linear regression. The contribution from x_1 will depend on the value of x_2, and vice versa. SHAP values are the contribution from a feature for an individual prediction, taking into account the values of all other features. Figure 2 in the paper shows the distribution of SHAP values for the features along with the feature value indicated by the color of the dot.

First, the paper is again misleading and written incorrectly.

Shapley (SHAP) valuesquantifying relative contributions of each predictor to the final ML model are displayed in Fig.

Recall that the final model is the average of 4 separate models. However, looking at the code, we can see that the SHAP values are calculated only on the random forest model, i.e. they represent the contributions for only 1 of the 4 models. See the following snippet from pipeline_train.py.

def compute_shap_values(models: dict, X: np.ndarray, feature_names: List[str],
                        output_dir: str) -> np.ndarray:
    """Compute SHAP values for the RF model and save summary plot."""
    try:
        import shap
        Xs = models['scaler'].transform(X)
        explainer = shap.TreeExplainer(models['rf'])

As well as this snippet from pipeline_score.py.

def compute_per_candidate_shap(models: dict, X: np.ndarray,
                                feature_names: List[str],
                                top_k: int = 3) -> pd.DataFrame:
    """Compute per-candidate SHAP top-k features using RF TreeExplainer."""
    try:
        import shap
        Xs = models['scaler'].transform(X)
        explainer = shap.TreeExplainer(models['rf'])

This is a mistake that should be caught in a competent peer review.

This reply is long and dense already, so I am going to write a followup discussing the results we see in Figure 2.

 

[yoshy]

tldr; the model is just a prediction of BV's opinion. "BV in, BV out"

The current study addresses several limitations of our prior work. Arguments that all transients are platedefects12,13 have been addressed by developing and validating an ML approach permitting quantitative discrimination between real transients and plate defects via ML-derived probability scores.

Before even getting to the data below, this is simply a baseless assertion. Even without the many issues, this methodology simply does not address all of the critiques. And I'll reiterate, the "real" transients that were used to train the model were designated as real by BV herself. Training a model on data that assumes you are correct will just give you a model that assumes you are correct on similar data.

On to interpreting the results of Figure 2. The model in paper is predicting if there are "real" transients in the image (with real being determined by BV). A positive SHAP value means the feature increased the probability of there being a real transient. In post #940, I mentioned how most of the features are either catalog- or plate-level features, and not features of the transients themselves. Figure 2 shows the contributions from each feature, ordered by the mean of their absolute value. I.e., the most influential feature on average is plate_low_snr_frac. The top 5 most influential features are all plate-level features.

Feature	Definition	SHAP value interpretation
plate_low_snr_frac	fraction of candidates on the same plate with SNR<5	Low feature values (blue dots) greatly increase the probability of "real" transient. Feature = # low SNR candidates / total # candidates on plate, so the better the SNR of the plate, the higher the probability of "real".
plate_quality_GOOD	Plate-level: binary, plate rated as GOOD (one-hot encoded)	One-hot encoded means the feature is either 1 (plate is rated "good") or 0 (plate is not rated "good"). SHAP indicates good quality plates decrease the probability of "real"
plate_quality_MODERATE	Plate-level: binary, plate rated as MODERATE (one-hot encoded)	MODERATE quality is nearly the exact opposite contribution of GOOD. Nearly identical magnitudes just different sign. Moderate plates => higher probability of "real"
plate_elongation_mean	Plate-level: mean elongation of all candidates on the same plate. Not defined any further in this paper.	Lots (all?) blue on right whereas left side has mix of red and blue. Low elongation can contribute in either direction while higher elongation decreases probability of "real". The average contribution is close to 0 from visual estimate.
plate_snr_std	Plate-level: standard deviation of SNR across all candidates on the same plate	Similar to elongation. Low standard deviation of SNR goes either way but high SNR decreases probability of "real"

SNR (signal-to-noise ratio) isn't described other than "FITS-measured SNR". The actual calculation of that SNR is not described, not even something like "FITS-measured SNR as calculated by software X". If it is the usual meaning of SNR, the human eye can perceive low vs high SNR. E.g., grainy or noisy images have lower SNR.

Plate quality is one of 'GOOD', 'MODERATE', 'POOR', 'NO_PLATE'. The determination of quality was not described in the paper. Note that POOR and NO_PLATE do not show up in the figure. This isn't explained anywhere, but from the code, those features were chosen to be excluded. The behavior of the GOOD and MODERATE features is extremely strange. Why would a good plate consistently decrease the probability of "real" while a moderate plate would increase the probability? What about cases where the plate is POOR (i.e. GOOD and MODERATE both 0, the blue dots)? Without a definition to confirm, I can only speculate, but I assume plate quality is again something evident to the naked eye.

Elongation and SNR standard deviation are harder to summarize due to the mixed contributions. candidate_score as mentioned before is another value calculated by BV in a previous work. Red gradient magnitude contribution appears pretty small average, with a cluster of low values that have positive contributions.

What can we conclude from the above? First, the peer review was a formality at best. There is a great deal of missing information, even beyond things that someone with better domain knowledge would know. Second, SHAP values are a great tool for model interpretation, but they don't explain everything. Partial dependence plots and plots of SHAP contribution vs feature value (feature value on the x axis, SHAP on y) would show a much clearer picture of their effects.

What are the important conclusions from this? The most important features for predicting "real" transients are visible to the naked eye. BV designated what is and what isn't real for the training data. The second reviewer was trained by BV, and in any case they disagreed, they discussed it together, meaning every single label is biased toward BV's opinion. Remember, this is being portrayed as a way to validate BV's earlier determinations with an "objective approach". Except it's not. Every step of the process has been determined by BV. It's data leakage all the way down. The model has simply learned how to predict what BV is going to say is a real transient.

@HoaxEye (since you know the data so well) or anyone else, please feel free to correct or add additional information where lacking.

 

[Mendel]

"Based on criteria below". I kept scrolling and never actually found the criteria lol. I did find it on their other page Lost Light Project

Criteria
Transient searches should be based on pairs of images of the same field obtained during a single night.

I think that's an entirely new method?
Unless we're back to the red vs. blue comparison?
That I think Villarroel never described in her papers? Or do I remember that wrong?