Transients in the Palomar Observatory Sky Survey

[Mendel]

The authors did exactly that: they tested every instrumental and astronomical explanation available to them, found most fit, and a few that didn't.

While this is technically true, it has no bearing on this paper, because it uses a data set that does not fit these criteria.

Article:

Discovering vanishing objects in POSS I red images using the Virtual Observatory
Enrique Solano, B Villarroel, C Rodrigo
Monthly Notices of the Royal Astronomical Society, Volume 515, Issue 1, September 2022, Pages 1380–1391, https://doi.org/10.1093/mnras/stac1552

Figure 9.
Spatial distribution in galactic coordinates of the final list of 5399 candidates (blue dots). A coloured POSS-II images is displayed in the background.

Source: https://academic.oup.com/mnras/article/515/1/1380/6607509?login=false

The paper applies "instrumental and astrological" criteria and ends up with " remaining unidentified transients (5399)". However, the paper never mentions damage from plate deformation.

I would comment on the uneven distribution of these points, but it doesn't actually matter here.

Article:

Aligned, Multiple-transient Events in the First Palomar Sky Survey
Beatriz Villarroel, Enrique Solano, Hichem Guergouri, Alina Streblyanska, Stephen Bruehl, Vitaly M. Andruk, Lars Mattsson, Rudolf E. Bär, Jamal Mimouni, Stefan Geier
Published 2025 October 17
Citation Beatriz Villarroel et al 2025 PASP 137 104504 DOI 10.1088/1538-3873/ae0afe

We use the transient candidates from Solano et al. (2022), but with the additional requirement that they have no counterparts within 5″ in Gaia, Pan-STARRS and NeoWise. Furthermore, we restrict our analysis to objects in the northern hemisphere (decl. > 0°). This yields a sample of 106,339 transients, which we use for our study.

Source: https://iopscience.iop.org/article/10.1088/1538-3873/ae0afe#paspae0afes4

Elsewhere, the paper mentions 167 transients per plate, which confirms this number.

You'll notice that 106,339 far exceeds the 5399 data points that Solano(2022) arrived at by "testing every instrumental and astronomical explanation available to them" (but not including plate deformation defects). Because 106339 >> 5399, the data set in this paper includes over 100,000 points that have an "instrumental or astrological explanation".

Villarroel knows this, because she co-authored Solano(2022) and cites it here: "a set of ∼5000 short-lived POSS-I transients (Solano et al. 2022). This highly curated data set, in which diagnostics based on photometry and morphometric parameters have been carefully applied to the sample to reduce false positives (e.g., plate defects), suggests that the phenomenon of multiple transients can be found even when stringent diagnostic criteria are applied."

For reasons unknown to me, she opted not to use this curated data set for the IOPscience paper.

 

[Mauro]

When I said "rule out all non-instrumental causes," I obviously meant "all known instrumental causes." Otherwise every paper in astronomy would have to end with "we can't rule out gremlins in the optics." ‍

I don't understand your logic (or your writing). When you said "rule out all non-instrumental causes" how could you mean "all known instrumental causes" instead? Non-instrumental (known or unknow) does not overlap by definition with instrumental (known or unknown).

Fair catch - in pure logic, yes, you can't prove a negative. But that's not what "ruling out" means in empirical work. In science we rule out in the provisional sense by exhausting known mechanisms within the resolution of the data. The authors did exactly that: they tested every instrumental and astronomical explanation available to them, found most fit, and a few that didn't.

And this is not the problem of their paper, on the contrary, every scientist hopes to find something unknown! Had they stopped there, that would be fine.

The problem arises when they cherry-pick their pet hypothesis (aliens) even if the evidence against this hypothesis (you may call it the 'prior', but 'prior' just means 'all past evidence') is overwhelmingly against it (1 to 10EXPa-big-number against) and when much more probable explanations are available (ie.: some unknown instrumental cause).

And the problem gets worse when they doubled down on this by looking for correlations with nuclear tests, which even if they were real (which I very much doubt) are evidence against alien satellites, because they require bizarre aliens, thus, even more improbable ones. They are able to come to know the day we're going to do a nuclear test, but then, misteriously, they need to send in satellites in geosynchronous orbit to observe it; then they often send the satellite to the wrong emisphere; they are misteriously no more interested after a few years (and, coincidentally, when a new type of photographic plates started to be used). Geez.

 

[Montauk]

Apologize if it's already been posted, but I thought I'd share Hank Green's videos with commentary on the paper and the responses to it by the UAP community:

He published the first 5 days ago:


Source: https://youtu.be/sZYSjqr6mIc

In this one, Green's take(s) are that:

• People often jump to "aliens" because it's exciting and fits what they want to believe, not because the evidence supports it.
• Extraordinary claims require extraordinary, falsifiable evidence.
• Good evidence means high-quality data from multiple independent instruments that rule out natural and instrumental explanations.
• "Unexplained" does not equal "alien." Lack of an immediate explanation is common.
• Observation and selection bias (the way we look and where we look) can make findings seem more surprising than they actually are.
• The alien hypothesis can become unfalsifiable (i.e., used to explain anything,) which shuts down real investigation.
• Perverse incentives (clicks, attention, book deals) encourage quick, headline-friendly alien claims.

His final takeaway is to stay open to life elsewhere but insist on rigorous, repeatable, testable evidence; or as shown on his (imo) cool t-shirt, we want to know, not just to believe.

Then, after receiving a multitude of angry and polemical comments from a bevy of UAP believers, he published the second (follow-up/response to the aforementioned comments) video 12 hours ago:


Source: https://youtu.be/DP9qYWhbSuQ

In this one, Green:

• is not convinced we're alone (the universe is huge) but emphasizes we only have one data point, so frequency is unknown; he avoids the word "believe."
• shares his view that the "Buga Sphere" is a hoax
• explains how brightness/distance misperception matters when investigating cases. He uses the example of the "Phoenix Lights"; investigations of it pointed to a formation of military planes for one event and dropped flares for the other.

And most importantly for this thread:

• shares his view that the Palomar "transients" are real but likely non-alien.
• explains further that the paper shows the spots appear when sunlit, correlate fairly tightly with atmospheric nuclear tests, and could be tumbling debris/glints rather than extraterrestrials.

Feel free to delete if someone before me shared this already (although I don't think anyone has shared the above second video before me)

 

[jarlrmai]

This shows the issue with the current situation and in some ways demonstrates Ufology as a microcosm of the problem of social media.

Science and replication takes time, but this paper has been doing the rounds in public Ufo circles since the pre-print was uploaded to the internet, they chummed the water and got ahead of the narrative.

If you are a person used to being on social media then you are used to the angry UAP crowd, but a potential non public subject matter expert replicating this paper and then publicising the results (that might demonstrate flaws in the method/conclusions) is likely to involve

Taking a lot of time and effort (potentially even more time than the original paper/s)
Probably going to be complicated and wordy and thus not a "slam dunk"
Putting their head up into the howling gale of Ufo fandom (social media abuse etc)

 

[Eburacum]

The authors did exactly that: they tested every instrumental and astronomical explanation available to them, found most fit, and a few that didn't.

This still isn't correct. The anomalies that they are highlighting have a perfectly plausible explanation (emulsion anomalies), which Villaroel admits cannot be dismissed as an explanation. They have found 'patterns' in a small number of these anomalies which they think are significant, but these patterns do not rule out the emulsion flaw explanation.

The anomalies can be interpreted in a number of ways, and one of those ways is by emulsion flaws. They can also be explained by very large, dark, irregularly shaped objects with tiny reflective mirrors on them, but we don't even know whether these 'flashes' occurred at regular intervals, or in sequential order (if they occurred at all).

 

[Mendel]

I thought I'd share Hank Green's take(s) and commentary on the paper

you shared the videos, but not the take, which means I won't know what it is

 

[orianda]

This still isn't correct. The anomalies that they are highlighting have a perfectly plausible explanation (emulsion anomalies), which Villaroel admits cannot be dismissed as an explanation. They have found 'patterns' in a small number of these anomalies which they think are significant, but these patterns do not rule out the emulsion flaw explanation.

The anomalies can be interpreted in a number of ways, and one of those ways is by emulsion flaws. They can also be explained by very large, dark, irregularly shaped objects with tiny reflective mirrors on them, but we don't even know whether these 'flashes' occurred at regular intervals, or in sequential order (if they occurred at all).

You (accidentally) just reinforced my point. The authors explicitly acknowledge that emulsion flaws are possible, but that's not the same as demonstrated. Acknowledging uncertainty isn't a concession, it's good science.

The fact that the residual transients don't match the morphology or distribution of known emulsion defects is precisely what made them worth further study. That's why Villarroel et al. shared their methods and dataset: so others could test whether those residuals really are instrument artifacts.

If you're convinced they are, great! That's a falsifiable claim. The next step is to publish a replication or rebuttal using those same plates. That's how scientific disagreements get resolved. Forum conjecture, however confident, isn't the same as evidence.

 

[JMartJr]

Forum conjecture, however confident, isn't the same as evidence.

This is a forum where we crowd source trying to tease data out of this sort of stuff. It is a process that has seen a good deal of success...

 

[orianda]

This is a forum where we crowd source trying to tease data out of this sort of stuff. It is a process that has seen a good deal of success...

Duly noted, my point is just that theories traded here are still just that, theories. The humility comes in keeping them provisional until they're tested. Villarroel et al. at least put theirs into a framework that others can falsify. That's the part that makes it science...otherwise we risk falling into the very traps debunking is meant to expose (false confidence in priors, selective reasoning, etc.)

 

[Ann K]

The fact that the residual transients don't match the morphology or distribution of known emulsion defects is precisely what made them worth further study.

Do we know that? Are there still unexposed plates using the older emulsion that have been examined for defects, perhaps submitting them to the same kind of handling processes and then developing them without exposure?

And the word is "hypotheses", not "theories"...

 

[HoaxEye]

Do we know that? Are there still unexposed plates using the older emulsion that have been examined for defects?

I don't think we need unexposed plates to know.

Edit: post updated to follow Metabunk link policy (see the next post)

See:
https://arxiv.org/abs/2402.00497

On the nature of apparent transient sources on the National Geographic Society-Palomar Observatory Sky Survey glass copy plates
Nigel Hambly, Adam Blair

We examine critically recent claims for the presence of above-atmosphere optical transients in publicly-available digitised scans of Schmidt telescope photographic plate material derived from the National Geographic Society-Palomar Observatory Sky Survey. We employ the publicly available SuperCOSMOS Sky Survey catalogues to examine statistically the morphology of the sources. We develop a simple, objective and automated image classification scheme based on a random forest decision tree classifier. We find that the putative transients are likely to be spurious artefacts of the photographic emulsion. We suggest a possible cause of the appearance of these images as resulting from the copying procedure employed to disseminate glass copy survey atlas sets in the era before large-scale digitisation programmes.

https://pages.astronomy.ua.edu/keel/techniques/photo.html

The emulsions most commonly used in astronomy were long the so-called spectroscopic emulsions from Kodak, such as 103a-O, IIa-O, IIIa-J, and IIIa-F. The "a" denotes a special process to reduce reciprocity failure, the greatest special problem in astronomical photography. Normally, the product of intensity and exposure time gives the level of opacity in the emulsion (the reciprocity law). However, for long exposures, the emulsion becomes relatively less efficient, so that for ordinary emulsions long exposures eventually give no added advantage. Astronomical emulsions are designed to minimize this, so that while slower for very short exposures, they are much faster at low light levels and long exposures. There also exist various sensitizing or hypersensitizing tecgniques that may decrease reciprocity failure and fog as well as increasing sensitivity. These include:

• cooling, often to solid CO2​ temperatures (dry ice). This reduces thermal effects, and was one of the first effective ways of reducing reciprocity failure.
• baking at a few hundred degrees C in either nitrogen, hydrogen (very dangerous!) or forming gas (nitrogen with 5% hydrogen). Even vacuum baking often helps. Gains from this step are so large that class III emulsions are almost never used without such preprocessing.
• liquid baths, removing silver halide ions to increase the concentration of silver ions.
• preflashing, applying enough extra exposure to put the emulsion above the toe of its HD curve. This is not so much sensitizing as making most effective use of the emulsion's existing properties for detecting faint sources.

https://resources.culturalheritage.org/pmgtopics/2009-volume-thirteen/13_13_Weaver.html

Dye clouds in Kodak negative working prints can be divided into three distinct eras when examined with reflected cross-polarized light at very high magnifications (a 500x viewing magnification on a compound microscope was used in this study). Changes in emulsion manufacture and processing resulted in the edges of dye clouds becoming less diffuse over time, and this can be used in the dating of prints by classification into one of the three groups. The first period is from 1942 through the 1960s, and is identifiable by diffuse dye clouds. Starting in the early 1970s, dye clouds become slightly more defined, having a relatively circular shape with a moderately defined edge. This lasts until the early 1980s when dye clouds become very well defined with hard edges. This period continues to the current day. Since dye clouds in negative and positive working prints evolved at different rates it is imperative to determine layer order, and thus material, before using dye clouds to date prints.

 

[Mendel]

I don't think we need unexposed plates to know. See e.g. https://arxiv.org/pdf/2402.00497 or https://www.orau.org/health-physics-museum/collection/nuclear-weapons/trinity/kodak-film.html

Please review the Metabunk link policy : https://www.metabunk.org/threads/metabunks-link-policy.5158/

You really need to quote from or describe your links; there are few exceptions, but your links are not.

Also, please do not deeplink arxiv PDFs (or similar); if there is a landing page, link that instead, or in addition to the deeplink.

Quote using EX tags (square with arrow icon).

https://arxiv.org/abs/2402.00497

On the nature of apparent transient sources on the National Geographic Society-Palomar Observatory Sky Survey glass copy plates
Nigel Hambly, Adam Blair

We examine critically recent claims for the presence of above-atmosphere optical transients in publicly-available digitised scans of Schmidt telescope photographic plate material derived from the National Geographic Society-Palomar Observatory Sky Survey. We employ the publicly available SuperCOSMOS Sky Survey catalogues to examine statistically the morphology of the sources. We develop a simple, objective and automated image classification scheme based on a random forest decision tree classifier. We find that the putative transients are likely to be spurious artefacts of the photographic emulsion. We suggest a possible cause of the appearance of these images as resulting from the copying procedure employed to disseminate glass copy survey atlas sets in the era before large-scale digitisation programmes.

The orau.org link is irrelevant: "Kodak Film Fogged by the Trinity Test (1945)" is really not a factor here.

 

[HoaxEye]

Please review the Metabunk link policy : https://www.metabunk.org/threads/metabunks-link-policy.5158/

You really need to quote from or describe your links; there are few exceptions, but your links are not.

Also, please do not deeplink arxiv PDFs (or similar); if there is a landing page, link that instead, or in addition to the deeplink.

Quote using EX tags (square with arrow icon).

https://arxiv.org/abs/2402.00497

On the nature of apparent transient sources on the National Geographic Society-Palomar Observatory Sky Survey glass copy plates
Nigel Hambly, Adam Blair

We examine critically recent claims for the presence of above-atmosphere optical transients in publicly-available digitised scans of Schmidt telescope photographic plate material derived from the National Geographic Society-Palomar Observatory Sky Survey. We employ the publicly available SuperCOSMOS Sky Survey catalogues to examine statistically the morphology of the sources. We develop a simple, objective and automated image classification scheme based on a random forest decision tree classifier. We find that the putative transients are likely to be spurious artefacts of the photographic emulsion. We suggest a possible cause of the appearance of these images as resulting from the copying procedure employed to disseminate glass copy survey atlas sets in the era before large-scale digitisation programmes.

The orau.org link is irrelevant: "Kodak Film Fogged by the Trinity Test (1945)" is really not a factor here.

Sorry and thanks for the timely reminder! I was in a rush (as usual )

 

[Montauk]

you shared the videos, but not the take, which means I won't know what it is

You're right. I just amended the post to give a summary of both videos. It was late when I thought of posting this so I didn't have time to watch and take notes. Thanks for the heads up.

 

[Eburacum]

You (accidentally) just reinforced my point. The authors explicitly acknowledge that emulsion flaws are possible, but that's not the same as demonstrated. Acknowledging uncertainty isn't a concession, it's good science.
The fact that the residual transients don't match the morphology or distribution of known emulsion defects is precisely what made them worth further study.

But they do match the morphology of known emulsion defects. They explicitly stated this in this paper, which you have so far failed to acknowledge.

Thus, profile sharpness alone cannot conclusively distinguish between artifact and astrophysical origin.

Or are you disregarding this statement?

Since it is not possible to 'conclusively distinguish between artifact and astrophysical origin', they have attempted to demonstrate artificiality by looking at the distribution of these anomalies. This is the claim under discussion here, and it may be possible to show that the distribution is either not anomalous or that the distribution is caused by effects that Villaroel et al. have not yet considered.

 

[NorCal Dave]

That's why Villarroel et al. shared their methods and dataset: so others could test whether those residuals really are instrument artifacts.

Did they? There is an entire thread were people are trying to figure out exactly what Villarroel was looking at and how they were deciding what was an actual unexplained transient/possible techno-signature:

https://www.metabunk.org/threads/digitized-sky-survey-poss-1.14385/

To quote @HoaxEye from that thread, who has assembled most of the needed software:

Again: Villarroel et al. really should release all software and exact data sources. They want others to reproduce their work and work on new datasets, but I don't see how that's possible right now.

If all the data and software is readily available and you know of it, maybe clue these guys in and save them the trouble of trying to reconstruct what they think Villarroel was doing and finding the exact dataset.

The paper correlating nuclear tests and UFOs to the supposedly unexplained transients from the previous paper is completely lacking in any kind of data. We are told that nuclear test dates can be looked up on a number of websites. Then we are told there is a statistically relevant correlation between these test dates and the appearance of some transients. Not even one example is given of a nuclear test date and a particular plate with a particular transient appearing within a 3 day window of the test date. Let a lone an actual table or list of ALL the times a test date coincided with a 3 day window of a transients appearing on a specific plate. As with the supposed UFO links, it's all just "Trust me bro, they're aligned". Or maybe more accurately, "Go figure it out for yourself".

Maybe I'm just missing something.

 

[MapperGuy]

We are told that nuclear test dates can be looked up on a number of websites.

And I suspect there are many such lists and they are not all identical and do not contain identical details. So the data THEY actually used needs to be published, so the accuracy of THAT list can be evaluated and then further work can be done.

Just saying 'you can find that data for yourself' rather than releasing theirs is a cop out. So when somebody does some work and it does not agree with theirs they can just say its your fault for not using the "right" list... A fine way to delay the day someone might find a problem with your work.

 

[John J.]

Just idly musing, and if it's a factor at all it might have only a very small impact on any correlation:
Was the Samuel Oschin telescope used for the NGS-POSS on Sundays?
Were nuclear tests as likely to be conducted on a Sunday as any other day of the week?
If "no" and "no", that could create a small bias to finding transients within a day of a nuclear test
(e.g. as a reductio ad absurdum (I think!) if all NGS-POSS images were taken on a Tuesday, and all nuclear tests were on a Tuesday, even a 100% correlation would be irrelevant).

Admittedly I could be reaching here

 

[orianda]

Did they? There is an entire thread were people are trying to figure out exactly what Villarroel was looking at and how they were deciding what was an actual unexplained transient/possible techno-signature....

I hadn't read that other thread before, but after skimming through, I'm a bit unclear what the objective actually is - are you trying to replicate the original VASCO study or the recent Villarroel paper from July 2025 that added the nuclear-test correlation?

The thread seems to reference both (for instance, mentions of the 635 plates from the Scientific Reports paper versus the 5399 candidates from the original VASCO catalog ) so I'm not sure which analysis is being reverse engineered.

If the goal is just to reproduce the first study, the underlying POSS-I plate data are public and the workflow is mostly described, so it should be reproducible. The later "transients correlated with nuclear tests" paper is different - that one uses a derived dataset that hasn't been released yet.

 

[Carlo]

If the goal is just to reproduce the first study, the underlying POSS-I plate data are public and the workflow is mostly described, so it should be reproducible. The later "transients correlated with nuclear tests" paper is different - that one uses a derived dataset that hasn't been released yet.

So when you said this:

That's why Villarroel et al. shared their methods and dataset: so others could test whether those residuals really are instrument artifacts.

which methods and dataset were you referring to?

 

[orianda]

So when you said this:

which methods and dataset were you referring to?

I was referring to the original VASCO study, which used the public POSS-I plate data and described its candidate-selection workflow.

 

[Carlo]

Ah that clears up some confusion then, because I was looking for the data and methods for this PASP study and couldn't find which plates they used. I wasn't sure how to go about trying to reproduce the results when they don't provide the data. They also didn't provide any of the nuclear test dates that they used to measure how strong that correlation is either. I keep running into the argument that the way to refute this science is by analyzing the data and interpreting it and not just asserting claims, but how is that possible when the data is hidden?

If you were going to design an experiment that could potentially falsify the results of the PASP study how would you design it?

 

[beku-mant]

I guess I'm un-banned. It only took me a day to acquire all of the software, data, and knowledge, and I'm not even an astronomer. They list the transient positions on their website. You just need that list, the eso batch tool, EarthShadow software, sextractor, and then some competency in astronomy workflows and data science. The filtering of that data (for both papers) is described and not very complicated. Take the full set of transient candidates then filter out candidates with matches within 5″ in Gaia, Pan-STARRS or NeoWise, and those in the southern hemisphere. That's pretty much it. Sure it is hard if you are not an astronomer, or don't have coding/scripting experience. The guidelines for reproducibility in peer review is that a graduate student in the field can do it with a reasonable amount of effort.

Both papers say the final dataset is available upon reasonable request. Reasonable request usually means for academic research purposes so lay people out to discredit them on an internet forum wouldn't qualify. But the good news is everything you need is already available to you. Don't blame the authors for your own lack of reading comprehension and competency in astronomy or data science.

If you wanted to replicate the results, you might want to start from just the plates anyways.

There is no data or methods hiding, or anything unusual, about the support for reproducibility going on. It's unfortunate they are being accused of this.

 

[jdog]

Just idly musing, and if it's a factor at all it might have only a very small impact on any correlation:
Was the Samuel Oschin telescope used for the NGS-POSS on Sundays?
Were nuclear tests as likely to be conducted on a Sunday as any other day of the week?
If "no" and "no", that could create a small bias to finding transients within a day of a nuclear test
(e.g. as a reductio ad absurdum (I think!) if all NGS-POSS images were taken on a Tuesday, and all nuclear tests were on a Tuesday, even a 100% correlation would be irrelevant).

Admittedly I could be reaching here

Well, I did run some numbers on that quite a ways upthread -- I didn't get around to the British and Soviet test dates (but remember those days of week are shifted half a day from the POSS dates):

 

[orianda]

If you were going to design an experiment that could potentially falsify the results of the PASP study how would you design it?

So technically, I'd design an experiment to test a claim .... but regardless, here is an approach. For full disclosure, I asked Claude to organize my thoughts, and improve upon them:

Proposed Replication Design
Scope: Test the shadow deficit claim in the 2025 PASP paper only (not the earlier 5,399 catalog paper or the nuclear correlation preprint — those are separate studies)
Hypothesis: The reported 21.9σ shadow deficit will either replicate or fail to replicate when using the same source data and accounting for observational bias
Data acquisition:
1. Download the 5,399 vanishing source catalog
2. Cross-reference the RA/Dec coordinates with POSS-I plate metadata to identify which of the 635 plates contain these sources
3. Verify which scan source to use — the 2022 paper states ESO, but recent papers mention STScI. This matters because headers differ (including observation dates off by 1 day, as noted in post #146)
4. Download the identified plate FITS files from the confirmed source
Analysis pipeline:
Stage 1: Reproduce the candidate filtering
* Start with the 5,399 catalog
* Apply the morphological filters described in the paper (FWHM thresholds, elongation limits, etc.)
* Goal: reproduce the ~100 "high-confidence" candidate list that the shadow analysis was performed on
* If this fails, the rest is moot
Stage 2: Calculate shadow coverage
* For the filtered candidates: determine what fraction fall within Earth's shadow at GEO (42,164 km geocentric) at their observation times
* For the actual 635 plate footprints at their actual observation times: use Monte Carlo sampling (as they did with 180 points per plate) to calculate expected shadow coverage
* Critical: account for the 50-minute exposure window during which the shadow moves ~2 arcminutes
Stage 3: Statistical comparison
* Compare observed vs. expected shadow coverage
* Calculate significance using the same Poisson uncertainty framework they used
* Does it reproduce the reported 21.9σ deficit?
Success criteria:
* Replication: We observe a statistically significant deficit consistent with their reported 21.9σ
* Non-replication: Either (a) no significant deficit, or (b) we can't reproduce the candidate list, suggesting undocumented filtering criteria

What about the "missing" data?
You mentioned data being hidden, but actually:
* The 5,399 source catalog is public
* The POSS-I plates are public (multiple archives)
* Nuclear test dates are public (multiple databases)
* The exact derived candidate list (~100 sources) isn't published as a table
* The exact filtering parameters may require inference from descriptions
* Custom scripts (like Aladin tessellation) aren't released

So it's more "under-documented" than "hidden" — you'd need to reconstruct some steps from methodological descriptions rather than running their exact code. That's legitimate criticism for reproducibility, but it doesn't make falsification impossible.

Alternative/complementary approaches:
If the filtering proves too ambiguous to reproduce Stage 1, you could:
* Test the shadow hypothesis on the full 5,399 catalog (less clean but more transparent)
* Request the derived candidate list from the authors directly
* Focus on the subset they did publish coordinates for (the 9 examples in their Table 1)
The key is being explicit about what you're testing and documenting where your implementation necessarily diverges from theirs.

 

[beku-mant]

What about the "missing" data?
You mentioned data being hidden, but actually:
* The 5,399 source catalog is public

The entire list of transient candidates is public, but you have to filter those candidates as described in the paper: exclude those with matches within 5″ in Gaia, Pan-STARRS or NeoWise, and then by hemisphere (or reasonable request to get the already filtered data).

 

[Mendel]

I was referring to the original VASCO study, which used the public POSS-I plate data and described its candidate-selection workflow.

Solano(2022) did publish the full 5399 transient coordinates.

But that's not actually one of the two papers we're discussing here.

The shadow effect paper on IOPscience says, referencing table 1, "All 83 candidates are presented in the Appendix." I haven't found that appendix yet. It only presents 5 candidate "motion lines", plus the other transients that happen to be on the pictures they show (table 3).

 

[beku-mant]

Solano(2022) did publish the full 5399 transient coordinates.

But that's not actually one of the two papers we're discussing here.

The shadow effect paper on IOPscience says, referencing table 1, "All 83 candidates are presented in the Appendix." I haven't found that appendix yet. It only presents 5 candidate "motion lines", plus the other transients that happen to be on the pictures they show (table 3).

No, they published all of the transient candidates and their coordinates. You have to filter those candidates as described in the paper to get to the ~100,000.

 

[Mendel]

The entire list of transient candidates is public, but you have to filter those candidates as described in the paper: exclude those with matches within 5″ in Gaia, Pan-STARRS or NeoWise, and then by hemisphere (or reasonable request to get the already filtered data).

The 5399 data set is already filtered by these criteria.
Villarroel uses a different data set of 106,339 candidates for the IOPscience/PASP paper that is not public. You have to reproduce several steps of Solano (2022), but not all of them, to replicate it.

 

[Mendel]

No, they published all of the transient candidates and their coordinates.

Where?

That ~100,000 is the same subset for both papers.

Which two papers?

 

[Carlo]

Both papers say the final dataset is available upon reasonable request. Reasonable request usually means for academic research purposes so lay people out to discredit them on an internet forum wouldn't qualify.

Why would they need to protect the data from laypeople out to discredit them? If the data can withstand scrutiny why all the hoops? I have a lot of experience reading scientific papers and this one has quite the convoluted data section. Is this what scientific transparency looks like to you?

 

[beku-mant]

The 5399 data set is already filtered by these criteria.
Villarroel uses a different data set of 106,339 candidates for the IOPscience/PASP paper that is not public. You have to reproduce several steps of Solano (2022), but not all of them, to replicate it.

The 5399 is filtered by a different criteria. I believe the 5399 has a very strict criteria, while the ~100,000 is a new less strict criteria.

 

[beku-mant]

Why would they need to protect the data from laypeople out to discredit them? If the data can withstand scrutiny why all the hoops? I have a lot of experience reading scientific papers and this one has quite the convoluted data section. Is this what scientific transparency looks like to you?

I didn't say they are. But it is just ordinary practice to make data available upon reasonable request.

 

[beku-mant]

Where?

Which two papers?

Ok, maybe not exactly. Not sure why the first includes filtering with NeoWise critera, but not the second paper, and the second doesn't mention hemispheres, and the numbers are a little different. But regardless, you can just follow the descriptions of how they filter in each paper. I put more stock in the VASCO paper being more precise.

We use the transient candidates from Solano et al. (2022), but with the additional requirement that they have no counterparts within 5″ in Gaia, Pan-STARRS and NeoWise. Furthermore, we restrict our analysis to objects in the northern hemisphere (decl. > 0°). This yields a sample of 106,339 transients, which we use for our study.

https://iopscience.iop.org/article/10.1088/1538-3873/ae0afe

The initial transient dataset consisted of a list of 107,875 transients identified that occurred between 11/19/49 and 4/28/57. These transients were identified in publicly-available scanned images from the POSS-I survey available on the DSS Plate Finder website (https://archive.stsci.edu/cgi-bin/dss_plate_finder). The process used to identify transients and eliminate misidentifications was conducted via an automated workflow detailed fully in Solano et al.1​. In brief, transients were defined as distinct star-like point sources present in POSS-I E Red images that were absent both in images taken immediately prior to the POSS-I Red image and in all subsequent images. A final criterion for classifying an object as a transient was that there were no counterparts either in PanStarrs DR1 or Gaia DR3 at less than 5 arcsec.

https://www.nature.com/articles/s41598-025-21620-3

The data underlying this article are available in the SVO archive of
vanishing objects in POSS I red images available at http://svocats.ca
b.inta-csic.es/vanish/

https://academic.oup.com/mnras/article/515/1/1380/6607509

 

[orianda]

Ok, maybe not exactly. Not sure why the first includes filtering with NeoWise critera, but not the second paper, and the second doesn't mention hemisphers, and the numbers are a little different. But regardless, you can just follow the descriptions of how they filter in each paper. I put more stock in the VASCO paper being more precise.



https://iopscience.iop.org/article/10.1088/1538-3873/ae0afe



https://www.nature.com/articles/s41598-025-21620-3



https://academic.oup.com/mnras/article/515/1/1380/6607509

---

OK - I know I'm not the only one confused by all the paper references now. Can we just agree to call them A/B/C?


Paper A (PASP/Shadow): Villarroel et al. 2025
Uses 106,339 transients
https://iopscience.iop.org/article/10.1088/1538-3873/ae0afe


Paper B (Sci Reports/Nuclear): Villarroel et al. 2025
Uses 107,875 transients
https://www.nature.com/articles/s41598-025-21620-3


Paper C (Source catalog): Solano et al. 2022, MNRAS
The 5,399 "highly curated" catalog both A & B reference
https://academic.oup.com/mnras/article/515/1/1380/6607509

Both A and B claim to derive their datasets from C using the same filtering criteria (no counterparts in Gaia/Pan-STARRS/NeoWise within 5″), but somehow end up with different sample sizes (106,339 vs. 107,875). Am I missing something obvious here?

 

[Carlo]

Ok, maybe not exactly. Not sure why the first includes filtering with NeoWise critera, but not the second paper, and the second doesn't mention hemisphers, and the numbers are a little different. But regardless, you can just follow the descriptions of how they filter in each paper. I put more stock in the VASCO paper being more precise.

You're comfortable hand-waving away the differences now? What happened to all the condescension and surety from a few posts ago? I thought it was extremely easy to parse which plates they used but now that I'm reading these new posts it looks like you're also having a tough time identifying exactly which plates were used in the PASP study. I guess you just have to work on your reading comprehension a bit more.

 

[Mendel]

I have a feeling @orianda has blocked me, but here goes:

Proposed Replication Design
Scope: Test the shadow deficit claim in the 2025 PASP paper only (not the earlier 5,399 catalog paper or the nuclear correlation preprint — those are separate studies)
Hypothesis: The reported 21.9σ shadow deficit will either replicate or fail to replicate when using the same source data and accounting for observational bias
Data acquisition:
1. Download the 5,399 vanishing source catalog

the shadow deficit is determined using a list of 106,339 sources
If you compute the shadow coverage ratio in table 4 with n=5399, you'll see that the valued don't match by a factor of 20.

Analysis pipeline:
Stage 1: Reproduce the candidate filtering
* Start with the 5,399 catalog
* Apply the morphological filters described in the paper (FWHM thresholds, elongation limits, etc.)

the 5399 catalog is already filtered

* Goal: reproduce the ~100 "high-confidence" candidate list that the shadow analysis was performed on

83 candidate "motion lines", but that's not what the shadow analysis was performed on, see table 4 showing 349 objects in shadow

* For the actual 635 plate footprints at their actual observation times: use Monte Carlo sampling (as they did with 180 points per plate) to calculate expected shadow coverage

false positives are not equally distributed at 180 points per plate, they're astronomical objects that e.g. cluster on the Milky Way, and that should be used to set the expectation. It is true that Villarroel does not do that.

* Critical: account for the 50-minute exposure window during which the shadow moves ~2 arcminutes

how exactly? determined by the proportion of time that the coordinate was in shadow? or, like the paper seems to do, run 2 analysis, one with minimal and one with maximal area? (How was the minimal area determined?)

Stage 3: Statistical comparison
* Compare observed vs. expected shadow coverage

see above for what the expectation should be

* * Non-replication: Either (a) no significant deficit, or (b) we can't reproduce the candidate list, suggesting undocumented filtering criteria

the method you described will not reproduce the candidate list

What about the "missing" data?
You mentioned data being hidden, but actually:
* The 5,399 source catalog is public

irrelevant

* The POSS-I plates are public (multiple archives)
* Nuclear test dates are public (multiple databases)

are local dates used, or are test dates normalized to UTC? is this known for the Soviet tests? we tried to compile the test dates, but could not get thectotal to match, as there is ambiguity which low-yield or failed tests to include. The list of test dates used in the paper should have simply been published in an appendix.

* The exact derived candidate list (~100 sources) isn't published as a table
* The exact filtering parameters may require inference from descriptions
* Custom scripts (like Aladin tessellation) aren't released

So it's more "under-documented" than "hidden" — you'd need to reconstruct some steps from methodological descriptions rather than running their exact code. That's legitimate criticism for reproducibility, but it doesn't make falsification impossible.

Solano (2022) has already invalidated the data set used in the 2025 papers.

Alternative/complementary approaches:
If the filtering proves too ambiguous to reproduce Stage 1, you could:
* Test the shadow hypothesis on the full 5,399 catalog (less clean but more transparent)

good idea, will give different result, and lower significance due to drastically reduced n

* Request the derived candidate list from the authors directly

please try it, and report on how that went. There is zero reason for that data to not be published with the study.

 

[beku-mant]

good idea, will give different result, and lower significance due to drastically reduced n

I did this, and got a small deficit. Since at n=5399 the expected number of transients in shadow is only ~30 or so, you cannot get a powerful enough statistical result. Since it is expected that at least some portion of the transient candidates are not reflective objects, there is some expected amount of noise, and at n=5399 that noise will be too dominant.

 

[Mendel]

The 5399 is filtered by a different criteria. I believe the 5399 has a very strict criteria, while the ~100,000 is a new less strict criteria.

Villarroel:

Article:

We use the transient candidates from Solano et al. (2022), but with the additional requirement that they have no counterparts within 5″ in Gaia, Pan-STARRS and NeoWise. Furthermore, we restrict our analysis to objects in the northern hemisphere (decl. > 0°). This yields a sample of 106,339 transients, which we use for our study.

Source: https://iopscience.iop.org/article/10.1088/1538-3873/ae0afe#paspae0afes8

That describes the "candidate selection" steps of Solano (2022), plus filtering by a single IR sources catalogue (partial analysis step i).

 

[orianda]

I have a feeling @orianda has blocked me, but here goes:

I haven't blocked anyone, just been reading through the papers more carefully. You're right that I mixed up the sample sizes - will need to look at this more closely and correct my replication design.