Checked so far:
- Raw source detection recovers ~99% of the third-party list's rows. Detection sensitivity is not the issue. I've identified a more suitable sextractor parameter set using only one-pass call, without PSFex step. However, this doesn't matter as much as the "naive" tessellation method where plates are downloaded fully, including the edge areas that were excluded earlier
- Most of the gap between "strict pipeline" and "third-party list scale" comes from two places: how much sky area gets covered, and the Gaia/PanSTARRS/USNO-B cross-match veto stage. The veto stage is working correctly
- Testing suggests the strict pipeline, once matched for the same sky coverage, lands roughly in the right ballpark of scale. Early result, not fully validated yet
Unexpected finding (also early result, not fully validated yet):
- The third-party list's plate identifiers appear to have real errors on some plates. A meaningful fraction of rows are tagged with the wrong originating plate. Observation dates could be therefore partly wrong, too
Current status:
- Moved from single-plate pilots to processing plates one at a time across the sky, with checks at every stage
- I'll share more once there's a broader, more solid full-sky picture.
Quick update: I have now reached 84% recall rate compared to VASCO. I still have about 17K tiles to download and process, so the recall rate should hit at least 90%. That's a pretty good number.
The unpublished third-party list "V" (107K rows) does not contain coordinate/plate/date errors like the early results indicated. I double-checked everything against two download services: STScI and ESO. All coordinates have correct plate identifier, and the observation date/time.
Naive tessellation plan was not naive enough. It leaves gaps - meaning tiles that should have been downloaded. I added them to the working queue once I realized the gaps. The root-cause was the "smart tessellation" plan of Vasco60. I decided to re-use it, and just try to make it more "naive". Bad decision. I should have rewritten the whole tessellation module from scratch. Because the original code avoids plate edges and tries to avoid overlapping, it's not easy to make it naive by altering the plan only.
I also dropped the 30' circular cut described in MNRAS 2022. If code performs this cut, the pipeline will not reach a good recall rate.
The end result after basic pipeline process is currently 228 628 rows. It will grow after data from 17K additional tiles lands. But this number is not the "final survivor count" - like VASCOs unpublished "V" dataset (107K). Survivor count is subject to number of post-process veto-stages, which will reduce the number dramatically - depending on what kind of veto stages are used. I haven't made final decisions, but many trials. With four heavy veto-stages, the number went down to 11 341 rows.
VASCO's own ML ensemble shows that their V contains only 10-20% valid, star-like objects. I can't implement this kind of ML veto, because it is not public.
I've tested many things to rule out possible problems like tessellation using Aladin app (as mentioned in MNRAS 2022) - this was a disaster. Using ESO or STScI as the source for fits cutouts turned out to be interesting and time-consuming. I was surprised to learn they return different coordinates and slightly different FITS headers. ESO sometimes returns cutouts from different surveys (especially when DEC <=0) without warning. And many ESO tiles contain more garbage than STScI tiles. MNRAS 2022 mentions ESO as source, but later papers also mention STScI. I have no idea where they downloaded the basic data, because Vs coordinates don't match with ESO or STScI.
I also tried using locally mirrored full plate scans, but that's really not feasible.
I tested a large amount of source extractor and psfex configuration values. Even the silly "PHOTO"-mode which will crash the software or just give really bad results. I asked AI to review the full sextractor source code to identify the variables that will cause either more or less detections - this was genuinely interesting, but at the end, my original sextractor/psfex config didn't change that much.
This has been a difficult exercise. VASCO team members often speak about independent replication. Because it is important in science. But because they decided not to publish code, configurations, and not much data either, independent replication becomes a mission impossible. I'm pretty sure that 100% parity cannot be reached with the currently published material. It seems I will reach a good main dataset recall rate, but even that is not going to be enough. Results can be disputed just by saying "you don't have the same configuration or sky coverage" - even without evidence (meaning published sw/configs, and datasets). But I will finish this project and publish the code and results.