Study: Assessing correspondence between subjective reports and visual targets during attempted out-of-body experiences

[yoshy]

Assessing correspondence between subjective reports and visual targets during attempted out-of-body experiences

Saw this being passed around. A peer reviewed study, with a solid methodology for the most part, on out-of-body experiences tested if the individuals could OBE themselves and identify a target in another room. Drum roll........ They couldn't. But that's not the part worth discussing in my opinion... see the bolded portion below.

Claims of veridical perception often accompany out-of-body experiences (OBEs), but systematic evaluation under controlled conditions remains difficult. We examined whether descriptions produced after attempted OBEs corresponded to concealed visual targets beyond chance using computational semantic similarity. Twenty-one individuals affiliated with an institute specializing in OBE training attempted to perceive a randomly selected target displayed in a separate, inaccessible room. Targets were drawn from a standardized database and remained unknown to participants and researchers. Ten participants reported an OBE, and 13 provided target descriptions for analysis. Participant descriptions and target captions were converted into high-dimensional vector embeddings using a pretrained language model. For each trial, cosine similarity between the participant's description and each of 100 candidate targets was computed; the correct target was evaluated against the within-trial decoy distribution using a rank-based metric (our primary analysis) and standardized z-scores (a secondary, descriptive analysis). Control analyses confirmed that the pipeline identified correct targets when semantic correspondence was present, though these controls do not establish sensitivity to naturalistic human language. No individual trial showed statistically significant above-chance correspondence, and a combined rank-based test across all 13 trials likewise showed no evidence of above-chance performance. Two participants spontaneously reported accurate details about experimenters' locations and activities in an adjacent room. Although the target-identification results did not support anomalous cognition, the spontaneous environmental reports may have methodological value and warrant systematic investigation.

Including this in the abstract is very inappropriate, and it shouldn't have passed peer review. It's a glaring misuse of an abstract. An abstract should summarize the main findings of your experiment, i.e. the thing you actually planned and studied. Putting in anecdotes is not the proper use of an abstract. If they wanted to include this tidbit, I suppose they could put it in their discussion section or supplementary material, but absolutely not the abstract. It gives the appearance of credibility by including it in an abstract about an actual experiment with valid methodology. Another issue is the complete lack of context about these interviews. Abstracts should briefly include how they gathered the data presented in the abstract. This two sentence description doesn't describe essential information on the methodology for these interviews. (I haven't been able to find the full study document yet, so hopefully there are some answers in there.) This article from UVA provides some information, but not a lot.

• Were they planned, formal interviews?
• Did everybody have an interview?
• Were questions predetermined and uniform across test subjects?
• What do they mean "spontaneously"? Did they randomly shout "AND THERE TWO RESEARCHERS SITTING NEXT TO EACH OTHER AND ONE OF THEM LOOKED AT MY SOUL" in the middle of questions about the actual exam? If they were being interviewed, it seems likely they asked a question that would lead to them discussing seeing the researchers.
• Are there transcripts?
• Those who are familiar with the Ariel school UFO case know how easily interviewees can be influenced to give certain answers. Interviewers can do this completely unintentionally and even subconsciously. Transcripts are essential for determining if there were leading questions.

The UVA article hints at there being some sort of formal interview with the quote "But during interviews afterward", which I read as implying everyone was interviewed, but that's just an assumption.

Assessing the "accuracy" of the descriptions can be extremely subjective. On remote viewing threads, I have seen extremely vague drawings of shapes being called a "hit" for a remote viewer. Again, we need more info here.

My biggest criticism of including this in the abstract is what I'm calling the "anecdotal version of p-hacking". I'll explain what I mean and how it applies. One way p-hacking occurs: A researcher collects data with a specific test in mind. They run the test and get non-significant results. Bummer. At that point, researchers need to realize, but often don't, that any further statistical tests are invalid unless multiple hypothesis testing corrections are applied. Unfortunately, researchers will often keep running new tests until they inevitably find a significant value. Then they publish that test, maybe the first one too if we're lucky, but then make no mention of all of the other tests with non-significant results. I'll stress again that researchers who do this generally have no idea they're breaking the stats law so I am not making accusations of any deliberate fraud or deception. These researchers are doing the anecdotal version of p-hacking because they are reporting results of something that wasn't planned to be part of the study. Their actually study failed to produce the results they hoped for. Their abstract would be seen by many and quickly forgotten. But attaching this story to their abstract instantly makes it notable! (Judging by seeing it spread throughout forums and social media.) When there are literally no constraints on where and what to look for, one can easily find something coincidental and then attach greater meaning to it. Again, we can see examples of this regarding claims of "synchronicity". Since the interview was not planned to be part of the actual experiment, none of the usual safeguards against mistakes are present. All the types of bias (selection, confirmation, etc), lack of objective criteria, and so on are not controlled for here.

The way they present this feels like a "you can have your cake and eat it too" moment to me. In the UVA article, they accurately state:

The participants' unexpected observations were not part of the trial's predefined outcome measures and therefore can't be considered evidence for out-of-body experiences, or OBEs, UVA Health investigators say.

They say that it can't be admitted as evidence, yet they put in the abstract! The most read part of any scientific paper! Peer review should have had that removed from the abstract. Like I mentioned above, feel free to write something in the paper about it, since they plan to use their experience when formulating methodologies and experiments in the future. But putting it in the abstract makes it seem like credible scientific data. They also reference it in their interviews

"I hope this study encourages researchers to think creatively about how we test these experiences. The unexpected findings give us a potentially useful new direction, but they also need to be tested prospectively and under rigorous conditions before we can know what they mean," said researcher Marina Weiler, PhD, of the University of Virginia School of Medicine's Division of Perceptual Studies. "For me, the most important takeaway is that unexpected findings can sometimes help us ask better questions. We now have a specific idea that can be tested in future experiments, and that is exactly what science should do."

They have successfully switched the focus from the actual experiment that showed no evidence for the things they believe in to the unscientific interviews, which they talk about as if they are valuable, paradigm shifting data.

"I hope these findings encourage us to think more deeply about what they might mean for our understanding of consciousness and, ultimately, the nature of reality. If studies demonstrate that people can obtain accurate information that they could not have accessed through their ordinary senses, we would have to reconsider some of our assumptions about the relationship between consciousness, perception and the physical world," Weiler said. "At its deepest level, this research is not only asking whether out-of-body experiences are real. It is asking what we mean by 'real' in the first place, and whether our current understanding of reality is broad enough to account for everything human consciousness can experience."

Their actual findings were null results! Grrrrrrrrr

Note again that I haven't found the full paper yet, so some of my questions may already be answered therein. If I do, I will correct whatever possible.

 

[FatPhil]

13 trials

WIth that N, any kind of highly significant result, rather than being more convincing, would scream to me "terribly designed experiment that permits cheating (by tester or testee, it matters not)" - such powers would be oft noticed and well documented if they actually existed. What kind of p values are they claiming?

And I agree, those final sentences have no business being in the abstract. Including them looks like fishing.

(...for science, or for funding, or does it even matter, either's bad)