PhD student in probability theory here, let me explain what is wrong with OPs argument.

The problem with all of OPs arguments is that he is simply

https://en.wikipedia.org/wiki/Cherry_picking evidence. Let me explain.

Let me make OPs argument mathematically rigorous: Let "A" denote the hypothesis that aliens visit earth. Let x_1,\dots, x_n denote n people that have given testimonies of Alien encounters and let CE_1, ..., CE_n denote the events that "x_1 has given testimony of Aliens", "x_2 has given testimony of Aliens", ..., and so on up to CE_n.

OP assumes that the CE_1,...,CE_n are independent conditional on A, and that P(CE_k | A)=1 as well as P(CE_k | not A) = p (in the original argument, OP uses p=0.01). Even though all three assumptions are problematic, as discussed in the thread, we will admit them in order to highlight a much more important flaw in OP's argument.

Under these assumptions, OP correctly calculates that, since the x_k's were chosen so that CE_k happened, P(A | CE_1, ..., CE_n) = P(CE_1, ..., CE_n | A) / P(CE_1, ..., CE_n) * P(A) = P(A) * 1 / (P(A) + p^n P(not A)). If P(A) is chosen very small, i.e. P(A)<<1 (our prior belief in Alien visitations is very small), then this is approximately P(A) / p^n.

OP correctly states that for n large and p small, this gives a Bayes factor of about p^{-n}, which is huge. The example that OP gives is n=10 and p=0.01, so p^{-n}=10^20 is very big.

If the x_1, ..., x_n had been fixed BEFORE testimonials were made (i.e. if you had to predict which people will make testimonials), this would be a valid scientific argument and hypotheses of Alien encounters would enter scientific debate.

However, the x_1, ..., x_n were chosen AFTER the testimonials were made. In other words, by Definition, the C_1, ..., C_n are events of testimonials that we know already happened (if different people had given testimonials, then the C_1, ..., C_n would have been chosen differently). This is the Definition of cherry-picking evidence.

This is why scientific theories need to make correct PREdictions and not POSTdictions.

How large is the pool of evidence from which one can cherry pick? This is impossible to determine in general, because the mechanism by which you choose your testimonials is unknown. Let's be modest and say that out of the 8 billion people on earth, 100 million have a good enough access to internet/media in order to popularize Alien testimonials, about 1 million will actually reach you (the other ones may be in languages you don't speak etc. so you will not have access to them) and of these 1 million people about 1'000 actually would choose to make Alien testimonials, conditional on the event that there are no Aliens (incentives for these 0.1% of people include delusion, monetary gain, attention gain, ideological reasons, etc.). The actual numbers don't matter much and are chosen pretty arbitrarily to explain how you cherry-picked data. As you did, we assume that the testimonials are independent conditional on not A.

Assuming that you cherry-picked evidence from 1 million possible testimonials, as motivated by the above calculation, the actually correct event to condition on, in this case, is the event "out of 1 million people, at least 10 chose to make Alien testimonials". Let's call the number of people who make such testimonials T. Under the assumption "not A" with the above numbers, the distribution of T is Binomial(1 million, 0.001). So P(T>= 10 | not A) is a bit more than 1 - 10^{-412}. Also, P(T>=10 | A) is 1 by assumption. The Bayes factor is thus about 1/(1-10^{-412}), i.e. about 1+10^{-412}. This is 1.0000(408 more zeros)1. In other words, negligibly small. This is not surprising: Assume that you observe 1 million people, out of which you expect 1000 to make Alien testimonials even if there are no Aliens. Then you will be completely unimpressed by 10 people giving Alien testimonials. After all, you would expect 1000 of them even if there are no Aliens!!

However, it is even worse: It is not clear how many testimonials you would expect if there were aliens. You assumed that P(C_k | A) = 1. If this assumption is also assumed for all possible testimonials, instead of just the ones that we know happened, then one would expect 1 million testimonials to take place, if A is true. If you then observe only 10, the posterior probability of A given this observation is exactly 0, since the assumption P(testimonial | A)=1 does not allow anyone to not make a testimonial.

Your problems could be ameliorated if you could demonstrate that the mechanism by which you choose the testimonials is not based on cherry-picking. For instance, if you were to choose people at random à priori and see if they turn out to make alien testimonials (this is the whole idea of random polling); [if you choose people (at random) à priori, the probability of just by chance getting all the people that give alien testimonials conditional on there not being any Aliens is exponentially small in the number of people that made testimonials that were in excess of the expectation of the number of such testimonials conditional on there not being any Aliens on earth, by

https://en.wikipedia.org/wiki/Large_deviations_theory]. However, the mechanism you use is, as mentioned above, based on choosing testimonials à posteriori. Therefore, this mechanism fails to generate Bayes factors in favor of A in excess of tiny factors such as 1+10^{-400}.

Let A = Hypothesis that aliens are visiting Earth

Let Not[A] = Hypothesis that aliens are not visiting Earth

Let { CE_i } = A set of i independent Close Encounter events

Using

Bayes' theorem we find that the (posterior) probability that aliens are visiting Earth given a set of i independent close encounters, P( A | { CE_i } ), is given by

P( A | { CE_i } ) = P( { CE_i } | A ) P( A ) / [ P( { CE_i } | A ) P( A ) + P( { CE_i } | Not[A] ) P( Not[A] ) ]

Now I argue that the probability of a set of i independent close encounters given the hypothesis that aliens are visiting Earth, P( { CE_i } | A ), can be taken to be one by definition. A believer in the alien hypothesis will not be surprised to learn of a set of close encounters with aliens.

I wish to argue backwards to find an expression for the prior probability for aliens P( A ) that gives us an appreciable posterior probability for aliens given reports of i close encounters, P( A | { CE_i } ) = 1/2.

Substituting P( { CE_i } | A ) = 1 and P( A | { CE_i } ) = 1/2 into Bayes' theorem we get

P( A ) / P( Not[A] ) = P( { CE_i } | Not[A] )

P( A ) / P( Not[A] ) = P( CE_1 | Not[A] ) * P( CE_2 | Not[A] ) * P( CE_3 | Not[A] ) * ...

Let us assume that aliens are not visiting Earth and that the witness was awake during the encounter (no bed-time accounts) and that the encounter was faithfully recorded soon after the event. I would say that the witness was either lying, hallucinating or the victim of a hoax.

For the sake of argument let us assume that we have a compelling case so that P( CE | Not[A] ) = 1 / 100

Let us assume that we have 10 good cases of close encounters. In order to end up with an appreciable posterior probability of aliens given those cases, i.e. P( A | { CE_i } ) = 1/2, the prior ratio of our belief to unbelief in aliens, P( A ) / P( Not[A] ) is given by

P( A ) / P( Not[A] ) = (1/100)*(1/100)*(1/100)*(1/100)*(1/100)*(1/100)*(1/100)*(1/100)*(1/100)*(1/100) = 10^-20

Thus if we combine data from compelling close encounter cases even the most hardened skeptic should begin to believe in the alien hypothesis.