Customarily, posting some code will get better help than a simple question
Your definition of "neighbor" appears (from the sample) to mean "the next-adjacent number -- after each test/remove/keep action." But this is NOT unambiguous, from your sample data.
Consider: (5,6,7,8,9,10, 15, 20...)
A tighter definition may actually reveal the answer you seek, as at least my reading of your sample makes me believe their can be cases where your final requirement cannot be satisfied for certain data.