It’s all about body language. When a cat enters a room all the cat lovers start staring at it, they often move towards it, extend their hands and make noises.
The non-cat person, on the other hand, may try to make herself (or himself) as invisible as possible so the cat does not jump on them. This person is likely to sit extremely still, look down and away from the cat, keeping their hands on their lap and maintaining absolute silence. The cat finds this latter display of body language far less threatening so may well move towards the ‘non-cat’ person to explore further!