The more it happens, the more annoying it gets, and the true regardless of your personality type. He's interested, and was fairly calm the first few times you saw him, but now he's feeling the pressure and it's made him too nervous to take it any further. This is important, because the way you get a shy guy is the same way you look desperate to a normal guy.
Just give this shy guy the opportunity to get to know you a little so he can open up and let you in. A shy man will pay attention to the smallest details just because. Has no problem talking to you — just seems bored or disinterested when he does.