I am highly involved in the recruitment process in my company and, believe me, it is hard. It is quite hard to determine with a so small interaction when somebody is going to fit in your company, the team, and the particular post.

Also, some times there are too many actors on the company side with different interest and grades of involvement and everybody has an opinion about how the candidates should be. Sometimes it is like an onion, the good layers (i.e. your future team) are hidden below the bad ones. At the end, people struggle to do what they can and some companies get to do it well and others just don't.

Is that representative of what working at that company would be? IMO it says something but I don't think it is enough to rule a company out. At least try to meet your future boss and team!