If the puppies aren't right for you, I don't think there's any reason to feel bad about it. I felt like that with the first litter of puppies we went to see. I also got a weird feeling from the breeder because she was trying to be very secretive and hide the dogs from the neighbors... very odd.
We also passed up on a blue merle puppy before we got Buffy, despite the fact that the breeder was extremely helpful and nice. If it doesn't feel right, it's not right. I think it's better to wait for the right puppy and be sure about it than rush into something where you're not 100%. Buying a dog is one of the few chances you have to pick one of your relatives. And if you want a cute dog, well, who can blame you? There's plenty of people (like me!) who think the littler puppies are cute. I'm small, so I like that Buffy is too.
I think that it's important to pay attention to the mom when you go to see puppies too. Mom is a very good indicator of temperament, size, conformation, etc. The puppy is an adult for much longer than it is a puppy. Buffy's dam was wandering around while our breeder was explaining her contract and answering our questions, and the mom was such a sweetie. She kept putting her front legs up to be petted. I later learned from this forum that Buffy's sire has a similarly sweet temperament. It makes sense that Buffy is the way she is since she had such good genes from those two.
Also, I pay a lot of attention to the socialization the breeder has done with the puppy. By the time we got Buffy she had been on all kinds of surfaces, was fully transitioned to solid food, was used to being handled, had been around kids, adults, and elderly people, and had all her vaccinations, etc. Some of the key times of socialization pass while at the breeder's, so it was good we had our bases covered there.
Also, at most of the good breeders, even the pet quality puppies are very pretty and cute. I guess it's kind of in the eye of the beholder, but I think Buffy is prettier than many of the show quality dogs I have seen. But of course, that's just me! She can never show because of course, she's too short, and also she has cute little buck teeth. You can only see them if you look closely. I think it adds to her charm.
By the way, Buffy was our first choice when we looked at the puppies, but there were 3 people in front of us to pick their girls first. I was certain one of them would take her, and I was worried that we would end up with one of those other puppies instead. (The whole week before I probably drove my husband crazy.) Anyway, it worked out in the end, so we were very lucky. But sometimes things are just meant to be.