Don't keep them around. You can walk on gravel, but you don't keep a driveway in your pocket. Only keep the good ones. Decide what qualities you admire in people and look for them in your interactions with others. And, also, maybe don't take the walking on metaphor too seriously. They're just people, and you're just a person too.