Depending on how your community it set up, you can either "follow" someone or add them as a friend ("friend" them). When you choose to friend or follow someone, they're added to your list of connections (although they might need to approve it first).
Friending someone creates a two-way relationship with them -- they become your friend, too.
Following someone is a one-way relationship. They show up in the list of people you're following, but you don't necessarily show up in theirs (unless they choose to follow you).
You can remove someone on your friends or connections page. Go to Your Stuff > Connections or Your Stuff > Friends, then find the person you want to remove from the list. To remove them, click the Remove link on their listing.