Cropping round edges

I want to crop an image and make the edges round instead of square. Do I do this with a mask? Can't figure out how to use a mask. I went to Mask Outline->Smooth and it cut the image out and left the round edges.