why does a note get shared with the contact in the database outside of the company?

When I add a note to a contact and view the sharing, it is shared with that contact as "viewer" by default. This seems so strange to me as I don't want them to see the note, but my only other option is "set by record" and I don't know what that means.