Question on Files/Content and security.

We are looking at the possibly using SF Files & Content for storing some slightly sensitive documents and would like some direction. Here is a scenario:

We are working on a customer file that will be shared with only certain users. We may want to share (content delivery) with the customer. This is not a big issue with content libraries and deliveries.

The issue we have is that we would like to associate this specific document(s) to the related account/customer record. But, when we do this, it is available for all users to view as we have our Organization-Wide Defaults are set at Public Read/Write.

So, what are our options? Even using the feed to add the doc. to the related Notes and Attachments make it visible to all.

Thanks in advance!