field level security restrictions on standard email message fields

I'm working on a security fix in our org and have run into a blocker I'm hoping you can help clarify. 

Issue: I'm trying to restrict Read access on standard email  message fields(Specifically to Address and from address) 

Contest: this is blocking a security fix. Users with this profile can currently click through the " To address" field on an email message to view contact records they shouldn't have access to. 

Happy to share screenshots or more detail if helpful. Appreciate any guidance you can offer