Process Builder should not update a field when it is updated manually.

Please someone help me with this requirtement. Thanks in advance.

I need to track when is the last date I contacted a Client. Say I have five Date fields as 1) Client 1st follow up 2) Client 2nd Follow up 3) Cleint 3rd Follow up 4) Client 4th Follow up 5) Last Date of Client Follow up. So, I wrote a workflow to update 'Last Date of Client Follow up' field with always the latest followup date of other four fields. Say I wanted to follow up 5th time, now I want the 'Last Date of Client Follow up' to be edited manually without firing the workflow. The problem is when I enter a date in the field manually and save it, the workflow fires and changes the date to the latest of the four dates. Is there a workaround for this problem?

Thanks..!!