6 answers
No, permission sets are used to grant additional access as opposed to remove acess. As a best practice, you want to start your org with the minimum access required, and then extend access to users as needed...not the other way around.
reference: https://www.forcetalks.com/salesforce-topic/is-it-possible-to-restrict-permission-for-users-using-permission-set-in-salesforce/
