Make Specific Cases not visible to certain users

I have created a case record type (Privacy Cases) that I only want visible to system admins. I have Service Agents that can still see these cases though. I can see that Service Cloud role case settings have sharing settings overrided but I can't seem to figure out how to edit that? Any help would be appreciated.

Note: I already have the Cases marked as Private in the OWD, and have edited Permission Set for the Case Object to Not Available #Service Cloud