How do I ensure only certain users can change the owner of a case rather than everyone?

Here's a VR with exemptions for Previous Case Owner, and multiple Users, Roles, and Profiles that you can borrow and customize

AND(
ISCHANGED(OwnerId),
PRIORVALUE(OwnerId) <> $User.Id, 
CASE($User.Username, 
"SteveMo@MyForce.org", 1, 
"Moe.Howard@3Stooges.org", 1, 
"Larry.Fine@3Stooges.org", 1, 
"Curly.Howard@3Stooges.org", 1, 
0) <> 1, 
CASE($UserRole.Name, 
"Exempt Role A", 1, 
"Exempt Role B", 1, 
"Exempt Role C", 1, 
0) <> 1, 
CASE($Profile.Name, 
"Exempt Profile X", 1, 
"Exempt Profile Y", 1, 
"Exempt Profile Z", 1, 
0) <> 1 
)