2 answers
There is no technical reason since permission sets are used to give some type of permission to an user. Usually deactivated users can cause technical issues when they are running user for a report for example. You mentioned that your users are not removing the inactive user from the permisison set. How about automating this process?
This blog post explains how you can do it: https://jenwlee.wordpress.com/2017/02/14/automated-cleanup-when-user-is-deactivated-remove-user-from-permission-sets/