Identity and Licenses

We have set up Salesforce as an identity provider. Does this work for users with all kinds of licenses, or are there any restrictions?

Especially, can I use some of my available Chatter Free licenses to manage users which do not need any access to Salesforce objects (but should be able to be served by SF IDP) or do I need to purchase External Identity user licenses for this purpose?