Only 10 checkboxes should allow in grid view if we select 11th one it should not select and should through an error

Looks like this is a custom app - You can use JS to restrict selection

Here is an example: https://stackoverflow.com/a/43456964