What is the limit of Result binding in CRM Analytics? Also can we use coalesce in post projection filter, is there syntax to serialize as 'not in ' like asEquality serialize as 'in' in SAQL ?

There is no limit to a result binding, but typically the first limit you might run into is the size of a query that can be posted via the query endpoint (assuming the results binding is being inserted into another query). I think that limit is 4K bytes, but I could be mistaken. I might be confusing that with the 6MB Apex request size limit. There's also a max query duration timeout of 2 minutes. SOQL has it's own limits if you're using direct.

You can use coalesce in a projection, and then filter on that projection. I don't think I've ever seen coalesce used in a filter statement, but it might help to see an example.

https://developer.salesforce.com/docs/atlas.en-us.salesforce_app_limits_cheatsheet.meta/salesforce_app_limits_cheatsheet/salesforce_app_limits_platform_soslsoql.htm

https://help.salesforce.com/s/articleView?id=sf.bi_limits.htm&type=5

https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_gov_limits.htm