Orchestration fails when checking ISBLANK or ISNULL of lookupfield if there is no value

I must be missing something simple. Hopefully, someone can point me in the right direction.

I have a record-triggered orchestration. The triggering record has multiple custom user lookup fields ( for internal assignments). I have a decision element that has the resource set to {!$Record.My_Custom_User_Lookup__c}, Operator IS NULL, and value False. Basically, to say if the lookup field has been set, go down this path of the orchestration. 

The problem is I'm getting the error "The flow failed to access the value for $Record.My_Custom_User_Lookup__c because it hasn't been set or assigned". How can I check for a Null value if the flow errors out because there is no value in the field?

EDIT: I've also tried creating a boolean formula of IF(NOT(ISNULL({!$Record.My_Custom_User_Lookup__c})), True, False) and then using the boolean in my decision element, but no luck.