I’m working with Salesforce Data Cloud and an unstructured web crawler, but some page sections appear to be JavaScript-rendered and are not being reliably captured by the crawler.
Has anyone worked with this type of scenario? I’d like to know whether this is a known limitation, and if there is any recommended workaround or alternative approach.
@Liliana Camargo that crawler reads the HTML the server returns. It does not run the page JavaScript, so a section that only exists after the browser executes a script is not in the fetch. That is a limitation of the connector, not a setting you missed.
The reliable fix is to put that content in the first HTML response. Server-side render it, or give the crawler a prerendered URL. If the site already has an API, a CMS export, or a PDF, ingest that instead of the rendered page.
I would not keep tuning the crawler and hoping it executes the script. It will not.
