Hi All,
I am trying to use the
Web Content (Crawler) connection to crawl over a static web page to read the data. Also I have established the connection by using the Basic Authenticationmethod.
And then I am creating the Data Stream, DLO and DMO. But when I am refreshing the Data Stream, I am not getting the actual data from the page.
The Data I am getting from the page as per below:
1. When I am
not using"Included Page Element", "Included Web Pages" in the Data Stream, then I am getting the some other data that is not present in that page.
2. When I am
using any of them or both("Included Page Element", "Included Web Pages") in the Data Stream, I am not getting any data.
N.B :
a) I have tried by setting the crawl depth to 0,1 and 2.
b) In the connection, I have tried by checking and unchecking the "Ignore robots.txt" options.
The website which I am trying to crawl, contains the data in HTML format. But still I am not getting the exact data in the DLO.
If anybody can provide any possible solutions or reference, that would be helpful.
Thanks in advance!!!
Is this issue happening due to connector's rate limit ?
If yes, is there any other way to relax the rate limit ?
