in reply to Authorization Negotiate Wep-Programming
You have not shown how you generate the value of $viewState, which is pertinent here. However it won't make any difference since you define the value of $param in single quotes, meaning $viewState won't be interpolated anyway.
You also are using the most inconvenient way of building and making your request. Consider using at least HTTP::Request::Common, if not something simpler such as HTTP::Tiny.
Sometimes sites are just so anti-scraping that you can't get there with LWP. This seems to be one of them. I don't think your problem is your proxy. I couldn't retrieve the page via LWP either. I am able to get search results using cURL with the same args: try running this from your command line and see what you get:
curl -L 'https://patentscope.wipo.int/search/en/search.jsf' -H 'Host: +patentscope.wipo.int' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel M +ac OS X 10.11; rv:53.0) Gecko/20100101 Firefox/53.0' -H 'Accept: text +/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Acce +pt-Language: en-US,en;q=0.5' --compressed -H 'Referer: https://patent +scope.wipo.int/search/en/search.jsf' -H 'Content-Type: application/x- +www-form-urlencoded' -H 'Cookie: JSESSIONID=CD31D066DF8710F9FE9B5C9C5 +397A977.wapp1nC' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Req +uests: 1' --data 'simpleSearchSearchForm=simpleSearchSearchForm&simpl +eSearchSearchForm%3Aj_idt379=FP&simpleSearchSearchForm%3AfpSearch=bru +shless+motor&simpleSearchSearchForm%3AcommandSimpleFPSearch=Search&si +mpleSearchSearchForm%3Aj_idt447=workaround&javax.faces.ViewState=7923 +733300114075152%3A8489940171963107341' | grep 'Tokyo Parts'
I don't know how long the tokens will be good for. That command will probably stop working after some time.
You may have to move up to WWW::Mechanize (which handles cookies) or even Selenium::Remote::Driver (which handles JavaScript).
Hope this helps!
update:: add mention of Mech, thanks Corion...
|
|---|