tcordes has asked for the wisdom of the Perl Monks concerning the following question:

Hi again! Had some more time to work on my W:M:F project. I installed the latest version today and it appears all my slowness/delay problems are fixed! Thanks for that! The whole thing runs way faster now and I don't have to ugly-hack the code.

My current problem is save_url isn't sending cookies with the GET request. I am get()ing a parent page then save_url a link to a large zip file in that page. The site *requires* cookies on all GETs, even for the zip, or it redirects to an error page.

I used wireshark to look at the traffic when I manually right-click and click save as in firefox vs. using W:M:F to save_url. Manually, the cookies get sent. save_url does *not* send the cookies (or referrer).

So I tried adding to the save_url code:
obj_Persist.persistFlags = obj_Persist.persistFlags | nsIWBP.PERSIST_FLAGS_FORCE_ALLOW_COOKIES;
after:
obj_Persist.persistFlags = flags | nsIWBP.PERSIST_FLAGS_FROM_CACHE;

As the mozilla docs seem to indicate that would do what I wanted. The cookies still do not get sent. I'm not sure why this doesn't work, unless the nsIWBP.PERSIST_FLAGS_FORCE_ALLOW_COOKIES option is broken. I checked and confirmed my Firefox/Gecko is new enough to support it, supposedly.

It makes sense to me that by default save_url should pass all cookies the site is expecting automatically, just as if you right-clicked and clicked save as manually.

Is there another approach I can take? Any ideas? I was thinking maybe pulling the cookies out and then passing them through manually into the obj_Persist.saveURI aExtraHeaders argument.

Thanks!

  • Comment on WWW::Mechanize::Firefox save_url doesn't send cookies

Replies are listed 'Best First'.
Re: WWW::Mechanize::Firefox save_url doesn't send cookies
by Corion (Patriarch) on Mar 08, 2011 at 21:46 UTC

    Thanks for the report! I had also assumed that the cookies are sent automatically, but if that's not the case, I think you will have to look at nsICookieService and ask it to tell you the cookie string that is to be sent as additional headers. If I get around to writing a self-contained testcase, I'll test this out and then add it to WWW::Mechanize::Firefox, but I don't see a quick workaround other than using ->get instead of ->save_url - and ->get may or may not save the target, depending on your Firefox setup.

[reply]
[d/l]
[select]

      I played around some more. My last report was erroneous, adding nsIWBP.PERSIST_FLAGS_FORCE_ALLOW_COOKIES does fix the problem! I had thought it didn't as I was updating the old .pm file not the new version, so my changes weren't even executing (doh!).

      So the following lines in save_url fixes it and now it works great!!!! Thanks for the great perl module, opens up a whole new world of mech'ing.

      obj_Persist.persistFlags = flags | nsIWBP.PERSIST_FLAGS_FROM_CACHE;
      obj_Persist.persistFlags = obj_Persist.persistFlags | nsIWBP.PERSIST_FLAGS_FORCE_ALLOW_COOKIES;

[reply]

        Great! Thanks for testing again and confirming that this works. I'll release a new version with the bugfix (and a privacy warning) some time in the next few days. The code will be changed to the following, to support earlier versions of Firefox 3 that don't have the flag yet:

            obj_Persist.persistFlags = flags | nsIWBP.PERSIST_FLAGS_FROM_CACHE
                                     | nsIWBP["PERSIST_FLAGS_FORCE_ALL
+OW_COOKIES"]
                                     ;

        [download]

        I just want to also ship a local test server to actually confirm locally that the appropriate cookies are sent.

[reply]
[d/l]

Back to Seekers of Perl Wisdom