in reply to is winproxy the problem? send additional header info?
in thread LWP, extract_cookies, etc.

I don't think your problem is releated to headers. Anyway you can add any header to the request using following code 

$request->header(Header => 'Value');

[download]
In you code you will have to replace 

$ua->request(POST ...);

[download]
with 

my $request = POST ...;
$request->header(Header => 'Value');
...
$us->request($request);

[download]
BTW you don't have to insert header User-agent. LWP sends it itself. See documentation for method agent in LWP::UserAgent.

P.S. Post full debug logs here. Maybe you have overlooked anything important there.

--
Ilya Martynov (http://martynov.org/)

Replies are listed 'Best First'.
Full LWP debug log
by inblosam (Monk) on May 24, 2002 at 01:32 UTC
    Here is the full LWP debug log from running my script. The problem is that it doesn't seem that the second post is using the cookie because the page is being directed to a default sign up page.

    One thing I can see is the "no proxy" message, but I don't know what that means or any resolution. THANKS!!

    use perl5::site_perl::5.005::LWP::UserAgent;    #perl2exe
use perl5::site_perl::5.005::HTTP::Request;    #perl2exe
use perl5::site_perl::5.005::HTTP::Message;    #perl2exe
use perl5::site_perl::5.005::HTTP::Headers;    #perl2exe
use perl5::site_perl::5.005::URI;    #perl2exe
use perl5::site_perl::5.005::URI::Escape;    #perl2exe
use /usr/libdata/perl/5.00503/overload.pm;    #perl2exe
use perl5::site_perl::5.005::HTTP::Response;    #perl2exe
use perl5::site_perl::5.005::HTTP::Status;    #perl2exe
use perl5::site_perl::5.005::HTTP::Date;    #perl2exe
use /usr/libdata/perl/5.00503/Time/Local.pm;    #perl2exe
use perl5::site_perl::5.005::LWP;    #perl2exe
use perl5::site_perl::5.005::LWP::Debug;    #perl2exe
use perl5::site_perl::5.005::LWP::Protocol;    #perl2exe
use perl5::site_perl::5.005::LWP::MemberMixin;    #perl2exe
use perl5::site_perl::5.005::HTTP::Request::Common;    #perl2exe
use perl5::site_perl::5.005::HTTP::Cookies;    #perl2exe
use perl5::site_perl::5.005::HTTP::Headers::Util;    #perl2exe
LWP::UserAgent::new: ()
use perl5::site_perl::5.005::URI::http;    #perl2exe
use perl5::site_perl::5.005::URI::_server;    #perl2exe
use perl5::site_perl::5.005::URI::_generic;    #perl2exe
use perl5::site_perl::5.005::URI::_query;    #perl2exe
LWP::UserAgent::request: ()
HTTP::Cookies::add_cookie_header: Checking www.sitedomain.com for cook
+ies
HTTP::Cookies::add_cookie_header: - checking cookie path=/
HTTP::Cookies::add_cookie_header:  - checking cookie LASTVISIT=%7Bts+%
+272002%2D05%2D22+23%3A02%3A55%27%7D
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie CFGLOBALS=HITCOUN
+T%3D53%23LASTVISIT%3D%7Bts+%272002%2D05%2D22+23%3A02%3A55%27%7D%23TIM
+ECREATED%3D%7Bts+%272002%2D05%2D20+23%3A47%3A29%27%7D%23
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie CFTOKEN=60606369
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie OLDDATE=%7Bts+%27
+2002%2D05%2D22+12%3A56%3A16%27%7D
HTTP::Cookies::add_cookie_header:    expired
HTTP::Cookies::add_cookie_header:  - checking cookie CFID=2772192
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie PGSID=60570720020
+513131253
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header: Checking .sitedomain.com for cookies
HTTP::Cookies::add_cookie_header: Checking sitedomain.com for cookies
HTTP::Cookies::add_cookie_header: Checking .com for cookies
LWP::UserAgent::send_request: POST http://www.sitedomain.com/shared/ge
+tuser.cfm
LWP::UserAgent::_need_proxy: Not proxied
use perl5::site_perl::5.005::LWP::Protocol::http;    #perl2exe
use perl5::site_perl::5.005::Net::HTTP;    #perl2exe
use /usr/libdata/perl/5.00503/mach/IO/Socket.pm;    #perl2exe
use /usr/libdata/perl/5.00503/mach/IO/Handle.pm;    #perl2exe
use perl5::site_perl::5.005::Net::HTTP::Methods;    #perl2exe
LWP::Protocol::http::request: ()
use perl5::site_perl::5.005::i386-freebsd::HTML::HeadParser;    #perl2
+exe
use perl5::site_perl::5.005::i386-freebsd::HTML::Entities;    #perl2ex
+e
use perl5::site_perl::5.005::i386-freebsd::HTML::Parser;    #perl2exe
LWP::Protocol::collect: read 209 bytes
LWP::UserAgent::request: Simple response: Found
LWP::UserAgent::request: ()
HTTP::Cookies::add_cookie_header: Checking www.sitedomain.com for cook
+ies
HTTP::Cookies::add_cookie_header: - checking cookie path=/
HTTP::Cookies::add_cookie_header:  - checking cookie LASTVISIT=%7Bts+%
+272002%2D05%2D22+23%3A02%3A55%27%7D
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie CFGLOBALS=HITCOUN
+T%3D53%23LASTVISIT%3D%7Bts+%272002%2D05%2D22+23%3A02%3A55%27%7D%23TIM
+ECREATED%3D%7Bts+%272002%2D05%2D20+23%3A47%3A29%27%7D%23
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie CFTOKEN=60606369
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie OLDDATE=%7Bts+%27
+2002%2D05%2D22+12%3A56%3A16%27%7D
HTTP::Cookies::add_cookie_header:    expired
HTTP::Cookies::add_cookie_header:  - checking cookie CFID=2772192
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie PGSID=60570720020
+513131253
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header: Checking .sitedomain.com for cookies
HTTP::Cookies::add_cookie_header: Checking sitedomain.com for cookies
HTTP::Cookies::add_cookie_header: Checking .com for cookies
LWP::UserAgent::send_request: POST http://www.sitedomain.com/developer
+s/index.cfm?sid=11247720020523182832&CFID=2772192&CFTOKEN=60606369
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 411 bytes
LWP::Protocol::collect: read 1736 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 4096 bytes
LWP::Protocol::collect: read 44 bytes
LWP::Protocol::collect: read 2081 bytes
HTTP::Cookies::extract_cookies: Set cookie CFGLOBALS => HITCOUNT%3D54%
+23LASTVISIT%3D%7Bts+%272002%2D05%2D23+18%3A28%3A33%27%7D%23TIMECREATE
+D%3D%7Bts+%272002%2D05%2D20+23%3A47%3A29%27%7D%23
HTTP::Cookies::extract_cookies: Set cookie LASTVISIT => %7Bts+%272002%
+2D05%2D23+18%3A28%3A33%27%7D
HTTP::Cookies::extract_cookies: Set cookie OLDDATE => %7Bts+%272002%2D
+05%2D22+23%3A02%3A55%27%7D
HTTP::Cookies::extract_cookies: Set cookie PGSID => 112477200205231828
+32
LWP::UserAgent::request: Simple response: OK
LWP::UserAgent::request: ()
HTTP::Cookies::add_cookie_header: Checking www.sitedomain.com for cook
+ies
HTTP::Cookies::add_cookie_header: - checking cookie path=/
HTTP::Cookies::add_cookie_header:  - checking cookie LASTVISIT=%7Bts+%
+272002%2D05%2D23+18%3A28%3A33%27%7D
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie CFGLOBALS=HITCOUN
+T%3D54%23LASTVISIT%3D%7Bts+%272002%2D05%2D23+18%3A28%3A33%27%7D%23TIM
+ECREATED%3D%7Bts+%272002%2D05%2D20+23%3A47%3A29%27%7D%23
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie CFTOKEN=60606369
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie OLDDATE=%7Bts+%27
+2002%2D05%2D22+23%3A02%3A55%27%7D
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie CFID=2772192
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie PGSID=11247720020
+523182832
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header: Checking .sitedomain.com for cookies
HTTP::Cookies::add_cookie_header: Checking sitedomain.com for cookies
HTTP::Cookies::add_cookie_header: Checking .com for cookies
LWP::UserAgent::send_request: POST http://www.sitedomain.com/developer
+s/devedit.cfm
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 188 bytes
LWP::UserAgent::request: Simple response: Found
LWP::UserAgent::request: ()
HTTP::Cookies::add_cookie_header: Checking www.sitedomain.com for cook
+ies
HTTP::Cookies::add_cookie_header: - checking cookie path=/
HTTP::Cookies::add_cookie_header:  - checking cookie LASTVISIT=%7Bts+%
+272002%2D05%2D23+18%3A28%3A33%27%7D
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie CFGLOBALS=HITCOUN
+T%3D54%23LASTVISIT%3D%7Bts+%272002%2D05%2D23+18%3A28%3A33%27%7D%23TIM
+ECREATED%3D%7Bts+%272002%2D05%2D20+23%3A47%3A29%27%7D%23
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie CFTOKEN=60606369
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie OLDDATE=%7Bts+%27
+2002%2D05%2D22+23%3A02%3A55%27%7D
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie CFID=2772192
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header:  - checking cookie PGSID=11247720020
+523182832
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header: Checking .sitedomain.com for cookies
HTTP::Cookies::add_cookie_header: Checking sitedomain.com for cookies
HTTP::Cookies::add_cookie_header: Checking .com for cookies
LWP::UserAgent::send_request: POST http://www.sitedomain.com/developer
+s/signup.cfm?sid=60570720020513131253&CFID=2772192&CFTOKEN=60606369
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 524 bytes
LWP::Protocol::collect: read 1736 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1380 bytes
LWP::Protocol::collect: read 1329 bytes
HTTP::Cookies::extract_cookies: Set cookie CFGLOBALS => HITCOUNT%3D55%
+23LASTVISIT%3D%7Bts+%272002%2D05%2D23+18%3A28%3A35%27%7D%23TIMECREATE
+D%3D%7Bts+%272002%2D05%2D20+23%3A47%3A29%27%7D%23
HTTP::Cookies::extract_cookies: Set cookie LASTVISIT => %7Bts+%272002%
+2D05%2D23+18%3A28%3A35%27%7D
HTTP::Cookies::extract_cookies: Set cookie PGSID => 605707200205131312
+53
LWP::UserAgent::request: Simple response: OK
worked

    [download]

    I appreciate all your help!

    Michael Jensen
    michael at inshift.com
    http://www.inshift.com
[reply]
[d/l]
      Well, I'm happy to say the script is working like a charm. I think the last thing we had to change was get a new session id from the website and I did that real simply with:

      #!/usr/bin/perl -w
#see IlyaM, I am being a good boy!
use strict;
use LWP::Simple;
my $page = get 'http://www.sitedomain.com/login.cfm';
$page =~ /sid" VALUE="(\w+)/;
my $sid = $1;
print $sid;

      [download]

      Thanks to everyone (especially IlyaM!) for their help!

      Michael Jensen
      michael at inshift.com
      http://www.inshift.com
[reply]
[d/l]
[reply]

In Section Seekers of Perl Wisdom