a particular portion of the web page gets updated through asynchronous postback. The HTML code is not actually seen in the souce code of the page.
My knowledge of ASP.NET is extremely marginal1, but the above description makes it sound as if Javascript is involved, as if portions of the web page are being retrieved (or created) and dynamically inserted, AJAX style, on the client side. It then sounds like you're trying to do client-side testing of the page, using Perl. If any of that's wrong, this would be a good time to reply and correct me...
Assuming I've correctly interpreted your situation, I have a couple of statements, a couple of suggestions, and a question or two...
- Perl is, on the whole, a pretty good language for doing client-side
testing of dynamic web stuff, although Javascript is much harder to
deal with than server-side code.
- Anything that consistently appears on the UI of the web page
(even if you view the page on another computer or with a different
browser) has got to be indicated somewhere in the source,
in some fashion or another. If you were seeing something that
was being inserted by your web browser or some other program
running on your computer, something not specified by the site,
then it would not appear if you visit the site from a different
computer. For the rest of this post, I'm going to assume that's
not the case, that what you are seeing is, in fact,
indicated somewhere in the source for the page, in
some fashion or another, and that you just haven't yet figured
out exactly where and how.
- You haven't stated which Perl modules you're using. Personally, I
generally use a combination of WWW::Mechanize and HTML::Tree, but
there are other options. (Since the site is ASP.NET, I would guess
that the various popular XML libraries are probably not suitable,
because the site is probably a hodgepodge of invalid SGML,
but I'm guessing here, so YMMV. Even setting XML libraries, of
which there are many for Perl, off to one side, there are still
several other options for working with HTML. HTML::Tree just
happens to be what I have used.) Anyway, all of that is to say,
if you are having trouble parsing the HTML and finding the
relevant parts, a module designed for working with HTML can
definitely help.
- If the page has just a small amount of relevant Javascript
code that always does the same thing and doesn't change from
one visit to the site to the next, you can look at that
Javascript with your human eyes, determine what it is doing,
and write Perl code that just does the same thing. This avoids
the need to parse the Javascript code or mess with it really.
For instance, if all the Javascript code does is set a form
variable and then submit the form (an annoyingly common
scenario), you can easily simulate that with WWW::Mechanize.
- There are Javascript modules on the CPAN, but I don't know how
well suited they are for what you're doing or how well they
integrate with the other modules I have mentioned. (I'm not
saying they're not suited and don't integrate well; I'm only
saying I don't happen to know.)
(By <q>extremely marginal</q> I mean that I probably know more about any random subject you should care to name than the topic in question. For instance, I probably know more about particle physics than I do about ASP.NET, though I only had two physics classes, both basic overviews, one in high school and one in college. I'm not even entirely certain I know what ASP.NET is that makes it different from regular ASP, and I don't know that much about regular ASP either, except that it reminds me of PHP, and looking at code written in it makes me want to wash my hands with iodine solution.)