Processing a dynamic array

lampros21_7 has asked for the wisdom of the Perl Monks concerning the following question:

Good evening monks, i have a question which has been puzzling me and i can't find a solution... Thus asking!

I have a loop which takes an array that already contains some url's. The array starts processing the url's and after some work for filtering which links i need it adds the links to the array which i use for the loop. And that seems to confuse the program. For and foreach loops don't work for sure, and while loops don't seem to work either.

If i put a number for example x < 900 then the script is executed perfectly but if i leave something like x < links then it doesn't. Is there a solution for a problem like this? (I presume there is!). Thanks for the help in advance.

for(my $x = 0; $x < @links; $x++) {
 
     $webcrawler->get($links[$x]);
          
     my @more_links = map { $_->[0] } $webcrawler->links; # get Links
    
    @more_links = map { URI->new_abs( $_, $base_url ) } @more_links; 
 
     @more_links = map { /^$base_url/ ? $_ : () } @more_links;
     
     # Add the filtered links from the more_links array to the links a
+rray
    push(@links,@more_links); 
    
    #Finally, recursively go through the array and remove URL's that a
+ppear twice or more
    @links = do { my %z; grep { !$z{$_}++ } @links };
 }
[download]

Comment on Processing a dynamic array Download Code

Replies are listed 'Best First'.
Re: Processing a dynamic array by merlyn (Sage) on Oct 29, 2005 at 15:19 UTC
Don't use an index on @links... use a processing queue: `my @links = (... initial set ...); while (@links) { my $one = shift @links; ... process $one ... push @links, (... more to do... ); }` [download] -- Randal L. Schwartz, Perl hacker Be sure to read my standard disclaimer if this is a reply.	[reply] [d/l]
Re: Processing a dynamic array by pg (Canon) on Oct 29, 2005 at 16:16 UTC
"...and while loops don't seem to work either. If i put a number for example x < 900 then the script is executed perfectly but if i leave something like x < links then it doesn't..." while loop should work, the issue is the condition you give to the while loop (whether you take any sanp shot): x < 900, a predetermined big number, that probably works MOST of the time, as long as the number is big enough (so that it is greater than the eventual size of the array). But that's not how you write nice and decent code. as you want your code work ALL the time. x <= $#links does not work as the array is growing, but the snap shot of the array size was only taken once. while works if you give the correct condition such as below, in which there is no snap shot taken: (the following example simulates your situation by saying that each even number has a link that is its half.) `use strict; use warnings; my @array = (1 .. 4); while (@array) { my $first = shift @array; print "[$first]"; push @array, $first / 2 if (!($first % 2)); }` [download] This prints the correct answer: `[1][2][3][4][1][2][1]` [download] A for loop works as well, as long as you don't take any snap shot: `use strict; use warnings; my @array = (1 .. 4); for my $element (@array) { print "[$element]"; push @array, $element / 2 if (!($element % 2)); }` [download] This also gives the correct answer.	[reply] [d/l] [select]
Re^2: Processing a dynamic array by lampros21_7 (Scribe) on Oct 29, 2005 at 16:32 UTC
thanks pg and merlyn, dont quite understand what you mean pg "the following example simulates your situation by saying that each even number has a link that is its half". I don't mention something like that in my post, could you explain it?Thanks	[reply]
Re^3: Processing a dynamic array by pg (Canon) on Oct 29, 2005 at 17:11 UTC
That was my fault to make up the link thing ;-) Sometime, you thought you read certain things, but they are not there. That's fine. The whole purpose was to simulate your situation: a growing array where the growing eventually stops.	[reply]
Re^4: Processing a dynamic array by lampros21_7 (Scribe) on Oct 29, 2005 at 21:20 UTC
Re^5: Processing a dynamic array by pg (Canon) on Oct 29, 2005 at 23:29 UTC
Re: Processing a dynamic array by rinceWind (Monsignor) on Oct 29, 2005 at 17:13 UTC
I did something very similar to what you want to achieve here. I check the hash %seen (equivalent of your %z) on the way in, which means that duplicates never get into the array in the first place. -- Oh Lord, won’t you burn me a Knoppix CD ? My friends all rate Windows, I must disagree. Your powers of persuasion will set them all free, So oh Lord, won’t you burn me a Knoppix CD ? (Missquoting Janis Joplin)	[reply]