Where does $system come from?

Can you reproduce the problem without the foreach my $uid (...) loop?

$systems is a hash of hashes i.e. $system{$sys}{$uid} where $sys is a system for example, email and $uid is a userid on that system. The module here is Mozilla::LDAP.

Could you show us the code where $system is populated?

Also, use Devel::Peek to print the internals of some of its values and let us see the output. For instance:

use Devel::Peek;
...
$system{some_host}{some_user} = $ldap->whatever();
Dump($system{some_host}{some_user});
[download]

For example, the following code will produce the "Use of freed value in iteration" error:

my $hr1 = {};
my $hr2 = {};
my $hr3 = {"$hr1" => $hr1, "$hr2" => $hr2};
my @values = sort values %$hr3;
print "@values\n";
foreach my $v (sort values %$hr3) {
  print "$v\n";
  delete $hr3->{$values[1]};
  # some_function()
}
[download]

After doing a simple test this definitely is the case. If I simply push a string onto an array after deleting the hash key $v ends up getting the string. On my machine the following code prints:

HASH(0xa00003da0) HASH(0xa00003f38)
HASH(0xa00003da0)
hello
[download]

my $hr1 = {};
my $hr2 = {};
my $hr3 = {"$hr1" => $hr1, "$hr2" => $hr2};
my @values = sort values %$hr3;
print "@values\n";
my (@arr);
foreach my $v (sort values %$hr3) {
  print "$v\n";
  delete $hr3->{$values[1]};
  push @arr, "hello";
}
[download]

This seems very dangerous.

The code is modifying the hash while iterating over its values and using those values as the keys for deletion. You perhaps already tried this, but if you change the loop to iterate over @values or the keys of $hr3 then the issue disappears. Same if you stringify the loop iterator $v.

Others can better comment on the details, but I assume perl is re-using memory and your code is reaching into the old locations.

For completeness, this value replacement also occurs when iterating over values of an array, and the replacement value can be from a new hash key or a new array element. The following code

my @arr = ("item 1","item 2");
my %h;
foreach my $v (values @arr) {
  print "$v\n";
  delete $arr[1];
  $h{x} = "sneaky hash value"
}

@arr = ("item 1","item 2");
my @arr2;
foreach my $v (values @arr) {
  print "$v\n";
  delete $arr[1];
  push @arr2, "sneaky array value";
}
[download]

produces

item 1
sneaky hash value
item 1
sneaky array value
[download]

foreach $uid (keys %{$systems{$sys}}){
[download]

@userids=keys %{$systems{$sys}};
foreach $uid (@userids){
[download]

Does anyone know what this is any different?

Yes, there is a bug on one of the modules you are using, that returns perl scalars with a wrong reference count. Perl perform an integrity check looking for this kind of errors inside for loops and that was why you were seeing the "Use of freed value in iteration" message.

And now, you are not using the faulty values, but copies of them (@userids). Perl doesn't see any problem because it's not looking at the faulty values any more... but that doesn't mean you have solved it, it's just hidden, waiting to crash your program at any time!!!

I succeeded in generating this error with a pure perl program (v5.10.1 (*) built for i686-cygwin-thread-multi-64int) with the following code:

#!/usr/bin/perl -w
use strict;
use Data::Dumper;
$Data::Dumper::Indent = 1;

my $hashref;

for my $i0 (1..2) {
  for my $i1 (1..2) {
    for my $i2 (1..2) {
      $hashref->{"$i0"}{"$i0:$i1"}{"$i0:$i1:$i2"} = "$i0:$i1:$i2";
    }
  }
}

deletehashlevel($hashref, 1);

print Dumper($hashref);

exit;

sub athashleveldo {
  my ($h, $n, $action) = @_;
  if ($n==0) {
    $action->($h);
  } else {
    athashleveldo($_, $n-1, $action) for values %$h;
  }
}

sub deletehashlevel {
  my ($h, $n) = @_;
  athashleveldo($h, $n, sub {
    my %tmp;
    for my $key (keys %$h) {
      my $val = delete $h->{$key};
      @tmp{keys %$val} = values %$val;
    }
    %$h = %tmp;
  });
}
__END__
[download]

Running this program:

$ ./testhashdel
Use of freed value in iteration at ./testhashdel line 27.
[download]

The error goes away when I apply the same solution as js1 did, replacing:

  athashleveldo($_, $n-1, $action) for values %$h;
[download]

with

  my @v = values %$h;
  athashleveldo($_, $n-1, $action) for @v;
[download]

Is this a bug in Perl, lurking to bite me at some later time?

foreach my $uid(keys %$userids){
[download]

foreach my $uid(keys %userids){
[download]

Not necessary. Without seeing his data structure, you cannot safely assume that $userids is not a hash ref. If it is a hash ref, then his syntax is alright.

Good point.

Forget that fear of gravity,
Get a little savagery in your life.