For complicated regexes, you should use the same advice you would for complicated code, break it up. For something this large, I would definitely use the x modifier to allow ignoring whitespace and comments. You also want to be more specific in your matches where possible.

  my $log_pattern = qr{
       ^
       ([\s.]+) \s  # match the IP address
       - \s - \s    # ignore these fields
       \[([^]]+)\]  # here's probably where your problem was

   ...

  }x;
[download]

Following the lead above, you should be able to construct the rest of the expression.

You might also want to check out Apache::LogRegex. I've never used it, but it looks like it might solve your problem.