What have I done wrong?

The index is zero based, so if you want to treat the first line in the file as 1st rather than 0th, you need to substract 1 from the number before looking it up in the index:

open INDEX, "<:raw","$index" or die "Can't open $index for reading: $!
+";
my $len = -s( INDEX );
sysread INDEX, my( $idx ), $len;
close INDEX;

open FILE, "<$oneper" or die "Can't open $oneper for reading: $!";

foreach my $lineNum (sort {$a cmp $b} keys %todo) {
    my $offset = unpack 'N', substr $idx, ( $lineNum - 1 ) * 4, 4;  ##
+ Modified!!
    print "offset is $offset for linenum $lineNum\n<br>";
    seek FILE, $offset, 0;
    my $line = <FILE>;
    print "found line $line\n";
}
[download]

That should do the trick.