in reply to How to generate random sequence of UTF-8 characters

A random sequence of all possible UTF-8 characters? That is going to be quire a huge string.

Anyway, don't worry about UTF-8. You must just create characters, which you can do with chr($code), where $code is an integer, a valid UNICODE character code point, which is basically any value between 0 and 0x10FFFF — except some of those will represent unprintable characters, such as control characters, and you probably don't want those.

Let Perl worry about converting it to UTF-8.

Replies are listed 'Best First'.
Re^2: How to generate random sequence of UTF-8 characters
by ted.byers (Monk) on Dec 20, 2012 at 20:33 UTC

    Thanks. I will give it a try.

    There is a point of misunderstanding, though, and that is I am aiming for a sample of random sequences. Each sequence would be five to ten characters, but the sample would be comprised of a few million such sequences. Thus, if my sample size is ten million strings, and each string is ten characters, and there are a million valid utf-8 characters, the each character would be in the sample an average of 100 times. It is a statistical approach; each item in the sample has just a tiny portion of all possible values, but the whole sample includes all possible values multiple times. I tend to be a bit thorough when testing code I am not familiar with (my code for computing eigensystems of general matrices was testing on 100 million randomly generated matrices - with not one failure BTW).

    Thanks again.


      Ah, OK. Well the system is the same as you used to generate passwords: once you have an array of valid code points for the characters, you can choose 10 random codepoints from that array, like
      $chosencodepoint = $codepoints[int rand @codepoints];
      and construct the string, for example with:
      $string = pack 'U*', @chosencodepoints;
      (pack('U', $int) is pretty much equivalent to chr($int) except it also guarantees the output will be turned into UTF-8, and most of all: with the star this can easily join multiple characters without an explicit join.)

      My point was: when printing it out, Perl will convert it to valid UTF-8. Don't worry about that.

      An in case you want no duplicates, you can repeat the process for each character until you find no duplicates. With so many characters to choose from that is virtually guaranteed to be faster than shuffling the whole array (with the Fisher Yates shuffle) and next picking the first 10 code points. Or, you can make custom version of Fisher-Yates that stops shuffling after 10 iterations.