What are those boxes, you ask? That's a very good question, and you'll need a very exact answer to that in order to be able to match them, but there's no way for us to answer it based on what you've told us so far.

I'm guessing that you are using some specific window application (or maybe a browser?) to view the data, and the boxes stand for characters that this application is unable to display correctly. The problem is, there might be broad assortment of character values that fall into this "undisplayable" category, and if you just keep using this app the same way to look at the data, you'll never be able to figure out what those characters really are.

That's why many people use one of the various "hex dump" tools on data files, so they can see the actual numeric values of the bytes that make up these undisplayable characters, and figure out what needs to be done once they know what these characters really are (in the binary sense at least, if not in a human-language sense).

Perl itself can be used easily to create a hex dump of the data -- something like: 

#!/usr/bin/perl

$/=undef; # turn on slurp mode for reading
$_ = <>;  # read all data into $_ (from STDIN or @ARGV file(

while (length()) {
    @s = split //, substr( $_, 0, 16 );
    print join " ", map { sprintf "%02x", ord($_) } @s;
    print "\n";
    print join " ", map { (/[ -~]/) ? " $_" : '~~' } @s;
    print "\n\n";
    $_ = substr( $_, 16 );
}

[download]
That will provide the hex codes for all the bytes in your data, along with the ascii characters for the bytes that happen to be in the printable ascii range.

Another thing you could do is figure out what character encoding is being assumed by your display application (whatever it is), and then see if you can find out what character encoding is represented in your data file. I expect there's a mismatch between those two encodings, and that is why you are seeing those boxes.

(The problem also would relate to the font that your application is using, because the boxes are the symbol provided by that font to represent code points for which it does not have a displayable character glyph.)

(update: obviously, almut has shown that you did in fact post enough information in order for someone here to tell you what your box characters are... but please be aware that you might encounter some other piece of data (which you haven't posted here yet) that will also show little boxes when you view it this way -- and it's not guaranteed that every box you see will always be a 0x19 byte value.)

In reply to Re: Searching between wierd box characters. by graff
in thread Searching between wierd box characters. by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.