What exactly is your question?

Maybe this can help you: When dealing with strange UTF-8 documents, I often use the following bash script.

#! /bin/bash

## Lists all nonASCII UTF-8 characters contained in the data, for each
## character it gives the number of occurences in each file and an
## example.
## Author: E. Choroba

export LC_ALL=C

codes=()
for code in c{{0..9},{a..f}} d{{0..9},{a..f}} ; do
    codes+=($(eval grep -ho "$'\x$code'". "$@" | sort -u))
done
for code in e{{0..9},{a..f}} ; do
    codes+=($(eval grep -ho "$'\x$code'".. "$@" | sort -u))
done
for code in f{0..4} ; do
    codes+=($(eval grep -ho "$'\x$code'"... "$@" | sort -u))
done
for code in "${codes[@]}" ; do
    hexdump <<< "$code" | sed '2d;s=000a==;s= 0a=='
done \
    | cut -f2 -d' ' \
    | sed '/^....$/s=\(..\)\(..\)=\\x\2\\x\1=;
           /^......$/s=\(..\)\(..\)\(..\)=\\x\2\\x\1\\x\3=' \
    | while read -r code ; do
    echo $code
    eval grep -c "$'$code'" "$@"
    eval grep -m1 --color=always "$'$code'" "$@"
done
[download]

I know it is ugly, but it works: it lists all the non-ASCII characters in the given files with counts and examples. It depends on the hexdump utility whose documentation says the following:

The hexdump command is part of the util-linux package and is available from Linux Kernel Archive ⟨ftp://ftp.kernel.org/pub/linux/utils/util-linux/⟩.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.