in reply to Language recognition of web pages

First note that anguage identification is different from charset identification. Several languages can often be expressed with the same charset. There's no Right Thing to do here, not only because eight bit encodings have overlapping characters.

If you want to try a heuristic you can pick a language identification scoring function, then feed in the data and choose the best match. If the input might also be in UTF-8 (or other encodings), you have to feed it in more than once. You do have a shortcut here; some streams are not valid UTF-8 and you can tell that pretty quickly.

One way to score langauge match is to look at bigram frequency. Another is to use compression analysis.