Esteemed Brethren,

as you may know, we (creamygoodness, dpavlin, and I) are working on a full-text perlmonks search. Corion provided us with the first 100,000 nodes in XML format. What I have noticed is that there are about 40 nodes whose XML cannot be parsed dude to funny characters (see example).

This brings up several questions:

• How do those funny characters get in there in the first place? Should we strip them out as user post things?
• How did Corion create the XML nodes (which library did he use) and which library does perlmonks.org use to display nodes in XML?
• How should the MonkSearch team proceed? Would it be wise to just strip out those characters before parsing XML out?
• Shouldn't XML encode those characters somehow? Something like &0x1234; or however HTML entities are handled?

Thanks,
  - Dmitri.

Full list:

657 672 5132 5549 5767 7641 7888 10277 11869 15824 16378 19648 21510 21645 22045 26561 27261 29149 32115 32715 33870 35485 36797 36962 37826 42307 51955 52426 62887 75822 79160 85146 87117 87271 87991 89029 91802 94571 96186 97045

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.