in reply to Module to extract text from HTML

Have you tried using XPath to search with some Perl wrapper(s) around XML parsing librar(y|ies)? I personally do not remember Perl ones (Python has lxml package around libxml2 C library).

time passes XML::LibXML could have been the one.

Replies are listed 'Best First'.
Re^2: Module to extract text from HTML
by marto (Cardinal) on Feb 27, 2024 at 11:46 UTC

    Mojo::DOM is a parser which makes this trivial, however I get the impression from question that it's less about selecting a particular parts of the page ('just extracting the p tags which is not quite good enough'), and more about 'all' of the text.

      I get the impression from question that it's less about selecting a particular parts of the page

      We already hold the website of our customers (typically UK charities). We want them to complete a short section about their organisation. This is used to construct prompts for AI tools around our site that they use to streamline their workload.

      I am trying to make it easier for them to complete the description of their organisation by pulling text from their own website. This will give them something to work with instead of having to begin with a blank canvas (or contenteditable div).

        If I understood correctly that you are in control of websites and the formatting of their content, perhaps you could add some tags to the content by means of html comments or, better, custom attributes for html tags <p "data-purpose"="description" "data-index"="1">blah blav</p> and then you just reconstruct the text content from html.