Hello,

I am new to Perl. I am currently trying to write a very simple vector space algorithm. I have searched the site for similar questions to mine, but I have not found anything.

I have programmed this algorithm in another language in the past (i.e. LabView), so I know everything that I want to do. Sadly, I do not know how to do these things in Perl!

The first thing that I need to do is read a large dataset of xml documents (which are all different folders in one parent directory) and pick out all of the words in them. Then I need to refine that word list into a dictionary.

I have tried following the example vector space algorithm here: http://www.perl.com/pub/2003/02/19/engine.html

However, it reads the dataset into the RAM. I have about 1gb of RAM and the dataset is 2.5gb, so I am guessing that this will not work.

Here's what I have started with:

use File::Find;

my $localdir = '[PARENT DIRECTORY]';

find(
sub { print $File::Find::name, "\n" if /\.xml$/ },
$localdir);

[download]

This outputs all of the file names and locations for the xml files. I guess that this is a good start for reading the files in one at a time so that I can create my dictionary.

Can anyone tell me how to then create the list of words as a separate file?

I am very grateful for your assistance.

Adam

In reply to Vector space algorithm by AdamEdinburgh

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.