A 5000 by 5000 matrix contains 25 million elements. That is 200 million bytes of 64 bit doubles, together with 5001 perl array header structures. You need three of those monsters to do an array multiplication. Do you suppose you're hitting swap?

Since most N-element matrix multiplication schemes involve O(N**3) multiplications, you're on the bad side of a scaling problem, as well.

Your strategy of serializing to disk looks pretty good in light of that. You may want to transpose the right hand matrix in order to get the columns as a single array ref.

The best cure would be less brute force. Try to reduce your problem to a block diagonal form if you can. Think about transformations that may simplify things. Would approximate knowledge of the eigenvectors help?

In any case, PDL is superior to pure perl math packages for speed and flexibility.

After Compline,
Zaxo

In reply to Re: Fast matrix multiplication by Zaxo
in thread Fast matrix multiplication by ludo

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.