> I have currently done this using arrays over the 2 million pairs that arise from 2,000 products and it ran for 4 days before

I think your speed problems result from linear searches.

You may want to organise your data in a nested structure.

See perldsc And perlref

Like (draft)

$h_record = { 
         date     => '2018-01-01', 
         product  => 'apple', 
         price    => 200, 
         quantity => 50000,
     };
$product{apple}{'2018-01-01'} = $h_record; 
$date{'2018-01-01'}{apple}    = $h_record;
[download]

• Like this you can easily lookup products and dates and pass records to subroutines.
• Another hint is to cache and not repeat already done calculations.
• Last but not least keep an eye on memory consumption, as soon as the system needs to swap heavily, you'll be lost in a black hole. (Only processing chunks may help here)

Please note (like others mentioned) that this approach can't compete with an SQL DB in terms of efficiency.

Those hashes need lots a memory where a SQL is managing an index on columns of the same table. (Think of all the hashes you'd need to easily look up year, month and day separately)

Also reading files to build the data structure will cause you overhead where SQL is persistent.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.