A lot of people have given very sound advice so far; all of which should be heeded. But I am surprised that everyone appears to have missed the most obvious point of all...

Who says you have to be online to run this script?

So you have to process a bajillion messages? It's not like you have to press [enter] between each message.

Put your list of subscribers in a text file that is held on the machine. Use your web form only for updated versions of the file. Then write a script (say "send-newsletter"), that sends out the message to each person. When you run it, run it as "send-newsletter &", i.e., run it in the background. Bingo! You script just took two seconds to run.

You can log off, and if the silly thing takes six hours to complete, it doesn't matter, you don't have to hang around.

There are a number of other things you can do. You can write a logfile to a password-protected URL on your site, to watch the script trundle along.

You could also divide the script in two, if disk space is not a problem. Firstly, you write a script to generate all the emails. (Note, placing more than 1000 files in a directory is not a good idea, find some logical way to split them up among multiple directories). This script will probably be pretty fast, short enough for you to (want to) hang around and check the results. When everything is hunky-dory, you run a second script whose sole purpose is to take all those generated email messages and feed them to your local MTA, taking as long as it takes. That part you might want to run in the background and you go away and do something else.

This approach is nice, because it gives you one more barrier from doing something tremendously stupid, like accidently mailing the subscribers a love-letter from your significant other. You can give a couple of the messages a final once-over, before committing to sending them out.

--
g r i n d e r
just another bofh

print@_{sort keys %_},$/if%_=split//,'= & *a?b:e\f/h^h!j+n,o@o;r$s-t%t#u';

In reply to Re: Speeding up a mailing list script by grinder
in thread Speeding up a mailing list script by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.