I am asking this as a hypothetical with some serious applications and I hope that this may help someone else in the future.

I wrote a data entry web application with states in it. I was not aware of CGI::Application or any of the state modules so it took in data and time issued and generated a key made of an MD5 and a random component that allowed me to get the data back out when the page came back. This was stored in a MySQL database.

FUNCTION MAKE_STATECODE

ARGUMENTS: 

    UserID
    What state the should go to when this page is recieved
    Some additional data if needed
    When the code was issued
    When it expires ( because some things have a longer timeout )

[download]

RETURNS:

A code of speccified length that refers to the data stored in a table allowing the end user to only get the information and pages pertaining to him or her.

This worked really well when I was generating at most five or six codes per page. And we used it for two years. But I worked out a script that masks SQL tables or views and can auto generate tables and forms. I did see some of them in cpan but I did not like the API. Anyway I was generating one statecode for each record to link to each editable record page.

I banged it up so serverely trying to get it to work. That I dont want to show the really ugly code but i will explain the algorithm. 

Step One ( to speed things up ) run by cron once a night

Open database 
Loop based on previous days load ( about 10k linkcodes ) 
    Make random statecode with no data.
End Loop


Step Two ( when i generate a page with linkcodes )

Open Database

Try to get statecode not in use

If avaliable 
    write data to statecode
    return statecode
Else
    make new statecodes ( about 100 )
    write data to statecode
    return statecode
End If

[download]

Then things started to slow way down when I needed a bunch of statecodes ( 1000+ ). I pre-compiled the statement by moving the prepare out of the loop and that helped but not enougth. Multiple inserts did not help much either since the problem is the speed of which data is exchanged between my server and my database server. We are talking about 10 linkcodes a second.

Thus I need a new state module to wrap my existing module around so that the rest of the program is unaware. ( OO rocks! ) what do you reccomend. My disk space is mapped so having it running on several servers at the same time is not an issue.

What I am concerned with is when I pull the data out when the statecode comes back if I do use a flat text file when I have two big tables in use ( which is only for admin view ) thats 4 K linkcodes plus the users and every retrieve is doing a linear search.

Any ideas since using CGI::Session is just going to put me in the same situation.

In reply to Maintaining State efficiently for large numbers of statecodes by Angel

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.