I am not sure what the naming convention is? Is "uniquedata" something that changes on a per month basis? And if so, can you predict with certainty what it will be next month?

Could you take a couple of example documents and show what the full URL is to these documents for say June 2022 and July 2022?.

Are you saving the .pdf doc as a "BLOB" inside the SQLite DB or are you just saving a directory path on your local machine?

I have a similar web application. My app runs once per hour, looks around on part of a particular website for any new links. If it finds one, it "clicks" on it, to see there is anything there is "interesting or not". If so, the interesting data is saved in an SQLite DB. In any event, I save the URL in the DB so that I don't go there again. If nothing changed on the website, it figures that out very efficiently. This thing has been running every hour for the past 6 years, so it is possible for apps like this to work out very well. I use WWW::Mechanize but you don't seem to need the sophistication of making sense of a webpage? Or do you?

Update:
Could you show your table schema?
I would be thinking along the lines of:
URL text (where the data came from)
Version Datetime (this is actually just text (not numeric) yyyy-mm-dd hh:mm:ss)
- you can omit time and I think also the dd if not available
- leading zeroes are mandatory because this column must be in
- ASCII sort order
Downloaded Datetime (optional but often handy to know)
title text (Name of the document)
pdf blob (actual pdf file)

In reply to Re: Combining Ffile:fetch with MySQL by Marshall
in thread Combining Ffile:fetch with MySQL by justin423

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.