Re: How to check for new records in mysql database every minute
by cLive ;-) (Prior) on Oct 05, 2004 at 21:55 UTC
|
Add a timestamp field (epoch time) to each table - int(10) unsigned - and create an index for it, then just select records created in the last minute.
my $min_old = time-60;
my $sth = $dbh->prepare("SELECT * FROM table WHERE timestamp>$min_old"
+);
Well, that's what I'd do, anyway.
cLive ;-)
| [reply] [d/l] |
Re: How to check for new records in mysql database every minute
by TedPride (Priest) on Oct 06, 2004 at 01:39 UTC
|
Yes, a timestamp is by far the best method. You can even set up mySQL to automatically update your timestamp for you, by using the TIMESTAMP column type:
"The TIMESTAMP column type provides a type that you can use to automatically mark INSERT or UPDATE operations with the current date and time. If you have multiple TIMESTAMP columns, only the first one is updated automatically.
Automatic updating of the first TIMESTAMP column occurs under any of the following conditions:
* The column is not specified explicitly in an INSERT or LOAD DATA INFILE statement.
* The column is not specified explicitly in an UPDATE statement and some other column changes value. (Note that an UPDATE that sets a column to the value it already has will not cause the TIMESTAMP column to be updated, because if you set a column to its current value, MySQL ignores the update for efficiency.)
* You explicitly set the TIMESTAMP column to NULL.
TIMESTAMP columns other than the first may also be set to the current date and time. Just set the column to NULL, or to NOW(). "
All you have to do is store the time you last checked the database in a text file, and update the text file with the new timestamp whenever you make a check. | [reply] |
Re: How to check for new records in mysql database every minute
by mifflin (Curate) on Oct 05, 2004 at 21:37 UTC
|
You could put a trigger on the table.
See MySql trigger docs here...
http://dev.mysql.com/doc/maxdb/en/a7/41ee0b605911d3a98800a0c9449261/content.htm
You could have this trigger insert into another table for each row that is inserted, updated or deleted.
You would then write a perl program to select from this new 'transaction' table.
This will keep you from doing full table scans off the actual table trying to determine the new records and gives you a way of knowing the records that were deleted.
Update:
Ok, dragonchild points out that triggers are not prime time yet. Can you change the API that does the Insert/Update/Deletes to the table? If you can then you can simulate a trigger by populating a 'transaction' table there. Deletes are the tricky part of your requirements and will require some kind of transaction queueing to manage. | [reply] |
|
|
Triggers are not production-ready yet. They're in the 5.x series, which is still under new development.
Being right, does not endow the right to be rude; politeness costs nothing. Being unknowing, is not the same as being stupid. Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence. Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.
| [reply] |
|
|
well that sucks. I guess I've been spoiled doing all my db development in Oracle
| [reply] |
|
|
Re: How to check for new records in mysql database every minute
by dragonchild (Archbishop) on Oct 05, 2004 at 21:43 UTC
|
You can enable query caching and see whether or not the query cache has been flushed for that table or not. The theory is that the cache remains for a given query so long as the table is unchanged. Once the table is changed, the cache of any queries against that table is invalidated. I'm not sure how you'd check to see if the cache was invalidated or not.
Another option is to maintain an auditing table. Ideally, you would use triggers to maintain this, but this can also be maintained within the application as well. Basically, every time you modify the table in question, you insert a record into the audit log. Then, you can check to see if the audit log has grown in the intervening period.
My question is why you want to know this - what problem are you trying to solve? There is probably an easier way to solve it than what you're trying to do ...
Being right, does not endow the right to be rude; politeness costs nothing. Being unknowing, is not the same as being stupid. Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence. Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.
| [reply] |
Re: How to check for new records in mysql database every minute
by ikegami (Patriarch) on Oct 05, 2004 at 21:38 UTC
|
It would probably be simpler to do something like UPDATE CounterTable SET Counter=Counter+1 when you do an INSERT. You can find the changes by selecting with the clause ORDER BY auto_increment_field DESC or ORDER BY creation_date_field DESC, and reading as many records as the difference in Counter since the last time you checked.
Caveat: Some databases allows you to specify value for an auto-increment field. You can't use that feature if you want ORDER BY auto_increment_field DESC to work.
| [reply] [d/l] [select] |
Re: How to check for new records in mysql database every minute
by Happy-the-monk (Canon) on Oct 05, 2004 at 21:33 UTC
|
Maybe you want to read the SQL log for that?
Edit:
And I dreamed it was that easy. You would have to parse the binary Update log instead, so I hope there is an easier way to solve your problem.
Cheers, Sören
| [reply] |
|
|
The mysqlbinlog utility will parse the binlog for you and takes all sorts of arguments like a start and stop time for instance. You don't need to but wrapping it with XS would be easy.
| [reply] |
Re: How to check for new records in mysql database every minute
by talexb (Chancellor) on Oct 05, 2004 at 21:36 UTC
|
# Make a connection to the database;
# While not killed by a signal
# Check to see if there are any new records, and do
# something with the new records;
# Sleep for some period;
# Close the connection to the database.
Alex / talexb / Toronto
"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds
Update: Well, my comments were intended for some definition of new. I was leaving the details of the solution to the Original Poster, as the problem definition was itself somewhat vague. If that wasn't useful, I'll try to reply to open-ended nodes like this one less often. | [reply] [d/l] |
|
|
"Check to see if there are any new records" ain't easy!! Can't just count them, cause it must support DELETEs.
| [reply] |
Re: How to check for new records in mysql database every minute
by Anonymous Monk on Oct 06, 2004 at 11:15 UTC
|
- You use an update trigger on the tables involved. Oh, wait, MySQL doesn't have triggers.
- You use a stored procedure to do inserts in the tables involved, and modify all programs not to touch the tables directly, but go through the stored procedures. Oh, wait, MySQL doesn't have stored procedures.
- You replicate the tables involved using a repserver, dropping any deletions and updates, and check the tables at the other end. Oh, wait, MySQL doesn't have a repserver.
- You use a real database.
| [reply] |