Working on a new DBD::SQLite release

A bit of history

The last official release of DBD::SQLite at the time of writing is DBD::SQLite 1.14, from September 2007. In 2008, audreyt picked up the baton with DBD::SQLite::Amalgamation, which was a fork of DBD::SQLite. It introduced the use of the SQLite amalgamation, the distribution of the SQLite code in one huge C file.

DBD::SQLite::Amalgamation also absorbed some bug fixes and enhancements over DBD::SQLite, so that the two have somewhat diverged in their feature set. While this was unavoidable, it's not really convenient for the programmer, because it's confusing if one version of DBD::SQLite delivers the column metadata while the next higher version does not.

Now Adam Kennedy has kicked off a new round of working on DBD::SQLite, together with Alexandr Ciornii and Kenichi Ishigaki. I've joined them, transplanting some of the features that I pushed into DBD::SQLite::Amalgamation into the mainline DBD::SQLite.

How can you help?

I need your help in building DBD::SQLite and possibly testing the new developer releases with your applications. My test environments (Debian and Windows XP, SQLite 3.6.12) don't necessarily match the other setups out there in the world (64bit Perl, your applications). Also, your eyes and ideas can help in triaging or even resolving bugs and bug reports on RT.

My approach to fixing things in XS or C is to do a modification and then check that all tests still pass. This won't ferret out memory leaks due to bad refcounting (missing dec_SvRef... for example). Such kinds of bugs only manifest themselves outside the limited test suite in applications and hopefully in the test suite of applications. This approach also doesn't ferret out all dangling pointers, but I hope that casting a larger net will also catch some segfaults that have gone undetected so far.

What's left to do?

With DBD::SQLite 1.19_05, I've well run out of things within my expertise. I'm going through the RT bug queue and am triaging some the open bugs. If some of the bugs below are a particular itch of yours or patches scratch such a particular itch, your voice could move them to the front of the queue.

Things that should work now:

user defined collations, are in from DBD::SQLite::Amalgamation
support for column_info, support for column_info, transplanted
Fix memory leaks (possibly integrated, ~~or not yet~~ 1.19_06)

Things that add features, need some tests or simple verification:

closing open active sth fix, report of closing active sth - these have been transplanted.
Faster parameterless ->do() - this looks simple to implement but doesn't have a test for it.
Add META_ADD configuration prerequisites - looks simple enough. It doesn't have any immediate impact though (1.19_01).
schema change test fixes (also, rt://30502) - needs to be investigated but should be simple to replicate on Windows (1.19_07)
segfault in t/07busy.t - "fixed" from DBD::SQLite::Amalgamation. At least, the change makes the segfault disappear for me (1.19_05)
Bus error on unique key conflict, needs test (test added in 1.19_07)
`make test` fails if $ENV{DBI_DSN} is set, needs test/check (1.19_06)

Things that need deeper looking into, or more tests, or are possibly upstream bugs:

bug in SQLite pre 3.5.9?
Bus error t/08create_function test, needs investigation, worksforme
Test suite (t/t50*.t) is order dependent, by me - need retriage (since 1.19_05 likely)

Things that won't get fixed or are not reproducible:

DBD::SQLite leaks file descriptors, bad description, no tests, retriage
dot doesn't work in quoted column aliases (1.19_08)
Bind parameter in GROUP BY fails (wontfix, as designed)

Even the bugs in the last category are worth working on, if you're interested in such things. For example tracking down in the SQLite changelog whether there was a fix for quoted column aliases with dots in their name.

If you want to adopt a bug from the RT queue or add in some feature of SQLite that is missing access from Perl, it's quite likely that Adam will give you a commit bit to http://svn.ali.as, which is where the current development incarnation of DBD::SQLite lives. And even if you don't want or think you can't actively contribute, consider this node as a heads up on what changes might come with the next release of DBD::SQLite.

Comment on Working on a new DBD::SQLite release Download Code

Replies are listed 'Best First'.
Re: Working on a new DBD::SQLite release by andreas1234567 (Vicar) on Apr 03, 2009 at 12:04 UTC
(I would like to discuss this issue here before updating the bug on rt.cpan.org) 'make test' fails when DBI_DSN is not DBD::SQLite: Does it make sense to honor `DBI_DSN` during make test? Does it make sense to honor `DBI_DSN` during make test if it doesn't match `/^DBI:SQLite/`. I have my `DBI_DSN` often set to `DBI:DB2:instance_name`, which means 'make test' will fail if it is honored. I don't know if `DBI_DSN` is commonly used. In case 1. a fix would be simply `$ diff t/lib.pl.cp t/lib.pl 36,37c36 < $test_dsn = $ENV{'DBI_DSN'} < \|\| "DBI:$dbdriver:dbname=$table_dir"; --- > $test_dsn = "DBI:$dbdriver:dbname=$table_dir";` [download] In case 2. a fix could be `$ diff t/lib.pl.cp t/lib.pl 36c36 < $test_dsn = $ENV{'DBI_DSN'} --- > $test_dsn = $ENV{'DBI_DSN'} && $ENV{'DBI_DSN'} =~ m/^DBI:$mdriv +er/` [download] -- No matter how great and destructive your problems may seem now, remember, you've probably only seen the tip of them. [1]	[reply] [d/l] [select]
Re^2: Working on a new DBD::SQLite release by Corion (Patriarch) on Apr 03, 2009 at 14:32 UTC
I don't see much reason to honor `$ENV{DBI_DSN}` at all. Case 2 would at leave others the possibility to change which database tests are run against, but as at least some of the tests are destructive, I consider the potential risk to be far higher than the potential gains in running the SQLite tests. So unless somebody comes up with a convincing argument when `$ENV{DBI_DSN}` can be of help during testing a (potentially unstable or disastrous) new version of DBD::SQLite, it'll go out. Personally, I'm even in favor of trying `:memory:` databases instead of using temporary files, because a memory database leaves nothing behind if the test crashes. Update: In fact, after looking at the code, that part has already been removed, even though the blamelog doesn't tell me exactly where.	[reply] [d/l] [select]
Re^3: Working on a new DBD::SQLite release by andreas1234567 (Vicar) on Apr 03, 2009 at 19:45 UTC
Yes, it seems to be fixed in trunk. -- No matter how great and destructive your problems may seem now, remember, you've probably only seen the tip of them. [1]	[reply]