Re^6: Using the unicode61 tokenizer in DBD::SQLite

I got 1.41_03 installed using the cpan shell with install ISHIGAKI/DBD-SQLite-1.41_03.tar.gz.
Still no dice:

use strict;
use warnings;

use DBI;

my $dbh = DBI->connect( "dbi:SQLite:dbname=test.db",
                            "",
                            "", {
                                # RaiseError => 1,        # we'll hand
+le the errors ourselves
                                sqlite_unicode => 1,
                            }
                        ) or die "Can't connect to database: $DBI::err
+str";



my $sqliteversion = $dbh->{sqlite_version}; print "\nSQLite version: $
+sqliteversion\n";

$dbh->do("DROP TABLE IF EXISTS data") or print "drop table failed";



$dbh->do("CREATE VIRTUAL TABLE data USING fts4 (col1 TEXT, col2 TEXT, 
+tokenize=unicode61)") or print "failed to create fts4 table";
[download]

Output:

SQLite version: 3.8.0.2
DBD::SQLite::db do failed: unknown tokenizer: unicode61 at e:\uni612.p
+l line 22.

failed to create fts4 table
[download]

Comment on Re^6: Using the unicode61 tokenizer in DBD::SQLite Select or Download Code

Replies are listed 'Best First'.
Re^7: Using the unicode61 tokenizer in DBD::SQLite by Corion (Patriarch) on Jan 08, 2014 at 08:18 UTC
To expand a bit on the steps Anonymous Monk gave you, the steps would be: Start the CPAN shell: `cpan` [download] Download and unpack the DBD::SQLite distribution: `look DBD::SQLite` [download] Recompile DBD::SQLite with the options that Anonymous Monk proposed: `perl Makefile.PL DEFINE=-DSQLITE_ENABLE_FTS4_UNICODE61` [download] Watch the output for errors. Run the tests `make test` [download] Verify that Unicode now works in the way you expect it to. Install the module if all tests pass and things work for you `make install` [download]	[reply] [d/l] [select]
Re^8: Using the unicode61 tokenizer in DBD::SQLite by elef (Friar) on Jan 08, 2014 at 10:05 UTC
Thanks, Corion. I saw that you are one of the maintainers. Any plans to enable the unicode61 tokenizer in future releases?	[reply]
Re^9: Using the unicode61 tokenizer in DBD::SQLite by Corion (Patriarch) on Jan 08, 2014 at 10:12 UTC
I only contributed some minor fixes to SQLite, but I'm not a real maintainer. I don't follow the development of DBD::SQLite closely, so I don't know the implications of always enabling the tokenizer. If there are no side effects other than compiled code for a feature that is likely to be unused, I don't think the real maintainers will object. I would write a mail to Kenichi Ishigaki with the proposed change and the idea why this change is harmless. Also, the changed Makefile.PL resp. the patch against the current Makefile.PL is more likely to get your changes into the next release.	[reply]
Re^7: Using the unicode61 tokenizer in DBD::SQLite by Anonymous Monk on Jan 08, 2014 at 08:11 UTC
I got 1.41_03 installed using Previously I said I think recompile with perl Makefile.PL DEFINE=-DSQLITE_ENABLE_FTS4_UNICODE61* and give it a shot* , so using the way you did doesn't magically give Makefile.PL thos extra arguments, so I'm not surprised nothing changed for you	[reply]
Re^8: Using the unicode61 tokenizer in DBD::SQLite by elef (Friar) on Jan 08, 2014 at 10:13 UTC
Well, if you read back your post you'll see that it wasn't clear at all, which is why I asked if it works out of the box in 1.41_03. By 'out of the box' I meant 'out of the box'. It wouldn't have been that surprising as 1.41_03 has a newer version of sqlite than 1.40 (3.8.x as opposed to 3.7.x).	[reply]
Re^9: Using the unicode61 tokenizer in DBD::SQLite by Anonymous Monk on Jan 08, 2014 at 10:42 UTC
Well, if you read back your post you'll see that it wasn't clear at all, which is why I asked if it works out of the box in Well, it seems clear enough to me in Re: Using the unicode61 tokenizer in DBD::SQLite where I said you need -DEFINES When you asked if it works out of the box in Re^4: Using the unicode61 tokenizer in DBD::SQLite, I didn't answer the question at all ( Re^5: Using the unicode61 tokenizer in DBD::SQLite ), I basically said figure it out by reading what I said and trying stuff but I did answer to walto ( Re^4: Using the unicode61 tokenizer in DBD::SQLite), basically no it doesn't work out of the box because it requires -DEFINES Both answer also seem clear enough to me, maybe a bit short and non-repetitive ... what can I say I'm not chatty	[reply]