in reply to Using the unicode61 tokenizer in DBD::SQLite

I just updated DBD::SQLite to 1.40 and made sure I have SQLite version 3.7.17.

But what does DBD::SQLite have, what is sqlite_version? update: I see you got it #define SQLITE_VERSION "3.7.17"

Was SQLite compiled without enabling the unicode61 tokenizer? (Some sources mention compiling sqlite with SQLITE_ENABLE_FTS4_UNICODE61 in order to get this functionality.)

I see that that conditional is in the source /* fts3_unicode2.c (functions generated by parsing unicode text files) */ #ifdef SQLITE_ENABLE_FTS4_UNICODE61 but since you get an error message, it probably wasn't compiled with that turned one ( nothing turned on in source)

Do I have any options here?

I think recompile with   perl Makefile.PL DEFINE=-DSQLITE_ENABLE_FTS4_UNICODE61 and give it a shot

update: I just recompiled, all tests passed except some in t/51_table_column_metadata.t , same with  cpanp -z ISHIGAKI/DBD-SQLite-1.41_03.tar.gz

same with  perl Makefile.PL "DEFINE=-DSQLITE_ENABLE_FTS4_UNICODE61 -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS "

Seems to work

$ perl -Mblib -S dbish dbi:SQLite:testgonernow
DBI::Shell 11.95 using DBI 1.628

WARNING: The DBI::Shell interface and functionality are
=======  very likely to change in subsequent versions!


Connecting to 'dbi:SQLite:testgonernow' as ''...
@dbi:SQLite:testgonernow> /table_info
TABLE_CAT,TABLE_SCHEM,TABLE_NAME,TABLE_TYPE,REMARKS,sqlite_sql
undef,'main','sqlite_master','SYSTEM TABLE',undef,undef
undef,'temp','sqlite_temp_master','SYSTEM TABLE',undef,undef
[2 rows of 6 fields returned]
@dbi:SQLite:testgonernow> CREATE VIRTUAL TABLE mytable USING fts4 (tok
+enize=unicode61);
[0E0 rows affected]
@dbi:SQLite:testgonernow> /table_info
TABLE_CAT,TABLE_SCHEM,TABLE_NAME,TABLE_TYPE,REMARKS,sqlite_sql
undef,'main','mytable_segdir','INDEX',undef,undef
undef,'main','sqlite_master','SYSTEM TABLE',undef,undef
undef,'temp','sqlite_temp_master','SYSTEM TABLE',undef,undef
undef,'main','mytable','TABLE',undef,'CREATE VIRTUAL TABLE mytable USI
+NG fts4 (tokenize=unicode61)'
undef,'main','mytable_content','TABLE',undef,'CREATE TABLE \'mytable_c
+ontent\'(docid INTEGER PRIMARY KEY, \'c0content\')
'
undef,'main','mytable_docsize','TABLE',undef,'CREATE TABLE \'mytable_d
+ocsize\'(docid INTEGER PRIMARY KEY, size BLOB)'
undef,'main','mytable_segdir','TABLE',undef,'CREATE TABLE \'mytable_se
+gdir\'(level INTEGER,idx INTEGER,start_block INTEG
ER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(le
+vel, idx))'
undef,'main','mytable_segments','TABLE',undef,'CREATE TABLE \'mytable_
+segments\'(blockid INTEGER PRIMARY KEY, block BLOB
)'
undef,'main','mytable_stat','TABLE',undef,'CREATE TABLE \'mytable_stat
+\'(id INTEGER PRIMARY KEY, value BLOB)'
[9 rows of 6 fields returned]
@dbi:SQLite:testgonernow> /exit
Disconnecting from dbi:SQLite:testgonernow.

$ rm testgonernow

[download]

Replies are listed 'Best First'.
Re^2: Using the unicode61 tokenizer in DBD::SQLite
by elef (Friar) on Jan 06, 2014 at 10:10 UTC
    Thanks, Anonymonk.
    I have no idea about any of the steps required for recompiling sqlite for DBD::SQLite. I don't even know where the executable or the source code is stored. Can you give me some pointers? I'm on Windows BTW.
[reply]
      Um, I'm on windows too .... if you don't recognize cpanp -z... perl Makefile... you probably want to read A Guide To Installing Modules ... among others
[reply]
        I seem to have missed that bit. Are you saying that the unicode61 tokenizer works out of the box in 1.41_03?
[reply]
        Yes, according to DBD::SQLite it should work out of the box.
[reply]

In Section Seekers of Perl Wisdom