Re: string arrays

Can you please tell us why you want to store 50 - 100 million variable length strings into an array in the first place? Is it for sorting purpose? Or is it because you want to find a particular element from the array? There are lots of algorithms to choose from depending on the problem dealing with. Perhaps you should give a bit of overview on the type of problem you are trying to solve.

In general, you could however store your strings in a file or database as suggested by davido and Zaxo, depending on your application of cause.

Personally I would favour the direct file approach, database insertion is a relatively expensive operation. Unless of cause if you use commands like 'bcp' in Sybase (just an example) that does native database import that is very fast.

Comment on Re: string arrays

Replies are listed 'Best First'.
Re: Re: string arrays by tachyon (Chancellor) on Oct 15, 2003 at 03:32 UTC
Fast native C import routines are pretty standard with any halfway decent RDBMS. `MySQL LOAD DATA INFILE '/tmp/blah.txt' INTO TABLE mytable FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'; MS SQL BULK INSERT [table name] FROM [filename to insert] WITH (FIELDTERMINATOR = '\t', FIRSTROW=2, ROWTERMINATOR = '\n') Oracle Two stage control file insert methadology similar to above` [download] I would make the rash presumption that the user already has the data in a (text) file (hardly likely to type in 100 million strings by hand as it would take literally years) and wants to manipulate them in some way. A RDBMS is probably the best solution. Multiple gigabytes of RAM is another possibility. A text file would seem the have rather dubious utility..... We use big RAM or a DB depending on the task. We have one data munging widget that takes several hours to run and consumes up to 4 GB of RAM. The same algorithm tied to disk to save memory took weeks to run. In this case the extra speed of in memory processing easily offets the RAM cost. cheers tachyon s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print	[reply] [d/l]

Replies are listed 'Best First'.

Re: Re: string arrays
by tachyon (Chancellor) on Oct 15, 2003 at 03:32 UTC

Fast native C import routines are pretty standard with any halfway decent RDBMS.

MySQL
LOAD DATA INFILE '/tmp/blah.txt' INTO TABLE mytable 
FIELDS TERMINATED BY '\t' 
LINES TERMINATED BY '\n';

MS SQL
BULK INSERT [table name] FROM [filename to insert] 
WITH (FIELDTERMINATOR = '\t',
FIRSTROW=2,
ROWTERMINATOR = '\n')

Oracle
Two stage control file insert methadology similar to above
[download]

I would make the rash presumption that the user already has the data in a (text) file (hardly likely to type in 100 million strings by hand as it would take literally years) and wants to manipulate them in some way. A RDBMS is probably the best solution. Multiple gigabytes of RAM is another possibility. A text file would seem the have rather dubious utility.....

We use big RAM or a DB depending on the task. We have one data munging widget that takes several hours to run and consumes up to 4 GB of RAM. The same algorithm tied to disk to save memory took weeks to run. In this case the extra speed of in memory processing easily offets the RAM cost.

cheers

tachyon

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

[reply]
[d/l]