Page 1 of 1
Best way to keep a dynamic (large) list of "stuff"
Posted: Thu Mar 24, 2011 8:18 pm
by FishNiX
If I had the need to keep a (potentially) large list (10k - 100k) of relatively short (300 characters-ish) strings what's the best way to do that in a plugin? Is there a "standard"/"best practices" way?
Text file -- slow and disk intensive
DB -- but where/how?
BerkleyDB -- more complexity
Memcache -- even more complexity
Thanks!
Re: Best way to keep a dynamic (large) list of "stuff"
Posted: Thu Mar 24, 2011 8:35 pm
by FishNiX
BTW -- I'm planning to implement a couple of these, but what should be canonical?
Re: Best way to keep a dynamic (large) list of "stuff"
Posted: Thu Mar 24, 2011 9:37 pm
by garvinhicking
Hi!
Good point. I think if you need lookup features, i.e. fetch certain strings depending on conditions, the DB is the only proper way to deal with. Make the plugin create its own database (see staticpag eplugin, look for "setup" method and how its called, andhow it uses "CREATE TABLE..." sql strings.
BerkeleyDB and memcache have too many dependencies so they should be avoided by default and maybe only added as an option.
I would use a textfile, if you always load the full set of data/strings though. This is faster than passing data from the SQL server to your application through a socket.
Regards,
Garvin
Re: Best way to keep a dynamic (large) list of "stuff"
Posted: Fri Mar 25, 2011 1:58 pm
by FishNiX
Yeah -- I'm thinking at some point in the future I will want to query the list one way or another. I'll shoot for the DB. Thanks!