Mailing List Archive: simple indexed file module?

simple indexed file module?

Apr 12, 1999, 10:08 AM

Post #1 of 6 (1647 views)

For a CGI script I'm working on, I need to keep a couple of indexed
files. These will contain variable-length data (descriptions), which
may be changed fairly frequently. So I imagine that in addition to the
data file, there will be an index file that keeps track of the position
and length of each record. New or expanded records will be stuffed in
wherever there is free space, or at the end if no sufficient free chunk
is available.

Is there already a module out there that does this? I know I could use
some sort of SQL interface, but I don't want to take up space on my web
site host for such a database engine. For this application, a little
Python implementation should work fine (and besides, that way I can
test it in the comfort of my Mac). Anybody have such a beast?

Thanks,
-- Joe

--
,------------------------------------------------------------------.
| Joseph J. Strout Biocomputing -- The Salk Institute |
| joe@strout.net http://www.strout.net |
`------------------------------------------------------------------'
Check out the Mac Web Directory! http://www.strout.net/macweb.cgi

simple indexed file module? [ In reply to ]

mspal at sangria

Apr 12, 1999, 8:47 PM

Post #2 of 6 (1599 views)

I'll suggest the obvious - Aaron Watters' Gadfly. It may be a bit of an
overkill, but it is python, and not very big either.

M.

simple indexed file module? [ In reply to ]

Jack.Jansen at oratrix

Apr 13, 1999, 3:05 AM

Post #3 of 6 (1603 views)

Joe Strout wrote:
>
> For a CGI script I'm working on, I need to keep a couple of indexed
> files. These will contain variable-length data (descriptions), which
> may be changed fairly frequently. So I imagine that in addition to the
> data file, there will be an index file that keeps track of the position
> and length of each record. New or expanded records will be stuffed in
> wherever there is free space, or at the end if no sufficient free chunk
> is available.

anydbm is probably the simplest here. Works on the Mac too, where gdbm is
available.

--
Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

simple indexed file module? [ In reply to ]

aaron_watters at my-dejanews

Apr 13, 1999, 6:15 AM

Post #4 of 6 (1610 views)

In article <joe-120419991008353558@chinacat.salk.edu>,
Joe Strout <joe@strout.net> wrote:
> For a CGI script I'm working on, I need to keep a couple of indexed
> files. These will contain variable-length data (descriptions), which
> may be changed fairly frequently. So I imagine that in addition to the
> data file, there will be an index file that keeps track of the position
> and length of each record. New or expanded records will be stuffed in
> wherever there is free space, or at the end if no sufficient free chunk
> is available.

If the data is small I'd say use an in memory approach with
load and store using marshal. If the data is large enough that
the time to load/unmarshal/marshal/store becomes an issue (probably
on the order of megs, I think) then use some sort of dbm implementation.
bplustree.py will work, but it doesn't do the kind of garbage
management you ask for (you'd have to do that periodically offline).
I haven't tested it on a mac, but it should require few if any
changes.

http://starship.skyport.net/crew/aaron_watters/bplustree/

also have a look at

http://www.chordate.com/gadfly.html

-- Aaron Watters

===
You can express emotion in many ways
and with varying degrees of loudness.
-- from my recently purchased French vocab tape.

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/ Search, Read, Discuss, or Start Your Own

simple indexed file module? [ In reply to ]

Apr 13, 1999, 11:13 AM

Post #5 of 6 (1606 views)

On Tue, 13 Apr 1999 aaron_watters@my-dejanews.com wrote:

> In article <joe-120419991008353558@chinacat.salk.edu>,
> Joe Strout <joe@strout.net> wrote:
> > For a CGI script I'm working on, I need to keep a couple of indexed
> > files. These will contain variable-length data (descriptions), which
> > may be changed fairly frequently. So I imagine that in addition to the
> > data file, there will be an index file that keeps track of the position
> > and length of each record. New or expanded records will be stuffed in
> > wherever there is free space, or at the end if no sufficient free chunk
> > is available.
>

I would suggest you use BoboPOS, the Object database used in Zope and
available as a seperate component. It supports transactions, multiple
versions and undo plus a lot of other cool things. It is very easy to use
and very stable.

Pavlos

simple indexed file module? [ In reply to ]

bhowes at cssun3

Apr 14, 1999, 10:02 AM

Post #6 of 6 (1603 views)

>>>> 'Joe Strout (joe@strout.net)' asked the following:

JS> For a CGI script I'm working on, I need to keep a couple of indexed
JS> files.[snip]

I have good luck with the `anydbm' module. This gives you keyed indexing
into a data file. It attempts to import dbhash, gdbm, dbm, or dumbdbm as
a backend. On my Windows/NT box, there is dbhash. I've used gdbm on
Unix. Not sure what comes with the Macintosh install.

Brad

--
Brad Howes bhowes@motorola.com
Principal Compass Hacker Work: +1 602 446 5219
Motorola Cell: +1 602 768 0735