Mailing List Archive

Newbie, need info on importing 500,000+ items
I'm a webmaster for a large bookstore and want to upgrade our database system (speed). I've read about the minivend system though
never had a change to get it.

I seen the demo allows you to add 'Single' items, however I have nearly half a million items currently in a flatfile database
Pipe(|) delimited. (I can have it delimited with any char. if needed)

I looked though all the archives but didn't find much that could help me. (mostly excel export problems)

When this question is answered (and no problems are foreseen) I'll go ahead and
install the software (I don't want to install then uninstall if it is not the solution I need)

Ken Lyons
CIS-Tech
Webmaster@ChristianBookServices.com
Newbie, need info on importing 500,000+ items [ In reply to ]
On 15 Jan, Company InterNet Services wrote:
> I'm a webmaster for a large bookstore and want to upgrade our database system (speed). I've read about the minivend system though
> never had a change to get it.
>
> I seen the demo allows you to add 'Single' items, however I have nearly half a million items currently in a flatfile database
> Pipe(|) delimited. (I can have it delimited with any char. if needed)
>
> I looked though all the archives but didn't find much that could help me. (mostly excel export problems)
>
> When this question is answered (and no problems are foreseen) I'll go ahead and
> install the software (I don't want to install then uninstall if it is not the solution I need)
>
> Ken Lyons
> CIS-Tech
> Webmaster@ChristianBookServices.com

From the doc (old Interchange Version 4.5)
-----snip-----
Interchange Database Operation

NOTE: In the following descriptions, we will use the following terms somewhat interchangeably:

key, code

Either one is a reference to the key for the database. In Interchange this is often the product code or SKU, which is the part number for the product.
Other key values may be used to generate relationships to other database tables.

It is required that the key be the first column of an ASCII source file for GDBM, Berkeley DB, or in-memory built-in database formats. It is also strongly
suggested that you keep that practice for SQL databases, since Interchange's import, export, and search facilities will work much better with that practice.
field, column

This is a column of the database. One of the columns is always the key -- Interchange prefers that the key be the first column. Field is an
interchangeable reference.
table, database

A table in the database. Because of the evolution of Interchange from a single-table database to an access method for an unlimited number of tables (and
databases, for that matter), we will sometimes refer to a table as a database. The only time database refers to something different is when describing that
concept as it relates to SQL -- where a database contains a series of tables. Interchange cannot create SQL databases, but given the proper
permissions it may drop and create tables within that database.

If necessary, Interchange reads the data to place in tables from standard ASCII-delimited files. All of these ASCII source files are kept in the products directory,
normally products in the catalog directory (where catalog.cfg is).

The ASCII files can have ^M (carriage return) characters if desired, but must have a newline character at the end of the line to work -- Mac users uploading
files must use ASCII mode, not binary mode!

Interchange's default ASCII delimiter is TAB.

IMPORTANT NOTE: The items must be separated by a single delimiter. The items are lined up for your reading convenience.

TAB

Fields separated by ^I characters. No whitespace should be at the beginning of the line.

code description price image
SH543 Men's fine cotton shirt 14.95 shirts.jpg

PIPE

Fields separated by | characters. No whitespace should be at the beginning of the line.

code|description|price|image
SH543|Men's fine cotton shirt|14.95|shirts.jpg

CSV

Fields enclosed in quotes, separated by commas. No whitespace should be at the beginning of the line.

"code","description","price","image"
"SH543","Men's fine cotton shirt","14.95","shirts.jpg"

NOTE: Using the default TAB delimiter is highly recommended if you are planning on searching the ASCII source file of the database. PIPE works fairly well, but
CSV delimiter schemes cause problems with searching.
-----snip-----

Joachim

--
Hans-Joachim Leidinger
buch online jojo@buchonline.net
Munscheidstr. 14 FAX: +49 209 1671441
45886 Gelsenkirchen FAX: 0209 1671441
Newbie, need info on importing 500,000+ items [ In reply to ]
On Mon, 15 Jan 2001, Company InterNet Services wrote:

> I'm a webmaster for a large bookstore and want to upgrade our database system
> (speed). I've read about the minivend system though never had a change
> to get it.
>
> I seen the demo allows you to add 'Single' items, however I have nearly half
> a million items currently in a flatfile database Pipe(|) delimited. (I
> can have it delimited with any char. if needed)
>
> I looked though all the archives but didn't find much that could help me.
> (mostly excel export problems)
>
> When this question is answered (and no problems are foreseen) I'll go ahead
> and install the software (I don't want to install then uninstall if it
> is not the solution I need)
>
> Ken Lyons
> CIS-Tech
> Webmaster@ChristianBookServices.com
>
>

The import/export shouldn't be the problem. But how to edit a single
record with the new IC admin UI, is not yet clear to me.

There is one answer of Mike Heins with regards to large product tables
which gives a hint.

Mike Heins wrote (Jan. 10th, Subject line: [ic]/[mv] Large Product tables

> It is easy to shoot yourself in the foot with any application and a
> large database. I know, I have done it many times. Interchange's admin
> UI is optimized for DBs that are 50,000 records or less. If you have
> that many records, you need SQL to use the IC admin tool with any
> success. Even then, you have to make sure that proper indexing is done
> on all fields that are selected/sorted on.
>
> [snip]
>
> The Interchange application itself will not repeal the laws of physics,
> but if used in a reasonable manner it is capable of working with very
> large database tables. There are sites which have as many as a million
> items, and at least one that has 500,000 and uses GDBM. They don't try
> and run the item editor on that, though! And they use external
> import/export programs and don't try to do auto-import and export.

He gave this answer with regards to product tables of 3 million records.
There is no problem to run a GDBM table with 500 000 60 800 000 records,
when you index on all fields you want to search for, use binary searches
and use the static page build feature in addition.

With Mike's answer I am still not clear, if, in order to use IC's new
admin UI and be able to edit single records, he advises to split
a 500 000 records product table in 10 separated 50 000 record tables
and go from there.

I wonder if there is are bookstores using the older MV releases who have
already upgraded to IC and use its admin UI. Would be nice to hear from
them about their experience.

Birgitt