Mailing List Archive

Some notes on porting to MySQL
I have been working on porting the database creation SQL scripts to
MySQL 5.x (with a few delays caused by other projects). So far, I've
completed awl/dba/awl-tables.sql and awl/dba/schema-management.sql, and
am currently working on davical/dba/davical.sql. The biggest issues
I've encountered that could affect functionality are the limitations of
using TEXT columns with indexes and no built-in mechanisms for storing
time zone information. Here are my notes on the issues I've encountered
so far:

1. MySQL doesn't support storing time zone information in TIMESTAMP,
DATETIME, and DATE data types, and the TIMESTAMPTZ data type doesn't
exist. This affects all table columns declared as TIMESTAMPTZ or
TIMESTAMP WITH TIME ZONE. I haven't dug through the source code yet to
see how the time zone information is used.

2. MySQL requires special handling for TEXT columns used in indexes,
which includes primary key columns and UNIQUE constraints. The total
size of the index keys must be 1000 bytes or less (thats bytes, not
characters). For TEXT keys, a size must be specified in the PRIMARY KEY
or UNIQUE statements. This size specifies how much of the beginning of
each string is used to build the index. For example, using
"UNIQUE(user_no,dav_name(200))" in the collection table definition
(davical/dba/davical.sql) would mean that only the first 200 characters
in the dav_name column are considered to be unique.

TEXT columns are very convenient because they can store arbitrarily long
strings. However, they also have a significant impact on query
performance, especially when used as index keys. VARCHAR strings also
provide the variable-length storage benefits of TEXT strings and can be
initialized with DEFAULT values (see #8, below), but do have the cost of
a fixed maximum length. My suggestion is to use VARCHAR strings in
place of TEXT wherever the columns are used as index keys.

3. MySQL does not allow stored functions to return VOID. For example,
see the new_db_revision() function declared in
awl/dba/schema-management.sql. I changed it to return the calculated
new_id value (i.e. INT).

4. MySQL doesn't have a mechanism similar to the RAISE EXCEPTION
statement in PostgreSQL. There is a way to emulate it by trying to
execute a prepared statement containing an invalid SQL statement. This
generates an error with the same transaction rollback affects as the
RAISE EXCEPTION statement.

5. MySQL doesn't support the ARRAY data type. It is used for the
parameter to the privilege_to_bits() stored function in
davical/dba/caldav_functions.sql, which is only called from within
davical/dba/base-data.sql during the table data initialization. The PHP
source code uses its own equivalent PHP function. I haven't tackled how
to handle this one yet.

6. MySQL doesn't support PostgreSQL sequences and sequence manipulation
functions. This is easily emulated.

7. DEFAULT values in CREATE TABLE statements must be constants in MySQL
(i.e. no function calls allowed). This is fixed by adding the function
calls to the MySQL versions of INSERT statements that otherwise rely
upon the default values in the PostgreSQL versions.

8. MySQL doesn't allow DEFAULT values for TEXT columns in CREATE TABLE
statements. This is fixed the same way as #7, or by changing the data
type to a VARCHAR string.

Thoughts/Comments?

//Jeff
Some notes on porting to MySQL [ In reply to ]
On Thu, 2010-09-09 at 19:40 -0700, Jeff Kintscher wrote:
> I have been working on porting the database creation SQL scripts to
> MySQL 5.x (with a few delays caused by other projects). So far, I've
> completed awl/dba/awl-tables.sql and awl/dba/schema-management.sql, and
> am currently working on davical/dba/davical.sql.

Thanks for taking a look at this :-)


> The biggest issues
> I've encountered that could affect functionality are the limitations of
> using TEXT columns with indexes and no built-in mechanisms for storing
> time zone information. Here are my notes on the issues I've encountered
> so far:
>
> 1. MySQL doesn't support storing time zone information in TIMESTAMP,
> DATETIME, and DATE data types, and the TIMESTAMPTZ data type doesn't
> exist. This affects all table columns declared as TIMESTAMPTZ or
> TIMESTAMP WITH TIME ZONE. I haven't dug through the source code yet to
> see how the time zone information is used.

It's likely that I will need to add columns for timezone storage and
move away from PostgreSQL internal timezone handling also. DAViCal's
timezone handling is becoming more sophisticated, and even what
PostgreSQL does, which is pretty good, seems not to be flexible enough.

In particular, PostgreSQL doesn't let me store floating time in a
TIMESTAMP WITH TIME ZONE field, and it doesn't let me introspect the
actual timezone after I put the value in the field - it only lets me
retrieve the offset from UTC, which is not so much help when calculating
the UTC time for something that occurs 6 weeks later...

I expect to attack this after I release 0.9.9.1.


> 2. MySQL requires special handling for TEXT columns used in indexes,
> which includes primary key columns and UNIQUE constraints. The total
> size of the index keys must be 1000 bytes or less (thats bytes, not
> characters). For TEXT keys, a size must be specified in the PRIMARY KEY
> or UNIQUE statements. This size specifies how much of the beginning of
> each string is used to build the index. For example, using
> "UNIQUE(user_no,dav_name(200))" in the collection table definition
> (davical/dba/davical.sql) would mean that only the first 200 characters
> in the dav_name column are considered to be unique.
>
> TEXT columns are very convenient because they can store arbitrarily long
> strings. However, they also have a significant impact on query
> performance, especially when used as index keys. VARCHAR strings also
> provide the variable-length storage benefits of TEXT strings and can be
> initialized with DEFAULT values (see #8, below), but do have the cost of
> a fixed maximum length. My suggestion is to use VARCHAR strings in
> place of TEXT wherever the columns are used as index keys.

In PostgreSQL VARCHAR is a syntax for TEXT, although VARCHAR(nnn)
differs a little, adding a constraint on input value length. I don't
believe that anything indexed is expected to be very long, so it
shouldn't be aby issue to switch those columns to VARCHAR in the
database create SQL.

Send me a patch... :-)


> 3. MySQL does not allow stored functions to return VOID. For example,
> see the new_db_revision() function declared in
> awl/dba/schema-management.sql. I changed it to return the calculated
> new_id value (i.e. INT).

No problem. A patch will be fine here also.


> 4. MySQL doesn't have a mechanism similar to the RAISE EXCEPTION
> statement in PostgreSQL. There is a way to emulate it by trying to
> execute a prepared statement containing an invalid SQL statement. This
> generates an error with the same transaction rollback affects as the
> RAISE EXCEPTION statement.

I don't *know* that I depend on the RAISE EXCEPTION functionality much.
But I don't know that I don't. I suspect it can be worked around on a
case by case basis.


> 5. MySQL doesn't support the ARRAY data type. It is used for the
> parameter to the privilege_to_bits() stored function in
> davical/dba/caldav_functions.sql, which is only called from within
> davical/dba/base-data.sql during the table data initialization. The PHP
> source code uses its own equivalent PHP function. I haven't tackled how
> to handle this one yet.

Given the existing PHP function we can quite probably do without this.
In base-data.sql it's really just used in order to document what is
going on, but I can imagine a slightly more laborious SQL statement
which would document it just as well if the function only accepted a
single parameter.


> 6. MySQL doesn't support PostgreSQL sequences and sequence manipulation
> functions. This is easily emulated.

The biggest issue is the place where one sequence is used for a whole
bunch of tables. Still, as you say it should be doable.


> 7. DEFAULT values in CREATE TABLE statements must be constants in MySQL
> (i.e. no function calls allowed). This is fixed by adding the function
> calls to the MySQL versions of INSERT statements that otherwise rely
> upon the default values in the PostgreSQL versions.

Annoying.


> 8. MySQL doesn't allow DEFAULT values for TEXT columns in CREATE TABLE
> statements. This is fixed the same way as #7, or by changing the data
> type to a VARCHAR string.

I can't imagine that any fields with a default other than the empty
string would be


You need to maybe also look into dba/views - I don't know what MySQL
would do about that kind of thing.

You'll be pleased to know that I'm moving away from the
rrule_functions.sql also :-)


The best thing is to send me small concise patches that I will not have
to spend a lot of time reviewing. It sounds like there are a variety of
things that could be done relatively easily to make the job simpler and
with no downside, so lets at least get them out of the way.


Cheers,
Andrew.

--
------------------------------------------------------------------------
andrew (AT) morphoss (DOT) com +64(272)DEBIAN
Please remain seated until the ride has come to a complete stop.
------------------------------------------------------------------------