Mailing List Archive

insert failing without raising exception: postgresql, psycopg
Problem:
Database inserts that worked previously are now failing,
without raising exceptions!

Request:
Help!

Production Environment:
zope-2.7.4, postgresql-7.4.3p2, psycopg-1.1.15, OpenBSD 3.7.
Duplicated under test environment:
zope 2.7.4, postgresql 8.0.3, psycopg-1.1.15, OpenBSD 3.8.

Details:

When I call the Z SQL method directly, with test data, the database
insert works as expected.

When I call the same method from a python script, the database insert
fails, without raising an exception. I can only tell the insert failed
by looking at the database and seeing that the data was not stored.

Though the insert is failing, a sequence column in the target table is
apparently being incremented by the failing insert. (I deduce this
from the fact that subsequent Z SQL test inserts show sequence gap
of 12 after each failed insert attempt, while no gap occurs when I
call the Z SQL method repeatedly via Zope's "test" tab.)

I will appreciate any suggestions as to how I might troubleshoot this.

Thanks!

- Jerome

psql output showing "gaps" in the sequence number field, corresponding
to the failed inserts:

water=> select meterserial as mser, readingserial as rser, readdate, reading from meterreads where meterserial = 572 and readdate >= '2005-09-01' order by readdate;
mser | rser | readdate | reading [annotation]
------+------+------------+---------
572 | 4382 | 2005-09-02 | 72100
572 | 4383 | 2005-09-09 | 72230
572 | 4384 | 2005-09-16 | 72366
572 | 4385 | 2005-09-23 | 72398
572 | 4386 | 2005-09-30 | 72513
572 | 5681 | 2005-10-01 | 72600
572 | 5726 | 2005-10-02 | 72602 [test before failed insert]
572 | 5738 | 2005-10-03 | 72603 [test after failed insert]
572 | 5739 | 2005-10-04 | 72604 [2nd 'test' in a row]
572 | 5751 | 2005-10-05 | 72605 [test after failed insert]
572 | 5752 | 2005-10-06 | 72606 [2nd 'test' in a row]
572 | 5764 | 2005-10-07 | 72607 [test after failed insert]
572 | 5776 | 2005-10-08 | 72608 [test after failed insert]
(13 rows)

example Z SQL test output:
insert into MeterReads (MeterSerial, ReadDate, Reading
)
values (572,
'2005-10-08',
72608
)
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: insert failing without raising exception: postgresql, psycopg [ In reply to ]
On 2005-11-22 at 12:27:42 [+0100], freeman@tllc.com wrote:
>
> Though the insert is failing, a sequence column in the target table is
> apparently being incremented by the failing insert. (I deduce this
> from the fact that subsequent Z SQL test inserts show sequence gap
> of 12 after each failed insert attempt, while no gap occurs when I
> call the Z SQL method repeatedly via Zope's "test" tab.)
>
> I will appreciate any suggestions as to how I might troubleshoot this.

You should probably compile psycopg to give debug information and I would
suggest updating to the most recent build in the 1 series. In addition you
might want to try using our mxODBCZopeDA to see if the error is with the
DA/driver or in Zope itself.

Charlie
--
Charlie Clark
eGenix.com

Professional Python Services directly from the Source
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2005-10-17: Released mxODBC.Zope.DA 1.0.9 http://zope.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: insert failing without raising exception: postgresql, psycopg [ In reply to ]
freeman@tllc.com wrote at 2005-11-22 03:27 -0800:
>Problem:
> Database inserts that worked previously are now failing,
> without raising exceptions!

I expect that something in your application catches exceptions.

To verify this, I would add logging to "ZPsycopgDA" (indeed,
I did in our version of it) and look at the logs.

--
Dieter
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: insert failing without raising exception: postgresql, psycopg [ In reply to ]
Dieter Maurer <dieter@handshake.de> wrote:

> freeman@tllc.com wrote at 2005-11-22 03:27 -0800:
> >Problem:
> > Database inserts that worked previously are now failing,
> > without raising exceptions!
>
> I expect that something in your application catches exceptions.
>
> To verify this, I would add logging to "ZPsycopgDA" (indeed,
> I did in our version of it) and look at the logs.

I regret that I have not yet learned how to "add logging" to
ZPsycopgDA, and will appreciate learning how to do so.

In the meantime, I've solved my immediate problem by learning
how to look at the postgresql logs (in /var/postgresql/logfile)

There I saw the postgresql error messages.

I then enabled logging of sql statements (temporarily), by
editing /var/postgresql/data/postgresql.conf, and learned
ust why postgresql was refusing to insert the data.

In the process I verified that insert exceptions are not
currently noticed by zope, as I have it configured, while
postgresql syntax errors are noticed by zope.

Maybe insert errors never were noticed by zope, and I
simply never noticed before...

- Jerome
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: insert failing without raising exception: postgresql, psycopg [ In reply to ]
On Tue, 22 Nov 2005 20:39:11 +0100
Dieter Maurer <dieter@handshake.de> wrote:

> freeman@tllc.com wrote at 2005-11-22 03:27 -0800:
> >Problem:
> > Database inserts that worked previously are now failing,
> > without raising exceptions!
>
> I expect that something in your application catches exceptions.
>
> To verify this, I would add logging to "ZPsycopgDA" (indeed,
> I did in our version of it) and look at the logs.
>
> --
> Dieter
>
>
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: insert failing without raising exception: postgresql, psycopg [ In reply to ]
freeman@tllc.com wrote at 2005-11-23 06:51 -0800:
> ...
>I regret that I have not yet learned how to "add logging" to
>ZPsycopgDA, and will appreciate learning how to do so.

It looks like this ("ZPsycopgDA/db.py"):

from zLOG import LOG, ERROR, INFO, PROBLEM
...

LOG('Postgres', INFO, summary, details)

> ...
>In the process I verified that insert exceptions are not
>currently noticed by zope, as I have it configured, while
>postgresql syntax errors are noticed by zope.

That would be a bug. Please contact the "psycopg" auther.

--
Dieter
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: insert failing without raising exception: postgresql, psycopg [ In reply to ]
Dieter Maurer wrote:
> freeman@tllc.com wrote at 2005-11-23 06:51 -0800:
>
>>...
>>I regret that I have not yet learned how to "add logging" to
>>ZPsycopgDA, and will appreciate learning how to do so.
>
>
> It looks like this ("ZPsycopgDA/db.py"):
>
> from zLOG import LOG, ERROR, INFO, PROBLEM
> ...
>
> LOG('Postgres', INFO, summary, details)

please don't encourage this old and ugly logging style...

import logging
logger = logging.getLogger('event.Postgres')

logger.info('Count is %i', count)

cheers,

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk

_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db