Mailing List Archive

Paul J Stevens wrote:

> First bug found with the testsuite I'm building using python
> (unittest+imaputil).
Nice!
>
>
> More trouble in _ic_list I think:
>
> as testuser1:
>
> A01 login "testuser1" "testpw"
> A01 create "testaclbox"
> A01 setacl "testaclbox" "testuser2" "slarw"
> A01 logout
>
> next :
>
> A02 login "testuser2" "testpw"
> A02 list "" *
>
> this now barfs with an internal dbase error when trying to select
> mailbox_idnr = 0; which doesn't exist at all.
>
>
>
> Trace:
>
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]: arg[1]: '*'
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]: IMAPClientHandler():
> Executing command list...
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]: ic_list(): build the
> pattern: [^.*$]
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]: dbmysql.c,db_query:
> executing query [.SELECT mbx.name, mbx.mailbox_idnr, mbx.owner_idnr FROM
> mailboxes mbx LEFT JOIN acl ON mbx.mailbox_idnr = acl.mailbox_id WHERE
> (acl.user_id = '14' AND acl.lookup_flag = '1') OR mbx.owner_idnr = '14']
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]: dbmysql.c,db_query:
> executing query [SELECT userid FROM users WHERE user_idnr = '13']
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]: authsql.c,auth_get_userid:
> query_result = testuser1
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]: authsql.c,auth_get_userid:
> returning testuser1 as returnid
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]:
> misc.c,mailbox_add_namespace: owner name = testuser1
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]:
> misc.c,mailbox_add_namespace: returning fully qualified name
> [#Users/testuser1/testaclbox]
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]:
> db.c,db_list_mailboxes_by_regex: comparing mailbox
> [#Users/testuser1/testaclbox] to regular expression
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]:
> db.c,db_list_mailboxes_by_regex: regex match #Users/testuser1/testaclbox
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]:
> db.c,db_list_mailboxes_by_regex: comparing mailbox [INBOX] to regular
> expression
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]:
> db.c,db_list_mailboxes_by_regex: regex match INBOX
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]:
> db.c,db_list_mailboxes_by_regex: comparing mailbox [testbox] to regular
> expression
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]:
> db.c,db_list_mailboxes_by_regex: regex match testbox
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]:
> db.c,db_findmailbox_by_regex: found [3] mailboxes
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]: dbmysql.c,db_query:
> executing query [SELECT owner_idnr FROM mailboxes WHERE mailbox_idnr = '0']
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]: dbmysql.c, db_get_result:
> row = 0, field = 0, bigger than size of result set
> Mar 23 14:45:21 tsuki dbmail/imap4d[17524]: db.c,db_getmailboxname:
> error checking ownership of mailbox

Strange... I haven't succeeded in reproducing this error.

Somehow an mboxid of 0 was returned instead of an mboxid of > 0

can you run the query above:
SELECT mbx.name, mbx.mailbox_idnr, mbx.owner_idnr FROM
mailboxes mbx LEFT JOIN acl ON mbx.mailbox_idnr = acl.mailbox_id WHERE
(acl.user_id = '14' AND acl.lookup_flag = '1') OR mbx.owner_idnr = '14'

and check what the output is?

I'm currently at a loss here, especially since I cannot reproduce the
problem.

Ilja

Ilja Booij wrote:
> Paul J Stevens wrote:
>
>> First bug found with the testsuite I'm building using python
>> (unittest+imaputil).

> Strange... I haven't succeeded in reproducing this error.
>
> Somehow an mboxid of 0 was returned instead of an mboxid of > 0

Indeed. And if I do a 'setacl "testaclbox" "testuser1" "slrw" ' in stead of a
"slraw", I get a segfault.


> SELECT mbx.name, mbx.mailbox_idnr, mbx.owner_idnr FROM
> mailboxes mbx LEFT JOIN acl ON mbx.mailbox_idnr = acl.mailbox_id WHERE
> (acl.user_id = '14' AND acl.lookup_flag = '1') OR mbx.owner_idnr = '14'

The result was clean.


> and check what the output is?
>
> I'm currently at a loss here, especially since I cannot reproduce the
> problem.

I'm attaching the test-script. Create two users:

dbmail-adduser a testuser1 '{md5:}test' 0 0 test@mydomain
dbmail-adduser a testuser2 '{md5:}test' 0 0 test@mydomain

on host 'mail' and run....



--
________________________________________________________________
Paul Stevens paul@nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands_______________________________________www.nfg.nl

Paul J Stevens wrote:
> Ilja Booij wrote:
>
>> Paul J Stevens wrote:
>>
>>> First bug found with the testsuite I'm building using python
>>> (unittest+imaputil).
>
>
>> Strange... I haven't succeeded in reproducing this error.
>>
>> Somehow an mboxid of 0 was returned instead of an mboxid of > 0
>
>
> Indeed. And if I do a 'setacl "testaclbox" "testuser1" "slrw" ' in stead
> of a "slraw", I get a segfault.

It does not crash here..

> The result was clean.
>
>
> I'm attaching the test-script. Create two users:
>
> dbmail-adduser a testuser1 '{md5:}test' 0 0 test@mydomain
> dbmail-adduser a testuser2 '{md5:}test' 0 0 test@mydomain
>
> on host 'mail' and run....

It runs just fine.. No problem whatsoever.

It's beginning to sound like a memory error to me. I'll try some checks
with valgrind. Can you send a log of the whole session up until the
point it segfaults?

BTW, great test script. We should have more of these for our daemons
(and dbmail-smtp). I love the way these things can be coded in Python:
concise and clear.

Does everyone agree that we should put this test-script (and possibly
other test-scripts) into the source tree? Automated testing will be very
useful.

Ilja

Just an update. I've run the same test on a pristine dbmail installation
in an UML image. Same result. I've tried various compile flags. No
change. Sometimes it segfaults, sometimes it fails with an internal
dbase error. Ive attached the dumped dbmail database (very small), and
session log.

Just login as testuser2 pw:test and issue a list command.


Ilja Booij wrote:
> Paul J Stevens wrote:
>
>> Ilja Booij wrote:
>>
>>> Paul J Stevens wrote:
>>>
>>>> First bug found with the testsuite I'm building using python
>>>> (unittest+imaputil).
>>
>>
>>
>>> Strange... I haven't succeeded in reproducing this error.
>>>
>>> Somehow an mboxid of 0 was returned instead of an mboxid of > 0
>>
>>
>>
>> Indeed. And if I do a 'setacl "testaclbox" "testuser1" "slrw" ' in
>> stead of a "slraw", I get a segfault.
>
>
> It does not crash here..
>
>> The result was clean.
>>
>>
>> I'm attaching the test-script. Create two users:
>>
>> dbmail-adduser a testuser1 '{md5:}test' 0 0 test@mydomain
>> dbmail-adduser a testuser2 '{md5:}test' 0 0 test@mydomain
>>
>> on host 'mail' and run....
>
>
> It runs just fine.. No problem whatsoever.
>
> It's beginning to sound like a memory error to me. I'll try some checks
> with valgrind. Can you send a log of the whole session up until the
> point it segfaults?
>
> BTW, great test script. We should have more of these for our daemons
> (and dbmail-smtp). I love the way these things can be coded in Python:
> concise and clear.
>
> Does everyone agree that we should put this test-script (and possibly
> other test-scripts) into the source tree? Automated testing will be very
> useful.
>
> Ilja
>
>
>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev@dbmail.org
> http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>

--
________________________________________________________________
Paul Stevens mailto:paul@nfg.nl
NET FACILITIES GROUP PGP: finger paul@nfg.nl
The Netherlands________________________________http://www.nfg.nl

Paul J Stevens wrote:
> Just an update. I've run the same test on a pristine dbmail installation
> in an UML image. Same result. I've tried various compile flags. No
> change. Sometimes it segfaults, sometimes it fails with an internal
> dbase error. Ive attached the dumped dbmail database (very small), and
> session log.
>
> Just login as testuser2 pw:test and issue a list command.
I've now done some testing on an OpenBSD machine, and finally managed
to reproduce the SIGPIPE error.. Have not been able to reproduce Paul's
crashes though..

I'm currently looking in to the SIGPIPE problem (the python test script
Paul wrote comes in very handy for this).

Ilja

Ilja Booij wrote:
> Paul J Stevens wrote:
>
>> Just an update. I've run the same test on a pristine dbmail
>> installation in an UML image. Same result. I've tried various compile
>> flags. No change. Sometimes it segfaults, sometimes it fails with an
>> internal dbase error. Ive attached the dumped dbmail database (very
>> small), and session log.
>>
>> Just login as testuser2 pw:test and issue a list command.
>
> I've now done some testing on an OpenBSD machine, and finally managed
> to reproduce the SIGPIPE error.. Have not been able to reproduce Paul's
> crashes though..
>
> I'm currently looking in to the SIGPIPE problem (the python test script
> Paul wrote comes in very handy for this).

The SIGPIPE problem is quite a weird one. I figured I'd find out more if
I put an "if" around the fgets() statement which get commands from the
client. When I do this, I first get a EINPROGRRESS error, after which I
get a lot of errors indicating that a buffer is full (ENOBUFS).

strange...

I tried applying Paul J Stevens' patch that added some error checking to
all write actions on streams, and it seemed to work better, but still
not completely alright. It especially messes up the last respone ('OK
Completed') sent to the client, by completely messing up the tag, which
is kind of strange, because printing the tag to trace works ok. This is
probably some memory error.

Anyway, I'm a bit confused at the moment..

Ilja

Ilja Booij wrote:
> Ilja Booij wrote:
>> Paul J Stevens wrote:
> The SIGPIPE problem is quite a weird one. I figured I'd find out more if
> I put an "if" around the fgets() statement which get commands from the
> client. When I do this, I first get a EINPROGRRESS error, after which I
> get a lot of errors indicating that a buffer is full (ENOBUFS).
>
> strange...
>
> I tried applying Paul J Stevens' patch that added some error checking to
> all write actions on streams, and it seemed to work better, but still
> not completely alright. It especially messes up the last respone ('OK
> Completed') sent to the client, by completely messing up the tag, which
> is kind of strange, because printing the tag to trace works ok. This is
> probably some memory error.
>
> Anyway, I'm a bit confused at the moment..

Aha, there was an error in the ci_write function, which called fprintf()
instead of vfprintf().

When using these functions, I'm getting no problems at all. In fact, the
new IMAP test suite, which Paul J Stevens sent yesterday runs without
finding an error.

Ilja

Ilja Booij wrote:
>
> Aha, there was an error in the ci_write function, which called fprintf()
> instead of vfprintf().

Oops. Sorry about that. I'm just too green as a C coder.

> When using these functions, I'm getting no problems at all. In fact, the
> new IMAP test suite, which Paul J Stevens sent yesterday runs without
> finding an error.

Which is not a good thing, per se, since all the message-handling calls
haven't been implemented yet. Still nice to see it's useful. I'll start
adding more code, to reflect all imap_handler_functions in
imapcommands.c. After that, all testcases should be fleshed out to
reflect both typical and a-typical usage.

On a sidenote: aren't you checking in changes, or are you working on a
branch ? I havent' seen any changes in HEAD since last week.


--
________________________________________________________________
Paul Stevens mailto:paul@nfg.nl
NET FACILITIES GROUP PGP: finger paul@nfg.nl
The Netherlands________________________________http://www.nfg.nl

Paul J Stevens wrote:
> Just an update. I've run the same test on a pristine dbmail installation
> in an UML image. Same result. I've tried various compile flags. No
> change. Sometimes it segfaults, sometimes it fails with an internal
> dbase error. Ive attached the dumped dbmail database (very small), and
> session log.

I thought I had found the possible answer, but as it turns out, I've
only found a bug in another function.

in db_listmailboxchildren() (used by IMAP DELETE and RENAME), the
mailbox_name string was used after it was freed.

There also was a memory leak in db_findmailbox_by_regex(), where the
regex_t struct was not freed.

Paul, I'm stilll looking into your problem. I've added a new debug
statement in _ic_list() in imapcommands.c, before the call to
db_getmailboxname().

Can you try to reproduce your error and show us the trace?

Ilja

Paul J Stevens wrote:
> Ilja Booij wrote:
>
>>
>> Aha, there was an error in the ci_write function, which called
>> fprintf() instead of vfprintf().
>
>
> Oops. Sorry about that. I'm just too green as a C coder.
:)
>
>> When using these functions, I'm getting no problems at all. In fact,
>> the new IMAP test suite, which Paul J Stevens sent yesterday runs
>> without finding an error.
>
>
> Which is not a good thing, per se, since all the message-handling calls
> haven't been implemented yet. Still nice to see it's useful. I'll start
> adding more code, to reflect all imap_handler_functions in
> imapcommands.c. After that, all testcases should be fleshed out to
> reflect both typical and a-typical usage.
>
> On a sidenote: aren't you checking in changes, or are you working on a
> branch ? I havent' seen any changes in HEAD since last week.
I've made a couple of checkins today. Most of the other time I've been
trying to find and fix bugs, but if I've fixed nothing, there's nothing
to check in :(

Anyway, please try to run with the newest code in HEAD, which should
possibly give a bit more info on what goes wrong in your crashes.

Ilja

Ilja Booij wrote:
> Paul J Stevens wrote:
>
>> Just an update. I've run the same test on a pristine dbmail
>> installation in an UML image. Same result. I've tried various compile
>> flags. No change. Sometimes it segfaults, sometimes it fails with an
>> internal dbase error. Ive attached the dumped dbmail database (very
>> small), and session log.
>
>
> I thought I had found the possible answer, but as it turns out, I've
> only found a bug in another function.
>
> in db_listmailboxchildren() (used by IMAP DELETE and RENAME), the
> mailbox_name string was used after it was freed.
>
> There also was a memory leak in db_findmailbox_by_regex(), where the
> regex_t struct was not freed.
>
> Paul, I'm stilll looking into your problem. I've added a new debug
> statement in _ic_list() in imapcommands.c, before the call to
> db_getmailboxname().
>
> Can you try to reproduce your error and show us the trace?

Did that. Same result: children[0] = 0 which it should be.


Mar 31 12:17:56 sarge dbmail/imap4d[19781]: IMAPClientHandler(): line
read for PID 19781
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: COMMAND: [A01 LIST "" *]
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: arg[0]: ''
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: arg[1]: '*'
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: IMAPClientHandler():
Executing command list...
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: ic_list(): build the
pattern: [^.*$]
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: dbmysql.c,db_query:
executing query [.SELECT mbx.name, mbx.mailbox_idnr, mbx.owner_idnr FROM
mailboxes mbx LEFT JOIN acl ON mbx.mailbox_idnr = acl.mailbox_id WHERE
(acl.user_id = '3' AND acl.lookup_flag = '1') OR mbx.owner_idnr = '3']
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: dbmysql.c,db_query:
executing query [SELECT userid FROM users WHERE user_idnr = '2']
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: authsql.c,auth_get_userid:
query_result = testuser1
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: authsql.c,auth_get_userid:
returning testuser1 as returnid
Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
misc.c,mailbox_add_namespace: owner name = testuser1
Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
misc.c,mailbox_add_namespace: returning fully qualified name
[#Users/testuser1/testbox]
Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
db.c,db_list_mailboxes_by_regex: comparing mailbox
[#Users/testuser1/testbox] to regular expressionMar 31 12:17:56 sarge
dbmail/imap4d[19781]: db.c,db_list_mailboxes_by_regex: regex match
#Users/testuser1/testbox
Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
db.c,db_list_mailboxes_by_regex: comparing mailbox [INBOX] to regular
expression
Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
db.c,db_list_mailboxes_by_regex: regex match INBOX
Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
db.c,db_list_mailboxes_by_regex: comparing mailbox [testbox] to regular
expression
Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
db.c,db_list_mailboxes_by_regex: regex match testbox
Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
db.c,db_findmailbox_by_regex: found [3] mailboxes
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: imapcommands.c,_ic_list:
children[0] = 0
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: dbmysql.c,db_query:
executing query [SELECT owner_idnr FROM mailboxes WHERE mailbox_idnr = '0']
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: dbmysql.c, db_get_result:
row = 0, field = 0, bigger than size of result set
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: db.c,db_getmailboxname:
error checking ownership of mailbox
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: IMAPClientHandler():
Finished command list
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: IMAPClientHandler(): Closing
connection for client from IP [172.16.3.2]
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: PerformChildTask(): client
handling complete, closing streams
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: PerformChildTask():
connection closed
Mar 31 12:17:56 sarge dbmail/imap4d[19781]: PerformChildTask(): waiting
for connection





--
________________________________________________________________
Paul Stevens mailto:paul@nfg.nl
NET FACILITIES GROUP PGP: finger paul@nfg.nl
The Netherlands________________________________http://www.nfg.nl

Paul J Stevens wrote:
> Did that. Same result: children[0] = 0 which it should be.

I've added some more debug stuff. I'm beginning to suspect that maybe
the db_get_result_u64() function does not return the right numbers.

can you update from cvs and try again?

Ilja

>
>
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: IMAPClientHandler(): line
> read for PID 19781
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: COMMAND: [A01 LIST "" *]
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: arg[0]: ''
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: arg[1]: '*'
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: IMAPClientHandler():
> Executing command list...
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: ic_list(): build the
> pattern: [^.*$]
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: dbmysql.c,db_query:
> executing query [.SELECT mbx.name, mbx.mailbox_idnr, mbx.owner_idnr FROM
> mailboxes mbx LEFT JOIN acl ON mbx.mailbox_idnr = acl.mailbox_id WHERE
> (acl.user_id = '3' AND acl.lookup_flag = '1') OR mbx.owner_idnr = '3']
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: dbmysql.c,db_query:
> executing query [SELECT userid FROM users WHERE user_idnr = '2']
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: authsql.c,auth_get_userid:
> query_result = testuser1
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: authsql.c,auth_get_userid:
> returning testuser1 as returnid
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
> misc.c,mailbox_add_namespace: owner name = testuser1
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
> misc.c,mailbox_add_namespace: returning fully qualified name
> [#Users/testuser1/testbox]
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
> db.c,db_list_mailboxes_by_regex: comparing mailbox
> [#Users/testuser1/testbox] to regular expressionMar 31 12:17:56 sarge
> dbmail/imap4d[19781]: db.c,db_list_mailboxes_by_regex: regex match
> #Users/testuser1/testbox
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
> db.c,db_list_mailboxes_by_regex: comparing mailbox [INBOX] to regular
> expression
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
> db.c,db_list_mailboxes_by_regex: regex match INBOX
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
> db.c,db_list_mailboxes_by_regex: comparing mailbox [testbox] to regular
> expression
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
> db.c,db_list_mailboxes_by_regex: regex match testbox
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
> db.c,db_findmailbox_by_regex: found [3] mailboxes
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: imapcommands.c,_ic_list:
> children[0] = 0
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: dbmysql.c,db_query:
> executing query [SELECT owner_idnr FROM mailboxes WHERE mailbox_idnr = '0']
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: dbmysql.c, db_get_result:
> row = 0, field = 0, bigger than size of result set
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: db.c,db_getmailboxname:
> error checking ownership of mailbox
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: IMAPClientHandler():
> Finished command list
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: IMAPClientHandler(): Closing
> connection for client from IP [172.16.3.2]
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: PerformChildTask(): client
> handling complete, closing streams
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: PerformChildTask():
> connection closed
> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: PerformChildTask(): waiting
> for connection
>
>
>
>
>

Ilja Booij wrote:
> Paul J Stevens wrote:
>
>> Did that. Same result: children[0] = 0 which it should be.
>
>
> I've added some more debug stuff. I'm beginning to suspect that maybe
> the db_get_result_u64() function does not return the right numbers.

I did a backtrace and concluded the same. I also added some debugging
statements in db_get_result_u64 and got something interesting return
values from db_get_result:


Mar 31 13:10:25 shiko dbmail/imap4d[17930]:
db.c,db_list_mailboxes_by_regex: comparing mailbox
[#Users/testuser1/testbox] to regular expressionMar 31 13:10:25 shiko
dbmail/imap4d[17930]: db.c,db_get_result_u64: result [^S]
Mar 31 13:10:25 shiko dbmail/imap4d[17930]:
db.c,db_list_mailboxes_by_regex: regex match #Users/testuser1/testbox
Mar 31 13:10:25 shiko dbmail/imap4d[17930]: db.c,db_get_result_u64:
result [3]
Mar 31 13:10:25 shiko dbmail/imap4d[17930]:
db.c,db_list_mailboxes_by_regex: comparing mailbox [INBOX] to regular
expression
Mar 31 13:10:25 shiko dbmail/imap4d[17930]: db.c,db_get_result_u64:
result [2]
Mar 31 13:10:25 shiko dbmail/imap4d[17930]:
db.c,db_list_mailboxes_by_regex: regex match INBOX
Mar 31 13:10:25 shiko dbmail/imap4d[17930]: db.c,db_get_result_u64:
result [3]
Mar 31 13:10:25 shiko dbmail/imap4d[17930]:
db.c,db_list_mailboxes_by_regex: comparing mailbox [testbox] to regular
expression
Mar 31 13:10:25 shiko dbmail/imap4d[17930]: db.c,db_get_result_u64:
result [15]



>
> can you update from cvs and try again?
>
> Ilja
>
>>
>>
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: IMAPClientHandler(): line
>> read for PID 19781
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: COMMAND: [A01 LIST "" *]
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: arg[0]: ''
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: arg[1]: '*'
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: IMAPClientHandler():
>> Executing command list...
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: ic_list(): build the
>> pattern: [^.*$]
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: dbmysql.c,db_query:
>> executing query [.SELECT mbx.name, mbx.mailbox_idnr, mbx.owner_idnr
>> FROM mailboxes mbx LEFT JOIN acl ON mbx.mailbox_idnr = acl.mailbox_id
>> WHERE (acl.user_id = '3' AND acl.lookup_flag = '1') OR mbx.owner_idnr
>> = '3']
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: dbmysql.c,db_query:
>> executing query [SELECT userid FROM users WHERE user_idnr = '2']
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: authsql.c,auth_get_userid:
>> query_result = testuser1
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: authsql.c,auth_get_userid:
>> returning testuser1 as returnid
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
>> misc.c,mailbox_add_namespace: owner name = testuser1
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
>> misc.c,mailbox_add_namespace: returning fully qualified name
>> [#Users/testuser1/testbox]
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
>> db.c,db_list_mailboxes_by_regex: comparing mailbox
>> [#Users/testuser1/testbox] to regular expressionMar 31 12:17:56 sarge
>> dbmail/imap4d[19781]: db.c,db_list_mailboxes_by_regex: regex match
>> #Users/testuser1/testbox
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
>> db.c,db_list_mailboxes_by_regex: comparing mailbox [INBOX] to regular
>> expression
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
>> db.c,db_list_mailboxes_by_regex: regex match INBOX
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
>> db.c,db_list_mailboxes_by_regex: comparing mailbox [testbox] to
>> regular expression
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
>> db.c,db_list_mailboxes_by_regex: regex match testbox
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]:
>> db.c,db_findmailbox_by_regex: found [3] mailboxes
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: imapcommands.c,_ic_list:
>> children[0] = 0
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: dbmysql.c,db_query:
>> executing query [SELECT owner_idnr FROM mailboxes WHERE mailbox_idnr =
>> '0']
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: dbmysql.c, db_get_result:
>> row = 0, field = 0, bigger than size of result set
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: db.c,db_getmailboxname:
>> error checking ownership of mailbox
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: IMAPClientHandler():
>> Finished command list
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: IMAPClientHandler():
>> Closing connection for client from IP [172.16.3.2]
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: PerformChildTask(): client
>> handling complete, closing streams
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: PerformChildTask():
>> connection closed
>> Mar 31 12:17:56 sarge dbmail/imap4d[19781]: PerformChildTask():
>> waiting for connection
>>
>>
>>
>>
>>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev@dbmail.org
> http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>


--
________________________________________________________________
Paul Stevens mailto:paul@nfg.nl
NET FACILITIES GROUP PGP: finger paul@nfg.nl
The Netherlands________________________________http://www.nfg.nl

Ilja Booij wrote:
> Paul J Stevens wrote:
>
>> Did that. Same result: children[0] = 0 which it should be.
>
>
> I've added some more debug stuff. I'm beginning to suspect that maybe
> the db_get_result_u64() function does not return the right numbers.
>
> can you update from cvs and try again?


And now using your debugging:
Ilja Booij wrote:
> Paul J Stevens wrote:
>
>> Did that. Same result: children[0] = 0 which it should be.
>
>
> I've added some more debug stuff. I'm beginning to suspect that maybe
> the db_get_result_u64() function does not return the right numbers.
>
> can you update from cvs and try again?


And now using your debugging:

Mar 31 13:30:41 shiko dbmail/imap4d[17947]:
misc.c,mailbox_add_namespace: owner name = testuser1
Mar 31 13:30:41 shiko dbmail/imap4d[17947]:
misc.c,mailbox_add_namespace: returning fully qualified name
[#Users/testuser1/testbox]
Mar 31 13:30:41 shiko dbmail/imap4d[17947]:
db.c,db_list_mailboxes_by_regex: comparing mailbox
[#Users/testuser1/testbox] to regular expressionMar 31 13:30:41 shiko
dbmail/imap4d[17947]: db.c,db_list_mailboxes_by_regex: regex match
#Users/testuser1/testbox, mailbox_idnr = 0
Mar 31 13:30:41 shiko dbmail/imap4d[17947]:
db.c,db_list_mailboxes_by_regex: comparing mailbox [INBOX] to regular
expression
Mar 31 13:30:41 shiko dbmail/imap4d[17947]:
db.c,db_list_mailboxes_by_regex: regex match INBOX, mailbox_idnr = 2
Mar 31 13:30:41 shiko dbmail/imap4d[17947]:
db.c,db_list_mailboxes_by_regex: comparing mailbox [testbox] to regular
expression
Mar 31 13:30:41 shiko dbmail/imap4d[17947]:
db.c,db_list_mailboxes_by_regex: regex match testbox, mailbox_idnr = 15
Mar 31 13:30:41 shiko dbmail/imap4d[17947]:
db.c,db_findmailbox_by_regex: found [3] mailboxes
Mar 31 13:30:41 shiko dbmail/imap4d[17947]: imapcommands.c,_ic_list:
children[0] = 0



--
________________________________________________________________
Paul Stevens mailto:paul@nfg.nl
NET FACILITIES GROUP PGP: finger paul@nfg.nl
The Netherlands________________________________http://www.nfg.nl

Paul J Stevens wrote:
> I did a backtrace and concluded the same. I also added some debugging
> statements in db_get_result_u64 and got something interesting return
> values from db_get_result:
>
>
> Mar 31 13:10:25 shiko dbmail/imap4d[17930]:
> db.c,db_list_mailboxes_by_regex: comparing mailbox
> [#Users/testuser1/testbox] to regular expressionMar 31 13:10:25 shiko
> dbmail/imap4d[17930]: db.c,db_get_result_u64: result [^S]

hmm.. interesting indeed. I take it that the ^S is the value of the tmp
string in db_get_result_u64()?

If so, the result from db_get_result() is off. You're using MySQL,
right? Maybe you should add some debug statements in mysql/dbmysql.c, to
see if anything goes wrong there.

I've looked through db_get_result() again, but can't seem to find
anything wrong.. Glad we narrowed down the search though.

Ilja

Yep. Its in dbmysql.c all right.

I hateses globals :-/

Storing the last_row_nr in a global variable is a flawed approach. I
assume a static like last_row_nr is an optimization that wants to
outsmart the mysql api. Such optimizations however, should, have and
will occur in the mysql libs. So, is there a solid, benchmarked reason
for avoiding use of mysql_data_seek and mysql_fetch_row.

My trivial patch:

diff -r1.84 dbmysql.c
156a157
> /*
163a165,167
> */
> mysql_data_seek(res, row);
> last_row = mysql_fetch_row(res);

fixes the problem I've been having with _ic_list.

Looking into this some further I noticed db_query tries to avoid
mysql_store_result, where the mysql docs clearly state such calls are ok:


------ info: mysql_store_result -----
You must call `mysql_store_result()' or `mysql_use_result()' for every
query that successfully retrieves data (`SELECT', `SHOW', `DESCRIBE',
`EXPLAIN').


You don't have to call `mysql_store_result()' or `mysql_use_result()'
for other queries, but it will not do any harm or cause any notable
performance if you call `mysql_store_result()' in all cases. You can
detect if the query didn't have a result set by checking if
`mysql_store_result()' returns 0 (more about this later one).


------------------------------------


Reading this db_query just doesn't look right to me. What do you think
Ilja ?

Btw, the testsuite now runs ok with the patch mentioned :-) So can we
clean up dbmysql.c a little and move on ?

I read about you fixing the sigpipe error. Have you committed any
changes to that effect?





Ilja Booij wrote:
> Paul J Stevens wrote:
>
>> I did a backtrace and concluded the same. I also added some debugging
>> statements in db_get_result_u64 and got something interesting return
>> values from db_get_result:
>>
>>
>> Mar 31 13:10:25 shiko dbmail/imap4d[17930]:
>> db.c,db_list_mailboxes_by_regex: comparing mailbox
>> [#Users/testuser1/testbox] to regular expressionMar 31 13:10:25 shiko
>> dbmail/imap4d[17930]: db.c,db_get_result_u64: result [^S]
>
>
> hmm.. interesting indeed. I take it that the ^S is the value of the tmp
> string in db_get_result_u64()?
>
> If so, the result from db_get_result() is off. You're using MySQL,
> right? Maybe you should add some debug statements in mysql/dbmysql.c, to
> see if anything goes wrong there.
>
> I've looked through db_get_result() again, but can't seem to find
> anything wrong.. Glad we narrowed down the search though.
>
> Ilja
>
>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev@dbmail.org
> http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>

--
________________________________________________________________
Paul Stevens mailto:paul@nfg.nl
NET FACILITIES GROUP PGP: finger paul@nfg.nl
The Netherlands________________________________http://www.nfg.nl

Paul J Stevens wrote:
> Yep. Its in dbmysql.c all right.
>
> I hateses globals :-/
>
> Storing the last_row_nr in a global variable is a flawed approach. I
> assume a static like last_row_nr is an optimization that wants to
> outsmart the mysql api. Such optimizations however, should, have and
> will occur in the mysql libs. So, is there a solid, benchmarked reason
> for avoiding use of mysql_data_seek and mysql_fetch_row.
This was done in November, after Alex Wheeler reported that
mysql_data_seek() was way to slow. I agree that these optimizations
belong in the libs, not in the code using the libs.

From mailing.database.mysql, 2001-06-21:
""
mysql_store_result stores the result set in a linked list (for memory
efficiency). With older versions mysql_data_seek always starts from
the beginning of the list and therefore has an O(n) complexity even
for sequential traversal.

IIRC, this was fixed with newer MySQL client libs, you either want to
upgrade or write some small wrapper.
""

So, this supports your stance, more so because we demand MySQL version
>= 2.0.14 anyway. Those versions should have this optimized internally.
I guess Alex was using MySQL 3.xx. (I could be wrong of course)

>
> My trivial patch:
>
> diff -r1.84 dbmysql.c
> 156a157
> > /*
> 163a165,167
> > */
> > mysql_data_seek(res, row);
> > last_row = mysql_fetch_row(res);
>
> fixes the problem I've been having with _ic_list.
OK, I'll fix this and commit it.
>
> Looking into this some further I noticed db_query tries to avoid
> mysql_store_result, where the mysql docs clearly state such calls are ok:
>
>
> ------ info: mysql_store_result -----
> You must call `mysql_store_result()' or `mysql_use_result()' for every
> query that successfully retrieves data (`SELECT', `SHOW', `DESCRIBE',
> `EXPLAIN').
>
>
> You don't have to call `mysql_store_result()' or `mysql_use_result()'
> for other queries, but it will not do any harm or cause any notable
> performance if you call `mysql_store_result()' in all cases. You can
> detect if the query didn't have a result set by checking if
> `mysql_store_result()' returns 0 (more about this later one).
>
> ------------------------------------
>
>
> Reading this db_query just doesn't look right to me. What do you think
> Ilja ?

I took this code from earlier DBMail versions without thinking about it
too much.
So, we could just do a mysql_store_result(), and res will be set to NULL
when appropriate (i.e. a after a query which returns no result)? Nice.
This will make the function a lot simpler.

I guess we should do this then.


> Btw, the testsuite now runs ok with the patch mentioned :-) So can we
> clean up dbmysql.c a little and move on ?
I guess so, I'll get to it.
>
> I read about you fixing the sigpipe error. Have you committed any
> changes to that effect?
Haven't committed anything yet. I was wondering about the effects of the
'return EOF' statements after a ci_write() fails, because nothing is
cleaned up before them. I guess we should first clean up before
returning, don't you think?

Ilja

Ilja Booij wrote:

>> I read about you fixing the sigpipe error. Have you committed any
>> changes to that effect?
>
> Haven't committed anything yet. I was wondering about the effects of the
> 'return EOF' statements after a ci_write() fails, because nothing is
> cleaned up before them. I guess we should first clean up before
> returning, don't you think?

You're probably correct here. Resource allocation/deallocation is still mostly
unexplored territory for me. I've tried to be really carefull in leaving all
cleanup actions in place, even though moving most of them to a separate
imap_cleanup function. Some leaks are still there, but exposing them was part of
my motivation for writing that patch. Looks to me like adding some calls to
imap_cleanup here and there in ImapClientHandler should do the trick.



--
________________________________________________________________
Paul Stevens paul@nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands_______________________________________www.nfg.nl

Ilja Booij wrote:
>> My trivial patch:
>>
>> diff -r1.84 dbmysql.c
>> 156a157
>> > /*
>> 163a165,167
>> > */
>> > mysql_data_seek(res, row);
>> > last_row = mysql_fetch_row(res);
>>
>> fixes the problem I've been having with _ic_list.
>
> OK, I'll fix this and commit it.

Fixed and committed.

> I took this code from earlier DBMail versions without thinking about it
> too much.
> So, we could just do a mysql_store_result(), and res will be set to NULL
> when appropriate (i.e. a after a query which returns no result)? Nice.
> This will make the function a lot simpler.
>
> I guess we should do this then.

Done and committed
>
>
>> Btw, the testsuite now runs ok with the patch mentioned :-) So can we
>> clean up dbmysql.c a little and move on ?
>
> I guess so, I'll get to it.

Current CVS HEAD should work for you now. I guess we're finished with
this. I'll take another look at the cleaning up code in imap4.c for the
SIGPIPE stuff.

Ilja

Paul J Stevens wrote:
> You're probably correct here. Resource allocation/deallocation is still
> mostly unexplored territory for me. I've tried to be really carefull in
> leaving all cleanup actions in place, even though moving most of them to
> a separate imap_cleanup function. Some leaks are still there, but
> exposing them was part of my motivation for writing that patch. Looks to
> me like adding some calls to imap_cleanup here and there in
> ImapClientHandler should do the trick.

I've committed the ci_write() patch (for lack of a better name..) and
added some more ci_cleanup() calls in the appropriate places. It seems
to work OK here. I don't have any memleaks here in my test runs (using
valgrind to spot mem-leaks and other memory errors)

Paul, can you test it on your machines?

Ilja

Great.

No more sigpipes here :-)

Ilja Booij wrote:
> Paul J Stevens wrote:
>
>> You're probably correct here. Resource allocation/deallocation is
>> still mostly unexplored territory for me. I've tried to be really
>> carefull in leaving all cleanup actions in place, even though moving
>> most of them to a separate imap_cleanup function. Some leaks are still
>> there, but exposing them was part of my motivation for writing that
>> patch. Looks to me like adding some calls to imap_cleanup here and
>> there in ImapClientHandler should do the trick.
>
>
> I've committed the ci_write() patch (for lack of a better name..) and
> added some more ci_cleanup() calls in the appropriate places. It seems
> to work OK here. I don't have any memleaks here in my test runs (using
> valgrind to spot mem-leaks and other memory errors)
>
> Paul, can you test it on your machines?
>
> Ilja
>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev@dbmail.org
> http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>

--
________________________________________________________________
Paul Stevens mailto:paul@nfg.nl
NET FACILITIES GROUP PGP: finger paul@nfg.nl
The Netherlands________________________________http://www.nfg.nl

OK great.

Now I think we only need to fix the parser error that Aaron found. Aaron
won't be back until monday, and he's the only one who's had this
problem, which kind of prevents us from doing anything about it at the
moment.

Meanwhile, I'll add the IMAP test script to the distribution. Can I just
slam a GPL notice on it, or do you want another license (it's your code
:) )?

Ilja


Paul J Stevens wrote:

> Great.
>
> No more sigpipes here :-)
>
> Ilja Booij wrote:
>
>> Paul J Stevens wrote:
>>
>>> You're probably correct here. Resource allocation/deallocation is
>>> still mostly unexplored territory for me. I've tried to be really
>>> carefull in leaving all cleanup actions in place, even though moving
>>> most of them to a separate imap_cleanup function. Some leaks are
>>> still there, but exposing them was part of my motivation for writing
>>> that patch. Looks to me like adding some calls to imap_cleanup here
>>> and there in ImapClientHandler should do the trick.
>>
>>
>>
>> I've committed the ci_write() patch (for lack of a better name..) and
>> added some more ci_cleanup() calls in the appropriate places. It seems
>> to work OK here. I don't have any memleaks here in my test runs (using
>> valgrind to spot mem-leaks and other memory errors)
>>
>> Paul, can you test it on your machines?
>>
>> Ilja
>>
>> _______________________________________________
>> Dbmail-dev mailing list
>> Dbmail-dev@dbmail.org
>> http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>>
>

Ilja Booij wrote:
> OK great.
>
> Now I think we only need to fix the parser error that Aaron found. Aaron
> won't be back until monday, and he's the only one who's had this
> problem, which kind of prevents us from doing anything about it at the
> moment.

There's one other problem that's bugging me. I sometimes get empty
messages showing up. I thought it was a problem with thunderbird, but it
isn't. Apparently, messages sometimes only get one messageblk for the
headers, and that's it.

I could be related to Aaron's problem, but I'm not sure. It could also
be a problem with my dbmail database. I will investigate further.

> Meanwhile, I'll add the IMAP test script to the distribution. Can I just
> slam a GPL notice on it, or do you want another license (it's your code
> :) )?

Feel free to add the GPL notice, but please add my name as well :-)

Also, there's one project out there that looks interesting:

http://www.mozilla.org/projects/mstone/
"""
Mstone is a multi-protocol stress and performance measurement tool.
Mstone can test multiple protocols simultaneously and measures the
performance of every transaction. The performance can be graphed
throughout the duration of the test.
"""

Very useful as I'll resume work on my preforking patch, and we start
thinking about the C10K problem.

http://www.kegel.com/c10k.html

--
________________________________________________________________
Paul Stevens mailto:paul@nfg.nl
NET FACILITIES GROUP PGP: finger paul@nfg.nl
The Netherlands________________________________http://www.nfg.nl

I still get both the unparseable message and empty message problems
(post rc4 CVS) occasionally, the empty message happens more often, about
once a day or so. For the empty messages it's a matter of the imap
server not pulling the whole message, because dbmail-smtp does put it
all in there if I look with the command line mysql client. What can I
provide to help solve these problems? Is there a way to have the mysql
client dump the messageblks as hex or something which will better show
the non printing characters?

Paul J Stevens wrote:
>
>
> Ilja Booij wrote:
>
>> OK great.
>>
>> Now I think we only need to fix the parser error that Aaron found.
>> Aaron won't be back until monday, and he's the only one who's had this
>> problem, which kind of prevents us from doing anything about it at the
>> moment.
>
>
> There's one other problem that's bugging me. I sometimes get empty
> messages showing up. I thought it was a problem with thunderbird, but it
> isn't. Apparently, messages sometimes only get one messageblk for the
> headers, and that's it.
>
> I could be related to Aaron's problem, but I'm not sure. It could also
> be a problem with my dbmail database. I will investigate further.
>
>> Meanwhile, I'll add the IMAP test script to the distribution. Can I
>> just slam a GPL notice on it, or do you want another license (it's
>> your code :) )?
>
>
> Feel free to add the GPL notice, but please add my name as well :-)
>
> Also, there's one project out there that looks interesting:
>
> http://www.mozilla.org/projects/mstone/
> """
> Mstone is a multi-protocol stress and performance measurement tool.
> Mstone can test multiple protocols simultaneously and measures the
> performance of every transaction. The performance can be graphed
> throughout the duration of the test.
> """
>
> Very useful as I'll resume work on my preforking patch, and we start
> thinking about the C10K problem.
>
> http://www.kegel.com/c10k.html
>