Mailing List Archive

On Sun, 2011-03-13 at 04:11 +0100, Alex Cohen wrote:
> Hi Andrew,
>
> Finally I came to this and I am writing with a success report and a
> couple of questions.
>
> Palm Desktop 3, 4 and 6 can export only some strange *.dba format. I
> found some guidelines to import it in Yahoo! and export it as Outlook,
> but I never found this function in Yahoo! Fortunately after some
> research I found DIMEX (http://www.linkesoft.de/dimex/ - in German
> only..., USD 19,-) - I had to downgrade to Palm 4 and DIMEX exported a
> beautiful .ics file.
>
> There started the trouble. As I could not understand what happens, if
> I upload it via the web interface, I gave it to Sunbird to import it.
> Either Sunbird got stuck or I was a bit impatient, but I thought it
> takes long (more than 5 minutes) and stopped it in Sunbird and did it
> again in iCal. iCal did it, but I ended up with duplicates. I thought
> the events have unique ID and will not duplicate?
>
> I imported about 7 000 events for a 10 years period. Is this too much
> for DAViCal? Because while playing with two Sunbirds and an iCal in
> 10-15 minutes they somehow messed up and stopped synchronizing - I had
> them all display the same calendar, on the same day and tried to
> change events here and there to see how they update each other. Now
> the final state is that each of them has some event which the other
> does not, and they do not get synchronized and the iCal totally lost
> all events for this day. Looking directly into the database, I can see
> the events.

With 7000 events I can imagine they could well get confused. While
people have told me of calendars of such size I don't have any like that
myself, or even to test with - I think the biggest calendar in my
regression tests is around 1000 events, and the largest I use for
occasional testing is around 2500.

Sunbird will take a very long time to import a calendar into DAViCal, as
it splits it into individual events, and then PUTs each event to
DAViCal, but in parallel, firing off many, many requests all at once and
quite possibly overloading the server. It can also cause deadlocks, I
think.

Loading the events directly in DAViCal in the collection screen
uploading the file to replace an existing calendar will hopefully work
better. It should take less than five minutes, and in fact if it takes
more than that then you may need to increase your connection timeout in
Apache which defaults to 300 seconds. Another thing you probably will
need to increase during such an import would be the PHP memory limit.
I'd tend to put it to 512m during the import and reset it afterwards.

Loading them in iCal probably also works but I don't have any experience
of that. I doubt that it has the parallel request issue that Lightning
has though, because it doesn't have a browser engine providing it's HTTP
stack :-)

My recommendation would be to delete the calendar in DAViCal and create
it again. Both iCal and Lightning should see the changes, but if not
you should delete & recreate the accounts to ensure you have a clean
starting point.

From there I would try and import them using iCal. Confirm that they
are in the database, and try again to connect with Lightning to see if
it can see the events.

I would also tend to split such a large calendar into one or two
archives so that the events in the current calendar were just the ones
for this year (or since the beginning of 2010 if you often need to refer
back prior to January).

Cheers,
Andrew.
--
------------------------------------------------------------------------
http://andrew.mcmillan.net.nz/ Porirua, New Zealand
Twitter: _karora Phone: +64(272)DEBIAN
Practice yourself what you preach.
-- Titus Maccius Plautus

------------------------------------------------------------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.davical.org/pipermail/davical-dev/attachments/20110313/f64827c4/attachment.pgp>

> With 7000 events I can imagine they could well get confused. While
> people have told me of calendars of such size I don't have any like that
> myself, or even to test with - I think the biggest calendar in my
> regression tests is around 1000 events, and the largest I use for
> occasional testing is around 2500.

I would be happy to do regression tests with this calendar.

> Sunbird will take a very long time to import a calendar into DAViCal, as
> it splits it into individual events, and then PUTs each event to
> DAViCal, but in parallel, firing off many, many requests all at once and
> quite possibly overloading the server.

DAViCal is on an almost idle Dell T710 with 1 x 4 core CPU and 4 GB RAM. I did not check load during import, but do you think this server can be overloaded?

> It can also cause deadlocks, I
> think.

What do you mean?

> Loading the events directly in DAViCal in the collection screen
> uploading the file to replace an existing calendar will hopefully work
> better.

I am afraid the interface for this is not very clear regarding what will happen and what is to be done...

> It should take less than five minutes, and in fact if it takes
> more than that then you may need to increase your connection timeout in
> Apache which defaults to 300 seconds. Another thing you probably will
> need to increase during such an import would be the PHP memory limit.
> I'd tend to put it to 512m during the import and reset it afterwards.

That sounds reasonable.

> Loading them in iCal probably also works but I don't have any experience
> of that. I doubt that it has the parallel request issue that Lightning
> has though, because it doesn't have a browser engine providing it's HTTP
> stack :-)

I'll try to re-import in empty calendars during the next days and let you know.

> My recommendation would be to delete the calendar in DAViCal and create
> it again. Both iCal and Lightning should see the changes, but if not
> you should delete & recreate the accounts to ensure you have a clean
> starting point.

Why do they get confused on first place? Do they have some kind of cache or so?

> From there I would try and import them using iCal. Confirm that they
> are in the database, and try again to connect with Lightning to see if
> it can see the events.

Actually I use Sunbird. Is Lightning the same codebase? I was wondering if Lightning is more stable. Sunbird has quite a few issues and it flickers on refresh, which is a bit disturbing.

Is there some effort in direction of creating a web front end of DAViCal? I do not imagine it being very difficult. I had a look some time ago into phpicalendar or so... And I would imagine contributing. Our company is about 50 people and the calendars are very important and people are a bit tired of the ever changing things - Microsoft Office did a leap lately, which confused users for months and months and Thunderbird committed the same sin the last months. And Sunbird seems stuck in beta forever. I could imagine a web front end as quite an easy job. What do you think?

> I would also tend to split such a large calendar into one or two
> archives so that the events in the current calendar were just the ones
> for this year (or since the beginning of 2010 if you often need to refer
> back prior to January).

You say this in regard to the clients (Sunbird, etc.) not being very efficient, right? The server should be fine, or? Or is the protocol a bottleneck for larger calendars, as well?

Thank you,
Iv

On Sun, 2011-03-13 at 23:25 +0100, Iv Ray wrote:
> > With 7000 events I can imagine they could well get confused. While
> > people have told me of calendars of such size I don't have any like that
> > myself, or even to test with - I think the biggest calendar in my
> > regression tests is around 1000 events, and the largest I use for
> > occasional testing is around 2500.
>
> I would be happy to do regression tests with this calendar.

Possibly you misunderstand. The regression tests I refer to are
committed to the Git repository, so the information is entirely public.
Most of the time when I make a change to DAViCal I run the regression
tests to make sure that I haven't made things worse, and they certainly
get run before releases happen and so forth.


> > Sunbird will take a very long time to import a calendar into DAViCal, as
> > it splits it into individual events, and then PUTs each event to
> > DAViCal, but in parallel, firing off many, many requests all at once and
> > quite possibly overloading the server.
>
> DAViCal is on an almost idle Dell T710 with 1 x 4 core CPU and 4 GB RAM. I did not check load during import, but do you think this server can be overloaded?
>
> > It can also cause deadlocks, I
> > think.
>
> What do you mean?

Deadlocks are when two concurrent database transactions lock some of the
tables each one needs in such a way that neither transaction can
proceed. I'm currently adding some code to try and work around this
issue.


> > Loading the events directly in DAViCal in the collection screen
> > uploading the file to replace an existing calendar will hopefully work
> > better.
>
> I am afraid the interface for this is not very clear regarding what
> will happen and what is to be done...

In the DAViCal Admin UI go to the List Principals -> choose the
principal you want. At the bottom of the page there should be a list of
collections, choose the collection you want. On the page that comes up
there is a file browse widget which you should be able to use to select
your .ics file and click "Submit". This will delete the entire content
of the calendar collection replacing it with the content of the .ics
file.


> > It should take less than five minutes, and in fact if it takes
> > more than that then you may need to increase your connection timeout in
> > Apache which defaults to 300 seconds. Another thing you probably will
> > need to increase during such an import would be the PHP memory limit.
> > I'd tend to put it to 512m during the import and reset it afterwards.
>
> That sounds reasonable.

Given the hardware you quote it should not take five minutes, but the
PHP memory limit will still need to be temporarily increased.


> Why do they get confused on first place? Do they have some kind of cache or so?

Yes, they have local caching (well, iCal does certainly). iCal also
maintains some revision information so when you change the calendar it
will only fetch the newest changes. If it's somehow got out of sync
with what is really there, then it can still show changes but not stuff
that is actually already present. On the other hand it can also be a
little picky about the data itself, so (e.g.) I've seen it not display
events which have mixed CRLF / LF line endings. If it's out of sync the
best solution is to delete the account in iCal and re-create it.


> Actually I use Sunbird. Is Lightning the same codebase? I was
> wondering if Lightning is more stable. Sunbird has quite a few issues
> and it flickers on refresh, which is a bit disturbing.

Sunbird was the same codebase as Lightning but it is now discontinued.


> Is there some effort in direction of creating a web front end of
> DAViCal? I do not imagine it being very difficult. I had a look some
> time ago into phpicalendar or so... And I would imagine contributing.
> Our company is about 50 people and the calendars are very important
> and people are a bit tired of the ever changing things - Microsoft
> Office did a leap lately, which confused users for months and months
> and Thunderbird committed the same sin the last months. And Sunbird
> seems stuck in beta forever. I could imagine a web front end as quite
> an easy job. What do you think?

Michael Rasmussen has written a read-only web front-end, and is planning
to add write support to that. I believe there is also a plugin for
RoundCube coming soon. It may be that Horde can also work - not sure
about that.


> > I would also tend to split such a large calendar into one or two
> > archives so that the events in the current calendar were just the ones
> > for this year (or since the beginning of 2010 if you often need to refer
> > back prior to January).
>
> You say this in regard to the clients (Sunbird, etc.) not being very
> efficient, right? The server should be fine, or? Or is the protocol a
> bottleneck for larger calendars, as well?

Yes, there are places in the DAV protocol where client software may well
retrieve a property for all resources in a directory. Some of the
better clients try and avoid doing this, because it can consume large
amounts of bandwidth and server resources - just like I try not to use
'ls' in a folder with 7000 files. Any initial synchronisation has to
involve retrieving all of those records though, including some metadata.

More recent versions of Lightning can also cache the results, but I
don't think it's a default setting yet.

Cheers,
Andrew.

--
------------------------------------------------------------------------
andrew (AT) morphoss (DOT) com +64(272)DEBIAN
Flexibility is overrated. Constraints are liberating.
------------------------------------------------------------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.davical.org/pipermail/davical-dev/attachments/20110314/316d9dbf/attachment.pgp>