Mailing List Archive

Disappearing entries?
I have a set of huge calendars in DAViCal that have been maintained using
quite a few clients over the years.

When I inspect a specific collection:

Collection: Joan
Collection ID: 10929
DAV Path: /caldav.php/jv/joan/
Items in Collection: 675

Now I fetch the .ics file for this calendar from the server:

wget --user=... --password=... http://davical.../caldav.php/jv/joan.ics

After reloading it using the web interface, I get:

Collection: Joan
Collection ID: 10929
DAV Path: /caldav.php/jv/joan/
Items in Collection: 668

Apparently, a number of items have disappeared.

What is going on?

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general
Re: Disappearing entries? [ In reply to ]
On 28.05.2018 08:41, Johan Vromans wrote:
> I have a set of huge calendars in DAViCal that have been maintained using
> quite a few clients over the years.
>
> When I inspect a specific collection:
>
> Collection: Joan
> Collection ID: 10929
> DAV Path: /caldav.php/jv/joan/
> Items in Collection: 675
>
> Now I fetch the .ics file for this calendar from the server:
>
> wget --user=... --password=... http://davical.../caldav.php/jv/joan.ics
>
> After reloading it using the web interface, I get:
>
> Collection: Joan
> Collection ID: 10929
> DAV Path: /caldav.php/jv/joan/
> Items in Collection: 668
>
> Apparently, a number of items have disappeared.
>
> What is going on?
>
>

Using davical for quite a long time, I have noticed similar effects
every now and then: however, I do not know "who" is responsible for it. 
My personal god-feeling is that my variety of Windows, iOS and Android
based tools is repsonsible for it.  My primary suspect is my iPad...

If any debugging method on server-side can help, I'd be very much
interested in learning how to do it.

Thanks for sharing your insights.

Best regards
Lukas
Re: Disappearing entries? [ In reply to ]
On 29.05.2018 19:18, Lukas Ruf wrote:
> On 28.05.2018 08:41, Johan Vromans wrote:
>> I have a set of huge calendars in DAViCal that have been maintained using
>> quite a few clients over the years.
>>
>> When I inspect a specific collection:
>>
>> Collection: Joan
>> Collection ID: 10929
>> DAV Path: /caldav.php/jv/joan/
>> Items in Collection: 675
>>
>> Now I fetch the .ics file for this calendar from the server:
>>
>> wget --user=... --password=... http://davical.../caldav.php/jv/joan.ics
>>
>> After reloading it using the web interface, I get:
>>
>> Collection: Joan
>> Collection ID: 10929
>> DAV Path: /caldav.php/jv/joan/
>> Items in Collection: 668
>>
>> Apparently, a number of items have disappeared.
>>
>> What is going on?
>>
>>
>
> Using davical for quite a long time, I have noticed similar effects
> every now and then: however, I do not know "who" is responsible for
> it.  My personal god-feeling is that my variety of Windows, iOS and
> Android based tools is repsonsible for it.  My primary suspect is my
> iPad...
>

After sending, I realized: my sporadic data losses happen primarily when
I make use of davical for carddav -- at least, there I have noticed....


> If any debugging method on server-side can help, I'd be very much
> interested in learning how to do it.
>

Does anybody know of a easy way how to track the collections on a
DB-level on server, and can share this method?  I'd be happy to run such
a script frequently on server-side.

Thanks
Lukas
Re: Disappearing entries? [ In reply to ]
On Tue, 29 May 2018 19:18:11 +0200, Lukas Ruf <Lukas.Ruf@lpr.ch> wrote:

> My personal god-feeling is that my variety of Windows, iOS and Android
> based tools is repsonsible for it.  My primary suspect is my iPad...

My suspect would be DAViCal...

In the setup I reported, DAViCal says it has 675 items.
Export .ics (#1) and importing results in 668 items.
Export .ics (#2) and importing results in 668 items.

Both .ics files are of equal size, and contain the same UIDs. Even though
the VEVENTS and attributes are in a different order, I think the two files
can be considered equal.

My tentative conclusion is that the actual number of entries is, indeed,
668, but that DAViCal occasionally loses track of the item count due to
creating/removing/updating events.

-- Johan

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general
Re: Disappearing entries? [ In reply to ]
On Tue, 2018-05-29 at 19:28 +0200, Lukas Ruf wrote:
>
> > If any debugging method on server-side can help, I'd be very much
> > interested in learning how to do it.
> >
>
> Does anybody know of a easy way how to track the collections on a DB-
> level on server, and can share this method? I'd be happy to run such
> a script frequently on server-side.

How comfortable are you with SQL?

If you can connect to the database, then you can try:

select count(*) from caldav_data where collection_id = 10929;

To get the summary line for events, then try:

select summary from calendar_item where collection_id = 10929;

And so on.

Cheers,
Andrew

--
Andrew Ruthven, Wellington, New Zealand
andrew@etc.gen.nz | linux.conf.au 2019, Christchurch, NZ
https://catalystcloud.nz | https://lca2019.linux.org.au

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general
Re: Disappearing entries? [ In reply to ]
On 29.05.2018 23:17, Andrew Ruthven wrote:
> On Tue, 2018-05-29 at 19:28 +0200, Lukas Ruf wrote:
>>> If any debugging method on server-side can help, I'd be very much
>>> interested in learning how to do it.
>>>
>>
>> Does anybody know of a easy way how to track the collections on a DB-
>> level on server, and can share this method? I'd be happy to run such
>> a script frequently on server-side.
> How comfortable are you with SQL?
>
> If you can connect to the database, then you can try:
>
> select count(*) from caldav_data where collection_id = 10929;
>
> To get the summary line for events, then try:
>
> select summary from calendar_item where collection_id = 10929;

Thanks for the hint.

If I understand correctly, this would help me detect disappearances.

Is there any way of detecting which client would have caused "delete"
requests?

Thanks
Lukas
Re: Disappearing entries? [ In reply to ]
Johan Vromans wrote, on 29.5.2018 19:35:
> On Tue, 29 May 2018 19:18:11 +0200, Lukas Ruf <Lukas.Ruf@lpr.ch> wrote:
>
> In the setup I reported, DAViCal says it has 675 items.
> Export .ics (#1) and importing results in 668 items.
> Export .ics (#2) and importing results in 668 items.
>
> Both .ics files are of equal size, and contain the same UIDs. Even though
> the VEVENTS and attributes are in a different order, I think the two files
> can be considered equal.

What if you export the imported data and import them again? Is there any
further loss, or is it only the first import?

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general
Re: Disappearing entries? [ In reply to ]
On Tue, 29 May 2018 23:49:56 +0200, Josef Kufner <jk@frozen-doe.net> wrote:

> > In the setup I reported, DAViCal says it has 675 items.
> > Export .ics (#1) and importing results in 668 items.
> > Export .ics (#2) and importing results in 668 items.
> >
> > Both .ics files are of equal size, and contain the same UIDs. Even
> > though the VEVENTS and attributes are in a different order, I think the
> > two files can be considered equal.
>
> What if you export the imported data and import them again? Is there any
> further loss, or is it only the first import?

I thought that was just what I described I did...

Originally, DAViCal says it has 675 items.
Export to one.ics and importing one.ics results in 668 items.
Export to two.ics and importing two.ics results in 668 items.

After importing the first export set, there seem to be no more changes when
exporting/importing.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general
Re: Disappearing entries? [ In reply to ]
Johan Vromans wrote, on 30.5.2018 08:13:
> On Tue, 29 May 2018 23:49:56 +0200, Josef Kufner <jk@frozen-doe.net> wrote:
>
>>> In the setup I reported, DAViCal says it has 675 items.
>>> Export .ics (#1) and importing results in 668 items.
>>> Export .ics (#2) and importing results in 668 items.
>>>
>>> Both .ics files are of equal size, and contain the same UIDs. Even
>>> though the VEVENTS and attributes are in a different order, I think the
>>> two files can be considered equal.
>>
>> What if you export the imported data and import them again? Is there any
>> further loss, or is it only the first import?
>
> I thought that was just what I described I did...
>
> Originally, DAViCal says it has 675 items.
> Export to one.ics and importing one.ics results in 668 items.
> Export to two.ics and importing two.ics results in 668 items.
>
> After importing the first export set, there seem to be no more changes when
> exporting/importing.

Ah, I misunderstood that :)

I guess it is some incompatibility or flaw in data which gets caught by
newer version of import.

How many items contains one.ics? Try something like this to count them
(it should count one extra item for the file/collection itself):

grep -e '^BEGIN:' one.ics | wc -l

If one.ics contains 668(+1) items, it is bug in the export. If it
contains 675(+1) items, it is bug in the import.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general
Re: Disappearing entries? [ In reply to ]
On Wed, 30 May 2018 10:24:45 +0200, Josef Kufner <jk@frozen-doe.net> wrote:

> How many items contains one.ics? Try something like this to count them
> (it should count one extra item for the file/collection itself):
>
> grep -e '^BEGIN:' one.ics | wc -l
>
> If one.ics contains 668(+1) items, it is bug in the export. If it
> contains 675(+1) items, it is bug in the import.

The grep returns 851 occurrences of "^BEGIN".

To be more precise:

BEGIN:DAYLIGHT 2
BEGIN:STANDARD 2
BEGIN:VALARM 22
BEGIN:VCALENDAR 1
BEGIN:VEVENT 776
BEGIN:VTIMEZONE 2
BEGIN:VTODO 46

-- Johan

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general
Re: Disappearing entries? [ In reply to ]
Johan Vromans wrote, on 30.5.2018 12:15:
> On Wed, 30 May 2018 10:24:45 +0200, Josef Kufner <jk@frozen-doe.net> wrote:
>
>> How many items contains one.ics? Try something like this to count them
>> (it should count one extra item for the file/collection itself):
>>
>> grep -e '^BEGIN:' one.ics | wc -l
>>
>> If one.ics contains 668(+1) items, it is bug in the export. If it
>> contains 675(+1) items, it is bug in the import.
>
> The grep returns 851 occurrences of "^BEGIN".
>
> To be more precise:
>
> BEGIN:DAYLIGHT 2
> BEGIN:STANDARD 2
> BEGIN:VALARM 22
> BEGIN:VCALENDAR 1
> BEGIN:VEVENT 776
> BEGIN:VTIMEZONE 2
> BEGIN:VTODO 46
>
> -- Johan

So, DAViCal says it has 675 items, but you get 822 items in the export?
This is really wierd. Isn't just borken the counter of the items?

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general
Re: Disappearing entries? [ In reply to ]
On Wed, 30 May 2018 12:27:43 +0200, Josef Kufner <jk@frozen-doe.net> wrote:

> So, DAViCal says it has 675 items, but you get 822 items in the export?
> This is really wierd. Isn't just borken the counter of the items?

Maybe this indicates something:

$ grep -e '^UID:' one.ics | sort -u | wc -l
=> 669

-- Johan

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general
Re: Disappearing entries? [ In reply to ]
Johan Vromans wrote, on 30.5.2018 12:39:
> On Wed, 30 May 2018 12:27:43 +0200, Josef Kufner <jk@frozen-doe.net> wrote:
> Maybe this indicates something:
>
> $ grep -e '^UID:' one.ics | sort -u | wc -l
> => 669

Oh, that makes much more sense :) ... ok, so the export does not export
everything.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general
Re: Disappearing entries? [ In reply to ]
On Wed, 30 May 2018 09:17:16 +1200, Andrew Ruthven <andrew@etc.gen.nz>
wrote:

> select count(*) from caldav_data where collection_id = 10929;

select count(*) from caldav_data where collection_id = 10929;
count
-------
675
(1 row)

However:

select count(distinct uid) from calendar_item where collection_id = 10929;
count
-------
668
(1 row)

-- Johan

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general
Re: Disappearing entries? [ In reply to ]
On Wed, 2018-05-30 at 14:40 +0200, Johan Vromans wrote:
> On Wed, 30 May 2018 09:17:16 +1200, Andrew Ruthven <andrew@etc.gen.nz
> >
> wrote:
>
> > select count(*) from caldav_data where collection_id = 10929;
>
> select count(*) from caldav_data where collection_id = 10929;
> count
> -------
> 675
> (1 row)
>
> However:
>
> select count(distinct uid) from calendar_item where collection_id =
> 10929;
> count
> -------
> 668
> (1 row)

Ah ha, so davical is counting multiple versions of the same event. Your
count if 668 items in the exported ICS file is sane then.

To find the duplicate UIDs use:

select uid, count(*) from calendar_item
where collection_id = 10929
group by uid
having count(*) > 1;

I expect that davical should really be using the distinct uid when it
shows the item count.

Cheers,
Andrew

--
Andrew Ruthven, Wellington, New Zealand
andrew@etc.gen.nz | linux.conf.au 2019, Christchurch, NZ
https://catalystcloud.nz | https://lca2019.linux.org.au

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general