Mailing List Archive

Looking a way to archive between date a collection to another
On Sun, 2010-03-07 at 14:08 +0100, Bruno Friedmann wrote:
> Hi Guys,
>
> I've got here (due to super slow sunbird perf) a question about
> putting old events from a collection to
> another one.
>
> I imagine that's better to do this directly in postgresql as normal
> ical client ask event by event (there's 9'000 to move :-)
>
> I would have done some research on my own yesterday and today, but
> I've crash my car yesterday on ice. And been busy to organize
> fixing, doing insurance things etc ..
>
> So If someone has some shortcuts to help me building a good sql, it
> will be really appreciate.

Hi Bruno,

You need to update caldav_data table (calendar_item will be maintained
with a trigger) with the new path, and the new collection_id.

Here's an example moving non-repeating events, prior to the 1st of
January, from /user1/home/ (collection_id 10 for me) to /user1/archive/
(collection_id 1429 for me):


UPDATE caldav_data
SET dav_name = replace( caldav_data.dav_name, '/user1/home/', '/user1/archive/'),
collection_id = 1429
FROM calendar_item
WHERE caldav_data.dav_id = calendar_item.dav_id
AND caldav_data.collection_id = 10
AND rrule IS NULL
AND dtstart < '2010-01-01';


For that to work you would need to have created the 'archive' calendar
first, of course, and then something like:

SELECT dav_name, collection_id FROM collection
WHERE dav_name IN ( '/user1/home/', '/user1/archive/');

Obviously you could replace that in above, but makes the SQL a bit more
obscure...

UPDATE caldav_data
SET dav_name = replace( caldav_data.dav_name, '/user1/home/', '/user1/archive/'),
collection_id = (SELECT collection_id FROM collection
WHERE dav_name = '/user1/archive/')
FROM calendar_item
WHERE caldav_data.dav_id = calendar_item.dav_id
AND caldav_data.collection_id = (SELECT collection_id FROM collection
WHERE dav_name = '/user1/home/')
AND rrule IS NULL
AND dtstart < '2010-01-01';


Unfortunately repeating events in this way is *a lot* more complicated,
because you only want to move the ones that have finished, which means
expanding all the instances and working out whether they have
finished... It's probably easier to just manually move them in the
calendar itself.

Cheers,
Andrew.

------------------------------------------------------------------------
andrew (AT) morphoss (DOT) com +64(272)DEBIAN
You will be run over by a beer truck.
------------------------------------------------------------------------


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://lists.morphoss.com/pipermail/davical-users/attachments/20100308/850de111/attachment.pgp>
-------------- next part --------------
Looking a way to archive between date a collection to another [ In reply to ]
On 03/07/2010 08:44 PM, Andrew McMillan wrote:
> On Sun, 2010-03-07 at 14:08 +0100, Bruno Friedmann wrote:
>> Hi Guys,
>>
>> I've got here (due to super slow sunbird perf) a question about
>> putting old events from a collection to
>> another one.
>>
>> I imagine that's better to do this directly in postgresql as normal
>> ical client ask event by event (there's 9'000 to move :-)
>>
>> I would have done some research on my own yesterday and today, but
>> I've crash my car yesterday on ice. And been busy to organize
>> fixing, doing insurance things etc ..
>>
>> So If someone has some shortcuts to help me building a good sql, it
>> will be really appreciate.
>
> Hi Bruno,
>
> You need to update caldav_data table (calendar_item will be maintained
> with a trigger) with the new path, and the new collection_id.
>
> Here's an example moving non-repeating events, prior to the 1st of
> January, from /user1/home/ (collection_id 10 for me) to /user1/archive/
> (collection_id 1429 for me):
>
>
> UPDATE caldav_data
> SET dav_name = replace( caldav_data.dav_name, '/user1/home/', '/user1/archive/'),
> collection_id = 1429
> FROM calendar_item
> WHERE caldav_data.dav_id = calendar_item.dav_id
> AND caldav_data.collection_id = 10
> AND rrule IS NULL
> AND dtstart < '2010-01-01';
>
>
> For that to work you would need to have created the 'archive' calendar
> first, of course, and then something like:
>
> SELECT dav_name, collection_id FROM collection
> WHERE dav_name IN ( '/user1/home/', '/user1/archive/');
>
> Obviously you could replace that in above, but makes the SQL a bit more
> obscure...
>
> UPDATE caldav_data
> SET dav_name = replace( caldav_data.dav_name, '/user1/home/', '/user1/archive/'),
> collection_id = (SELECT collection_id FROM collection
> WHERE dav_name = '/user1/archive/')
> FROM calendar_item
> WHERE caldav_data.dav_id = calendar_item.dav_id
> AND caldav_data.collection_id = (SELECT collection_id FROM collection
> WHERE dav_name = '/user1/home/')
> AND rrule IS NULL
> AND dtstart < '2010-01-01';
>
>
> Unfortunately repeating events in this way is *a lot* more complicated,
> because you only want to move the ones that have finished, which means
> expanding all the instances and working out whether they have
> finished... It's probably easier to just manually move them in the
> calendar itself.
>
> Cheers,
> Andrew.
>
> ------------------------------------------------------------------------
> andrew (AT) morphoss (DOT) com +64(272)DEBIAN
> You will be run over by a beer truck.
> ------------------------------------------------------------------------
>

Thanks you very much Andrea, I will adapt this to my situation tomorrow

If it works ( no doubt about that ) I will try to add this use case to the wiki.
As it could become a needed classical task to davical hostmaster .


--

Bruno Friedmann
Looking a way to archive between date a collection to another [ In reply to ]
On Sun, 2010-03-07 at 22:02 +0100, Bruno Friedmann wrote:
>
> Thanks you very much Andrew, I will adapt this to my situation tomorrow
>
> If it works ( no doubt about that ) I will try to add this use case to the wiki.
> As it could become a needed classical task to davical hostmaster .

Yes, in fact I was so proud of my bit of research there that I already
pasted that e-mail into the wiki here:

http://wiki.davical.org/w/Move_events_to_a_different_collection

Feel free to extend it with actual scriptiness or such though :-)

In fact a little birdie told me that a python programmer somewhere is
well down the track towards publicly releasing a command-line admin
tool, and I'm pretty sure he could add that sort of function into it :-)

Cheers,
Andrew.

------------------------------------------------------------------------
andrew (AT) morphoss (DOT) com +64(272)DEBIAN
An avocado-tone refrigerator would look good on your resume.
------------------------------------------------------------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://lists.morphoss.com/pipermail/davical-users/attachments/20100308/ab058f6e/attachment-0001.pgp>
-------------- next part --------------
Looking a way to archive between date a collection to another [ In reply to ]
On 03/07/2010 11:09 PM, Andrew McMillan wrote:
> On Sun, 2010-03-07 at 22:02 +0100, Bruno Friedmann wrote:
>>
>> Thanks you very much Andrew, I will adapt this to my situation tomorrow
>>
>> If it works ( no doubt about that ) I will try to add this use case to the wiki.
>> As it could become a needed classical task to davical hostmaster .
>
> Yes, in fact I was so proud of my bit of research there that I already
> pasted that e-mail into the wiki here:
>
> http://wiki.davical.org/w/Move_events_to_a_different_collection
>
> Feel free to extend it with actual scriptiness or such though :-)
>
> In fact a little birdie told me that a python programmer somewhere is
> well down the track towards publicly releasing a command-line admin
> tool, and I'm pretty sure he could add that sort of function into it :-)
>
> Cheers,
> Andrew.
>

Ok just I've give this a try after a upgrade to the 0.9.8.4 davical / awl 0.42

Everything seems to have been done correctly with sql.
The difference between what you described in your sql and what I have to do is :
I want to move all event from /green/home/ ( a ressource collection ) to /archives/home/ ( also a ressource collection )

From what I've seen : each record has been correctly updated in db. But nor sunbird or kontact can display the archives/home
collection ?
When I try to download directly caldav.php/archives/home/ I only got the 3 events I've moved with sunbird.

There's something I miss, and really don't understand what is missing in DB.

in dav_data

1001;"/archives/home/0c37749c-164d-41c5-bc6a-168ed8152c96.ics";"fe94438f65ea3cbfd4e179b504b15919";"2009-07-10
20:23:53.858589+02";"2009-07-10 20:23:53.858589+02";"BEGIN:VCALENDAR
PRODID:-//davical.org//NONSGML AWL Calendar//EN
VERSION:2.0
CALSCALE:GREGORIAN
BEGIN:VEVENT
CREATED:20081112T061635Z
LAST-MODIFIED:20081112T061705Z
DTSTAMP:20081112T061705Z
UID:0c37749c-164d-41c5-bc6a-168ed8152c96
SUMMARY:Pesenti Paolo\, finition\, 9.30h
DTSTART;VALUE=DATE:20081117
DTEND;VALUE=DATE:20081118
TRANSP:TRANSPARENT
X-MOZ-GENERATION:1
END:VEVENT
END:VCALENDAR
";"VEVENT";1;661;22373

in calendar_item

1001;"/archives/home/0c37749c-164d-41c5-bc6a-168ed8152c96.ics";"fe94438f65ea3cbfd4e179b504b15919";"0c37749c-164d-41c5-bc6a-168ed8152c96";"2008-11-12
06:16:35";"2008-11-12 06:17:05";"2008-11-12 06:17:05";"2008-11-17 00:00:00+01";"2008-11-18 00:00:00+01";"";"Pesenti Paolo,
finition, 9.30h";"";"";;"PUBLIC";"TRANSPARENT";"";"";;"";"";"";661;22373

1001 : is the user ( one general user having all rights on all resources )
22373 : is the collection_id of archives resources

Did you have an idea ?

--

Bruno Friedmann

Ioda-Net S?rl
2830 Vellerat - Switzerland

T?l : ++41 32 435 7171
Fax : ++41 32 435 7172
gsm : ++41 78 802 6760
www.ioda-net.ch

Centre de Formation et de Coaching En Ligne
www.cfcel.com
Looking a way to archive between date a collection to another [ In reply to ]
Perharps in my case ( changing collection but also user ) I need to update also the user_no ?
Can you confirm this.


On 03/10/2010 07:32 AM, Bruno Friedmann wrote:
> On 03/07/2010 11:09 PM, Andrew McMillan wrote:
>> On Sun, 2010-03-07 at 22:02 +0100, Bruno Friedmann wrote:
>>>
>>> Thanks you very much Andrew, I will adapt this to my situation tomorrow
>>>
>>> If it works ( no doubt about that ) I will try to add this use case to the wiki.
>>> As it could become a needed classical task to davical hostmaster .
>>
>> Yes, in fact I was so proud of my bit of research there that I already
>> pasted that e-mail into the wiki here:
>>
>> http://wiki.davical.org/w/Move_events_to_a_different_collection
>>
>> Feel free to extend it with actual scriptiness or such though :-)
>>
>> In fact a little birdie told me that a python programmer somewhere is
>> well down the track towards publicly releasing a command-line admin
>> tool, and I'm pretty sure he could add that sort of function into it :-)
>>
>> Cheers,
>> Andrew.
>>
>
> Ok just I've give this a try after a upgrade to the 0.9.8.4 davical / awl 0.42
>
> Everything seems to have been done correctly with sql.
> The difference between what you described in your sql and what I have to do is :
> I want to move all event from /green/home/ ( a ressource collection ) to /archives/home/ ( also a ressource collection )
>
> From what I've seen : each record has been correctly updated in db. But nor sunbird or kontact can display the archives/home
> collection ?
> When I try to download directly caldav.php/archives/home/ I only got the 3 events I've moved with sunbird.
>
> There's something I miss, and really don't understand what is missing in DB.
>
> in dav_data
>
> 1001;"/archives/home/0c37749c-164d-41c5-bc6a-168ed8152c96.ics";"fe94438f65ea3cbfd4e179b504b15919";"2009-07-10
> 20:23:53.858589+02";"2009-07-10 20:23:53.858589+02";"BEGIN:VCALENDAR
> PRODID:-//davical.org//NONSGML AWL Calendar//EN
> VERSION:2.0
> CALSCALE:GREGORIAN
> BEGIN:VEVENT
> CREATED:20081112T061635Z
> LAST-MODIFIED:20081112T061705Z
> DTSTAMP:20081112T061705Z
> UID:0c37749c-164d-41c5-bc6a-168ed8152c96
> SUMMARY:Pesenti Paolo\, finition\, 9.30h
> DTSTART;VALUE=DATE:20081117
> DTEND;VALUE=DATE:20081118
> TRANSP:TRANSPARENT
> X-MOZ-GENERATION:1
> END:VEVENT
> END:VCALENDAR
> ";"VEVENT";1;661;22373
>
> in calendar_item
>
> 1001;"/archives/home/0c37749c-164d-41c5-bc6a-168ed8152c96.ics";"fe94438f65ea3cbfd4e179b504b15919";"0c37749c-164d-41c5-bc6a-168ed8152c96";"2008-11-12
> 06:16:35";"2008-11-12 06:17:05";"2008-11-12 06:17:05";"2008-11-17 00:00:00+01";"2008-11-18 00:00:00+01";"";"Pesenti Paolo,
> finition, 9.30h";"";"";;"PUBLIC";"TRANSPARENT";"";"";;"";"";"";661;22373
>
> 1001 : is the user ( one general user having all rights on all resources )
> 22373 : is the collection_id of archives resources
>
> Did you have an idea ?
>


--

Bruno Friedmann

Ioda-Net S?rl
2830 Vellerat - Switzerland

T?l : ++41 32 435 7171
Fax : ++41 32 435 7172
gsm : ++41 78 802 6760
www.ioda-net.ch

Centre de Formation et de Coaching En Ligne
www.cfcel.com
Looking a way to archive between date a collection to another [ In reply to ]
On Wed, 2010-03-10 at 07:39 +0100, Bruno Friedmann wrote:
> Perharps in my case ( changing collection but also user ) I need to update also the user_no ?
> Can you confirm this.

Yes, that should be done also.

Also it would be a good idea, once complete, to do:

update collection SET dav_etag = md5(random()) Where ... changed
collection...

This will ensure that software like iCal or Lightning which check the
collection tag to see if they need to check for changes are
appropriately notified.

Cheers,
Andrew.
>
>
> On 03/10/2010 07:32 AM, Bruno Friedmann wrote:
> > On 03/07/2010 11:09 PM, Andrew McMillan wrote:
> >> On Sun, 2010-03-07 at 22:02 +0100, Bruno Friedmann wrote:
> >>>
> >>> Thanks you very much Andrew, I will adapt this to my situation tomorrow
> >>>
> >>> If it works ( no doubt about that ) I will try to add this use case to the wiki.
> >>> As it could become a needed classical task to davical hostmaster .
> >>
> >> Yes, in fact I was so proud of my bit of research there that I already
> >> pasted that e-mail into the wiki here:
> >>
> >> http://wiki.davical.org/w/Move_events_to_a_different_collection
> >>
> >> Feel free to extend it with actual scriptiness or such though :-)
> >>
> >> In fact a little birdie told me that a python programmer somewhere is
> >> well down the track towards publicly releasing a command-line admin
> >> tool, and I'm pretty sure he could add that sort of function into it :-)
> >>
> >> Cheers,
> >> Andrew.
> >>
> >
> > Ok just I've give this a try after a upgrade to the 0.9.8.4 davical / awl 0.42
> >
> > Everything seems to have been done correctly with sql.
> > The difference between what you described in your sql and what I have to do is :
> > I want to move all event from /green/home/ ( a ressource collection ) to /archives/home/ ( also a ressource collection )
> >
> > From what I've seen : each record has been correctly updated in db. But nor sunbird or kontact can display the archives/home
> > collection ?
> > When I try to download directly caldav.php/archives/home/ I only got the 3 events I've moved with sunbird.
> >
> > There's something I miss, and really don't understand what is missing in DB.
> >
> > in dav_data
> >
> > 1001;"/archives/home/0c37749c-164d-41c5-bc6a-168ed8152c96.ics";"fe94438f65ea3cbfd4e179b504b15919";"2009-07-10
> > 20:23:53.858589+02";"2009-07-10 20:23:53.858589+02";"BEGIN:VCALENDAR
> > PRODID:-//davical.org//NONSGML AWL Calendar//EN
> > VERSION:2.0
> > CALSCALE:GREGORIAN
> > BEGIN:VEVENT
> > CREATED:20081112T061635Z
> > LAST-MODIFIED:20081112T061705Z
> > DTSTAMP:20081112T061705Z
> > UID:0c37749c-164d-41c5-bc6a-168ed8152c96
> > SUMMARY:Pesenti Paolo\, finition\, 9.30h
> > DTSTART;VALUE=DATE:20081117
> > DTEND;VALUE=DATE:20081118
> > TRANSP:TRANSPARENT
> > X-MOZ-GENERATION:1
> > END:VEVENT
> > END:VCALENDAR
> > ";"VEVENT";1;661;22373
> >
> > in calendar_item
> >
> > 1001;"/archives/home/0c37749c-164d-41c5-bc6a-168ed8152c96.ics";"fe94438f65ea3cbfd4e179b504b15919";"0c37749c-164d-41c5-bc6a-168ed8152c96";"2008-11-12
> > 06:16:35";"2008-11-12 06:17:05";"2008-11-12 06:17:05";"2008-11-17 00:00:00+01";"2008-11-18 00:00:00+01";"";"Pesenti Paolo,
> > finition, 9.30h";"";"";;"PUBLIC";"TRANSPARENT";"";"";;"";"";"";661;22373
> >
> > 1001 : is the user ( one general user having all rights on all resources )
> > 22373 : is the collection_id of archives resources
> >
> > Did you have an idea ?
> >
>
>

------------------------------------------------------------------------
andrew (AT) morphoss (DOT) com +64(272)DEBIAN
You will be held hostage by a radical group.
------------------------------------------------------------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://lists.morphoss.com/pipermail/davical-users/attachments/20100310/4c630959/attachment-0001.pgp>
-------------- next part --------------
Looking a way to archive between date a collection to another [ In reply to ]
Raaaah I'm facing a duplicate key

I've adapted the query like this one

UPDATE caldav_data
SET dav_name = regexp_replace( caldav_data.dav_name, '(/[a-z]+/)(home/)(.)', E'/archives/\\2\\3')
,collection_id = (SELECT collection_id FROM collection WHERE collection_id = 22483)
,user_no = (SELECT user_no FROM collection WHERE collection_id = 22483)
,dav_etag = md5(random()::text)
FROM calendar_item
WHERE caldav_data.dav_id = calendar_item.dav_id
AND caldav_data.collection_id IN
(SELECT collection_id FROM collection WHERE collection_id != 22483 )
AND rrule IS NULL
AND dtstart < '2010-01-01';

I've 10057 records on caldav_data, but if I ask a distinct like
select distinct(regexp_replace( caldav_data.dav_name, '(/[a-z]+/)(home/)(.)', E'/archives/\\2\\3')) as sid from caldav_data
it return only 10054

Is the special number after /user/collection/ shouldn't be unique ???


I've tried but without success to determine the 3 duplicate cases but I've not have success.
(must be too late here, and I need to get some sleep)
If some fayri have an idea during my sleep.



On 03/10/2010 10:44 AM, Andrew McMillan wrote:
> On Wed, 2010-03-10 at 07:39 +0100, Bruno Friedmann wrote:
>> Perharps in my case ( changing collection but also user ) I need to update also the user_no ?
>> Can you confirm this.
>
> Yes, that should be done also.
>
> Also it would be a good idea, once complete, to do:
>
> update collection SET dav_etag = md5(random()) Where ... changed
> collection...
>
> This will ensure that software like iCal or Lightning which check the
> collection tag to see if they need to check for changes are
> appropriately notified.
>
> Cheers,
> Andrew.
>>
>>
>> On 03/10/2010 07:32 AM, Bruno Friedmann wrote:
>>> On 03/07/2010 11:09 PM, Andrew McMillan wrote:
>>>> On Sun, 2010-03-07 at 22:02 +0100, Bruno Friedmann wrote:
>>>>>
>>>>> Thanks you very much Andrew, I will adapt this to my situation tomorrow
>>>>>
>>>>> If it works ( no doubt about that ) I will try to add this use case to the wiki.
>>>>> As it could become a needed classical task to davical hostmaster .
>>>>
>>>> Yes, in fact I was so proud of my bit of research there that I already
>>>> pasted that e-mail into the wiki here:
>>>>
>>>> http://wiki.davical.org/w/Move_events_to_a_different_collection
>>>>
>>>> Feel free to extend it with actual scriptiness or such though :-)
>>>>
>>>> In fact a little birdie told me that a python programmer somewhere is
>>>> well down the track towards publicly releasing a command-line admin
>>>> tool, and I'm pretty sure he could add that sort of function into it :-)
>>>>
>>>> Cheers,
>>>> Andrew.
>>>>
>>>
>>> Ok just I've give this a try after a upgrade to the 0.9.8.4 davical / awl 0.42
>>>
>>> Everything seems to have been done correctly with sql.
>>> The difference between what you described in your sql and what I have to do is :
>>> I want to move all event from /green/home/ ( a ressource collection ) to /archives/home/ ( also a ressource collection )
>>>
>>> From what I've seen : each record has been correctly updated in db. But nor sunbird or kontact can display the archives/home
>>> collection ?
>>> When I try to download directly caldav.php/archives/home/ I only got the 3 events I've moved with sunbird.
>>>
>>> There's something I miss, and really don't understand what is missing in DB.
>>>
>>> in dav_data
>>>
>>> 1001;"/archives/home/0c37749c-164d-41c5-bc6a-168ed8152c96.ics";"fe94438f65ea3cbfd4e179b504b15919";"2009-07-10
>>> 20:23:53.858589+02";"2009-07-10 20:23:53.858589+02";"BEGIN:VCALENDAR
>>> PRODID:-//davical.org//NONSGML AWL Calendar//EN
>>> VERSION:2.0
>>> CALSCALE:GREGORIAN
>>> BEGIN:VEVENT
>>> CREATED:20081112T061635Z
>>> LAST-MODIFIED:20081112T061705Z
>>> DTSTAMP:20081112T061705Z
>>> UID:0c37749c-164d-41c5-bc6a-168ed8152c96
>>> SUMMARY:Pesenti Paolo\, finition\, 9.30h
>>> DTSTART;VALUE=DATE:20081117
>>> DTEND;VALUE=DATE:20081118
>>> TRANSP:TRANSPARENT
>>> X-MOZ-GENERATION:1
>>> END:VEVENT
>>> END:VCALENDAR
>>> ";"VEVENT";1;661;22373
>>>
>>> in calendar_item
>>>
>>> 1001;"/archives/home/0c37749c-164d-41c5-bc6a-168ed8152c96.ics";"fe94438f65ea3cbfd4e179b504b15919";"0c37749c-164d-41c5-bc6a-168ed8152c96";"2008-11-12
>>> 06:16:35";"2008-11-12 06:17:05";"2008-11-12 06:17:05";"2008-11-17 00:00:00+01";"2008-11-18 00:00:00+01";"";"Pesenti Paolo,
>>> finition, 9.30h";"";"";;"PUBLIC";"TRANSPARENT";"";"";;"";"";"";661;22373
>>>
>>> 1001 : is the user ( one general user having all rights on all resources )
>>> 22373 : is the collection_id of archives resources
>>>
>>> Did you have an idea ?
>>>
>>
>>
>
> ------------------------------------------------------------------------
> andrew (AT) morphoss (DOT) com +64(272)DEBIAN
> You will be held hostage by a radical group.
> ------------------------------------------------------------------------
>


--

Bruno Friedmann

Ioda-Net S?rl
2830 Vellerat - Switzerland

T?l : ++41 32 435 7171
Fax : ++41 32 435 7172
gsm : ++41 78 802 6760
www.ioda-net.ch

Centre de Formation et de Coaching En Ligne
www.cfcel.com