Mailing List Archive

On Jul 12, 2011, at 1:09 PM, Nick Legg wrote:

> Can someone explain why an asset is republished and then expired upon deactivation? I've tried using bric_soap and a simple perl script using the bric API ($asset->deactivate()). I'm not understanding why the republish step is being created, but I definitely want the expire job to happen.

You likely have the EXPIRE_ON_DEACTIVATE bricolage.conf directive enabled, and that's what makes the republish jobs get scheduled when you call deactivate() && save().

> I'm trying to deactivate a rather large chunk of assets (20,000+) and don't want it to bog down my bric_queued for a couple hours.

It probably won't, though you should test it with a few. When those republish jobs are scheduled, the "expire" boolean should be set to TRUE in the database. When the job executes, it won't actually run the story through a template and distribute it, but will just delete it from the destination server.

That's the theory anyway. Poke at it a bit to confirm I'm not lying.

Best,

David

On 7/13/2011 2:15 AM, David E. Wheeler wrote:
> On Jul 12, 2011, at 1:09 PM, Nick Legg wrote:
>
>> Can someone explain why an asset is republished and then expired upon deactivation? I've tried using bric_soap and a simple perl script using the bric API ($asset->deactivate()). I'm not understanding why the republish step is being created, but I definitely want the expire job to happen.
> You likely have the EXPIRE_ON_DEACTIVATE bricolage.conf directive enabled, and that's what makes the republish jobs get scheduled when you call deactivate()&& save().

This is correct; I do have EXPIRE_ON_DEACTIVATE enabled.
>> I'm trying to deactivate a rather large chunk of assets (20,000+) and don't want it to bog down my bric_queued for a couple hours.
> It probably won't, though you should test it with a few. When those republish jobs are scheduled, the "expire" boolean should be set to TRUE in the database. When the job executes, it won't actually run the story through a template and distribute it, but will just delete it from the destination server.
>
> That's the theory anyway. Poke at it a bit to confirm I'm not lying.

I've re-published and re-expired a few stories to test this. Upon
ordering a story to deactivate, an Expire job is created. The 'expire'
field is true for this job. This Expire job splits into n Distribute
jobs (one for each of my Output Channels, but that's beside the point).
The Distribute jobs actually execute and deliver the file to the
destination server. The 'expire' field is false for these jobs. Once
this is complete, each Distribute job births a child Expire job (again,
one for each OC). The 'expire' field is true for these jobs. This is
where the actual deletion of the file from the server happens.

It seems odd to me that Bricolage distributes the file(s) immediately
before expiring them when deactivating an asset; on the other hand, when
a scheduled expire job eventually transpires it does not pre-distribute
the file prior to expiring it (I see failed expires occasionally due to
this behavior, where it cannot delete the file because the file does not
exist on the destination server).

-Nick

On Jul 14, 2011, at 5:41 AM, Nick Legg wrote:

> I've re-published and re-expired a few stories to test this. Upon ordering a story to deactivate, an Expire job is created. The 'expire' field is true for this job. This Expire job splits into n Distribute jobs (one for each of my Output Channels, but that's beside the point). The Distribute jobs actually execute and deliver the file to the destination server. The 'expire' field is false for these jobs. Once this is complete, each Distribute job births a child Expire job (again, one for each OC). The 'expire' field is true for these jobs. This is where the actual deletion of the file from the server happens.

Ah, it sounds like it needs to create a Distribution job as an expire job, rather than a Publish job as an expire job. Would you file a bug report for this (with the above details)?

> It seems odd to me that Bricolage distributes the file(s) immediately before expiring them when deactivating an asset; on the other hand, when a scheduled expire job eventually transpires it does not pre-distribute the file prior to expiring it (I see failed expires occasionally due to this behavior, where it cannot delete the file because the file does not exist on the destination server).

Hrm. Maybe that shouldn't fail…

Best,

David

On 7/14/2011 12:37 PM, David E. Wheeler wrote:
> On Jul 14, 2011, at 5:41 AM, Nick Legg wrote:
>
>> I've re-published and re-expired a few stories to test this. Upon ordering a story to deactivate, an Expire job is created. The 'expire' field is true for this job. This Expire job splits into n Distribute jobs (one for each of my Output Channels, but that's beside the point). The Distribute jobs actually execute and deliver the file to the destination server. The 'expire' field is false for these jobs. Once this is complete, each Distribute job births a child Expire job (again, one for each OC). The 'expire' field is true for these jobs. This is where the actual deletion of the file from the server happens.
> Ah, it sounds like it needs to create a Distribution job as an expire job, rather than a Publish job as an expire job. Would you file a bug report for this (with the above details)?

Upon investigating Burner.pm, I've found something a bit odd. See line
1314:
https://github.com/bricoleurs/bricolage/blob/master/lib/Bric/Util/Burner.pm#L1314.
This if case is false when I deactivate something, so the Burner goes
head and creates a Distribute job. Then look down at line 1375:
https://github.com/bricoleurs/bricolage/blob/master/lib/Bric/Util/Burner.pm#L1375.
This is where the Expire job gets set up, because the expire date is set
on an asset by the deactivate function.

Is there a reason to Distribute the asset, or can I fix that if case on
line 1314 to catch deactivate() calls without any trouble?

>
>> It seems odd to me that Bricolage distributes the file(s) immediately before expiring them when deactivating an asset; on the other hand, when a scheduled expire job eventually transpires it does not pre-distribute the file prior to expiring it (I see failed expires occasionally due to this behavior, where it cannot delete the file because the file does not exist on the destination server).
> Hrm. Maybe that shouldn't fail…
Dunno. It fails with a Net::SSH2 error for me: Error deleting
'/usr/local/htdocs/example.html' on 'preview.denison.edu': 2
SSH_FX_NO_SUCH_FILE

On Jul 14, 2011, at 12:03 PM, Nick Legg wrote:

> Upon investigating Burner.pm, I've found something a bit odd. See line 1314: https://github.com/bricoleurs/bricolage/blob/master/lib/Bric/Util/Burner.pm#L1314. This if case is false when I deactivate something, so the Burner goes head and creates a Distribute job. Then look down at line 1375: https://github.com/bricoleurs/bricolage/blob/master/lib/Bric/Util/Burner.pm#L1375. This is where the Expire job gets set up, because the expire date is set on an asset by the deactivate function.
>
> Is there a reason to Distribute the asset, or can I fix that if case on line 1314 to catch deactivate() calls without any trouble?

Don't know. I believe there are tests, maybe you could squeeze something in to make sure this case is tested.

> Dunno. It fails with a Net::SSH2 error for me: Error deleting '/usr/local/htdocs/example.html' on 'preview.denison.edu': 2 SSH_FX_NO_SUCH_FILE

I think we should probably silently ignore that error, don't you? We ignore this case in file system copies. Seems to me that if the file isn't there, there's nothing to delete, so it's not an error. Right?

If there's agreement on that, probably all of the Trans classes should be made to adhere to this behavior.

Best,

David

On 7/15/2011 12:14 PM, David E. Wheeler wrote:
>> Dunno. It fails with a Net::SSH2 error for me: Error deleting '/usr/local/htdocs/example.html' on 'preview.denison.edu': 2 SSH_FX_NO_SUCH_FILE
> I think we should probably silently ignore that error, don't you? We ignore this case in file system copies. Seems to me that if the file isn't there, there's nothing to delete, so it's not an error. Right?
>
> If there's agreement on that, probably all of the Trans classes should be made to adhere to this behavior.
I'll get on board with that. I've filed an issue in Lighthouse here:
http://bricolage.lighthouseapp.com/projects/29601-bricolage/tickets/275

-Nick

On Jul 15, 2011, at 10:35 AM, Nick Legg wrote:

>> If there's agreement on that, probably all of the Trans classes should be made to adhere to this behavior.
> I'll get on board with that. I've filed an issue in Lighthouse here: http://bricolage.lighthouseapp.com/projects/29601-bricolage/tickets/275

Great, thanks!

David

On 7/15/2011 12:14 PM, David E. Wheeler wrote:
> On Jul 14, 2011, at 12:03 PM, Nick Legg wrote:
>
>> Upon investigating Burner.pm, I've found something a bit odd. See line 1314: https://github.com/bricoleurs/bricolage/blob/master/lib/Bric/Util/Burner.pm#L1314. This if case is false when I deactivate something, so the Burner goes head and creates a Distribute job. Then look down at line 1375: https://github.com/bricoleurs/bricolage/blob/master/lib/Bric/Util/Burner.pm#L1375. This is where the Expire job gets set up, because the expire date is set on an asset by the deactivate function.
>>
>> Is there a reason to Distribute the asset, or can I fix that if case on line 1314 to catch deactivate() calls without any trouble?
> Don't know. I believe there are tests, maybe you could squeeze something in to make sure this case is tested.
I changed line 1314
(https://github.com/bricoleurs/bricolage/blob/master/lib/Bric/Util/Burner.pm#L1314)
to read:

if ($exp_date && ($exp_date lt $publish_date ||
!$ba->is_active())) {

And bulk-expired my assets. The expire jobs have completed and there
haven't been any noticeable issues so far. I'll test and commit the
code to my mirror when time permits.

On Jul 19, 2011, at 12:44 PM, Nick Legg wrote:

> I changed line 1314 (https://github.com/bricoleurs/bricolage/blob/master/lib/Bric/Util/Burner.pm#L1314) to read:
>
> if ($exp_date && ($exp_date lt $publish_date || !$ba->is_active())) {

Do you mean 1312?

> And bulk-expired my assets. The expire jobs have completed and there haven't been any noticeable issues so far. I'll test and commit the code to my mirror when time permits.

Great, that looks good.

Best,

David

On 7/19/2011 5:32 PM, David E. Wheeler wrote:
> On Jul 19, 2011, at 12:44 PM, Nick Legg wrote:
>
>> I changed line 1314 (https://github.com/bricoleurs/bricolage/blob/master/lib/Bric/Util/Burner.pm#L1314) to read:
>>
>> if ($exp_date&& ($exp_date lt $publish_date || !$ba->is_active())) {
> Do you mean 1312?
Hmm... yes. Although my browser highlights that line as 1314 on GitHub,
it's 1312 on the filesystem.