Mailing List Archive

On Mar 10, 2011, at 8:34 AM, Adam Wilson wrote:

> First question, anyone already built an Amazon S3 mover for Bricolage that they are willing to share?
>
> If not, I might give it a whirl. Looks like it is probably way over my head, but I used to say that about Perl, and now I can print a pretty mean "Hello World" page. I see the directions on how to add the mover here -> http://bricolagecms.org/docs/current/api/?Bric::Dist::Action::Mover, is there anywhere else I might look for additional tips and tricks?

Just look at the code in the other movers.

> Is there also a way to add additional parameters to either the Server Profile or to the Destination Profile? It would be really nice to be able to add a set of properties on the Server Profile page based on the type of Move Method specified in the Destination Profile.
> I suppose lacking this, I will also need to figure out how look up values in bricolage.conf.

Yes. See the email mover.

Best,

David

Any progress with this? I'm going to be making one as well. Was trying to
just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs
can't handle the way perl is trying to save files (and create directory
structure), so that's out.

best,
John Durkin

On Thu, Mar 10, 2011 at 9:04 AM, David E. Wheeler <david@kineticode.com>wrote:

> On Mar 10, 2011, at 8:34 AM, Adam Wilson wrote:
>
> > First question, anyone already built an Amazon S3 mover for Bricolage
> that they are willing to share?
> >
> > If not, I might give it a whirl. Looks like it is probably way over my
> head, but I used to say that about Perl, and now I can print a pretty mean
> "Hello World" page. I see the directions on how to add the mover here ->
> http://bricolagecms.org/docs/current/api/?Bric::Dist::Action::Mover, is
> there anywhere else I might look for additional tips and tricks?
>
> Just look at the code in the other movers.
>
> > Is there also a way to add additional parameters to either the Server
> Profile or to the Destination Profile? It would be really nice to be able
> to add a set of properties on the Server Profile page based on the type of
> Move Method specified in the Destination Profile.
> > I suppose lacking this, I will also need to figure out how look up values
> in bricolage.conf.
>
> Yes. See the email mover.
>
> Best,
>
> David
>
>
>

On Mar 21, 2011, at 12:11 PM, John Durkin wrote:

> Any progress with this? I'm going to be making one as well. Was trying to
> just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs
> can't handle the way perl is trying to save files (and create directory
> structure), so that's out.

Something should be do-able with Net::Amazon::S3.

Best,

David

On Mar 21, 2011, at 12:11 PM, John Durkin wrote:

> Any progress with this? I'm going to be making one as well. Was trying to
> just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs
> can't handle the way perl is trying to save files (and create directory
> structure), so that's out.

Something should be do-able with Net::Amazon::S3.

Best,

David

For the time being, we are using the filesystem mover with a fuse mount without any issues. I haven't had any issues that I have noticed with this approach, but I only just implemented this a couple of weeks ago, perhaps there are issues I am just unaware of.

I started looking into creating the new mover, but I was immediately overwhelmed by the task at hand. I am planning on using the Net::Amazon::S3,as David stated, but I am not the best perl programmer and it is going to take some time to get anything close to as sophisticated as the existing movers in place. I will keep the thread informed as I make progress though. It will help to motivate me to know that I am not the only one interested in this capability.

Adam

From: John Durkin [mailto:john.durkin@gmail.com]
Sent: Monday, March 21, 2011 3:12 PM
To: users@lists.bricolage.cc
Cc: Adam Wilson
Subject: Re: New amazon S3 mover

Any progress with this? I'm going to be making one as well. Was trying to just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs can't handle the way perl is trying to save files (and create directory structure), so that's out.

best,
John Durkin
On Thu, Mar 10, 2011 at 9:04 AM, David E. Wheeler <david@kineticode.com<mailto:david@kineticode.com>> wrote:
On Mar 10, 2011, at 8:34 AM, Adam Wilson wrote:

> First question, anyone already built an Amazon S3 mover for Bricolage that they are willing to share?
>
> If not, I might give it a whirl. Looks like it is probably way over my head, but I used to say that about Perl, and now I can print a pretty mean "Hello World" page. I see the directions on how to add the mover here -> http://bricolagecms.org/docs/current/api/?Bric::Dist::Action::Mover, is there anywhere else I might look for additional tips and tricks?
Just look at the code in the other movers.

> Is there also a way to add additional parameters to either the Server Profile or to the Destination Profile? It would be really nice to be able to add a set of properties on the Server Profile page based on the type of Move Method specified in the Destination Profile.
> I suppose lacking this, I will also need to figure out how look up values in bricolage.conf.
Yes. See the email mover.

Best,

David

oh, that's encouraging to hear the fuse mount is working for you... version
of fuse are you using, and on what platform? what version of s3fs?

On Tue, Mar 22, 2011 at 8:23 AM, Adam Wilson <adam@rfxtechnologies.com>wrote:

> For the time being, we are using the filesystem mover with a fuse mount
> without any issues. I haven’t had any issues that I have noticed with this
> approach, but I only just implemented this a couple of weeks ago, perhaps
> there are issues I am just unaware of.
>
>
>
> I started looking into creating the new mover, but I was immediately
> overwhelmed by the task at hand. I am planning on using the
> Net::Amazon::S3,as David stated, but I am not the best perl programmer and
> it is going to take some time to get anything close to as sophisticated as
> the existing movers in place. I will keep the thread informed as I make
> progress though. It will help to motivate me to know that I am not the only
> one interested in this capability.
>
>
>
> Adam
>
>
>
> *From:* John Durkin [mailto:john.durkin@gmail.com]
> *Sent:* Monday, March 21, 2011 3:12 PM
> *To:* users@lists.bricolage.cc
> *Cc:* Adam Wilson
> *Subject:* Re: New amazon S3 mover
>
>
>
> Any progress with this? I'm going to be making one as well. Was trying to
> just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs
> can't handle the way perl is trying to save files (and create directory
> structure), so that's out.
>
> best,
> John Durkin
>
> On Thu, Mar 10, 2011 at 9:04 AM, David E. Wheeler <david@kineticode.com>
> wrote:
>
> On Mar 10, 2011, at 8:34 AM, Adam Wilson wrote:
>
> > First question, anyone already built an Amazon S3 mover for Bricolage
> that they are willing to share?
> >
> > If not, I might give it a whirl. Looks like it is probably way over my
> head, but I used to say that about Perl, and now I can print a pretty mean
> "Hello World" page. I see the directions on how to add the mover here ->
> http://bricolagecms.org/docs/current/api/?Bric::Dist::Action::Mover, is
> there anywhere else I might look for additional tips and tricks?
>
> Just look at the code in the other movers.
>
>
> > Is there also a way to add additional parameters to either the Server
> Profile or to the Destination Profile? It would be really nice to be able
> to add a set of properties on the Server Profile page based on the type of
> Move Method specified in the Destination Profile.
> > I suppose lacking this, I will also need to figure out how look up values
> in bricolage.conf.
>
> Yes. See the email mover.
>
> Best,
>
> David
>
>
>

I used this page heavily for tips on getting it all working: http://code.google.com/p/s3fs/wiki/FuseOverAmazon

Linux 2.6.26-2-686 #1 SMP Thu Sep 16 19:35:51 UTC 2010 i686 GNU/Linux
Fuse: libfuse-dev_2.8.4-1.1_i386, libfuse2_2.8.4-1.1_i386
S3FS: s3fs-1.40

My fstab looks like this:
s3fs#shared.rfxwebsites.com /mnt/s3fs/shared.rfxwebsites.com fuse use_rrs=1,allow_other,nonempty,uid=www-data,gid=www-data,rw,auto,default_acl=public-read 0 0

I set up an additional mount for each website that we host that is related to a bucket on Amazon's S3 architecture.

Let me know if you have any more questions, I am happy to share my limited knowledge when I can.
Adam

From: John Durkin [mailto:john@j2d3.com]
Sent: Tuesday, March 22, 2011 12:09 PM
To: Adam Wilson
Cc: users@lists.bricolage.cc
Subject: Re: New amazon S3 mover

oh, that's encouraging to hear the fuse mount is working for you... version of fuse are you using, and on what platform? what version of s3fs?
On Tue, Mar 22, 2011 at 8:23 AM, Adam Wilson <adam@rfxtechnologies.com<mailto:adam@rfxtechnologies.com>> wrote:
For the time being, we are using the filesystem mover with a fuse mount without any issues. I haven't had any issues that I have noticed with this approach, but I only just implemented this a couple of weeks ago, perhaps there are issues I am just unaware of.

I started looking into creating the new mover, but I was immediately overwhelmed by the task at hand. I am planning on using the Net::Amazon::S3,as David stated, but I am not the best perl programmer and it is going to take some time to get anything close to as sophisticated as the existing movers in place. I will keep the thread informed as I make progress though. It will help to motivate me to know that I am not the only one interested in this capability.

Adam

From: John Durkin [mailto:john.durkin@gmail.com<mailto:john.durkin@gmail.com>]
Sent: Monday, March 21, 2011 3:12 PM
To: users@lists.bricolage.cc<mailto:users@lists.bricolage.cc>
Cc: Adam Wilson
Subject: Re: New amazon S3 mover

Any progress with this? I'm going to be making one as well. Was trying to just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs can't handle the way perl is trying to save files (and create directory structure), so that's out.

best,
John Durkin
On Thu, Mar 10, 2011 at 9:04 AM, David E. Wheeler <david@kineticode.com<mailto:david@kineticode.com>> wrote:
On Mar 10, 2011, at 8:34 AM, Adam Wilson wrote:

> First question, anyone already built an Amazon S3 mover for Bricolage that they are willing to share?
>
> If not, I might give it a whirl. Looks like it is probably way over my head, but I used to say that about Perl, and now I can print a pretty mean "Hello World" page. I see the directions on how to add the mover here -> http://bricolagecms.org/docs/current/api/?Bric::Dist::Action::Mover, is there anywhere else I might look for additional tips and tricks?
Just look at the code in the other movers.

> Is there also a way to add additional parameters to either the Server Profile or to the Destination Profile? It would be really nice to be able to add a set of properties on the Server Profile page based on the type of Move Method specified in the Destination Profile.
> I suppose lacking this, I will also need to figure out how look up values in bricolage.conf.
Yes. See the email mover.

Best,

David

On Mar 22, 2011, at 11:23 AM, Adam Wilson wrote:

> For the time being, we are using the filesystem mover with a fuse mount without any issues. I haven't had any issues that I have noticed with this approach, but I only just implemented this a couple of weeks ago, perhaps there are issues I am just unaware of.

Seems like a perfectly reasonable approach.

> I started looking into creating the new mover, but I was immediately overwhelmed by the task at hand. I am planning on using the Net::Amazon::S3,as David stated, but I am not the best perl programmer and it is going to take some time to get anything close to as sophisticated as the existing movers in place. I will keep the thread informed as I make progress though. It will help to motivate me to know that I am not the only one interested in this capability.

Thanks, but if you have something working, there's obviously no rush. Take your time, we can help you to get it right.

Best,

David

On Mar 22, 2011, at 11:23 AM, Adam Wilson wrote:

> For the time being, we are using the filesystem mover with a fuse mount without any issues. I haven't had any issues that I have noticed with this approach, but I only just implemented this a couple of weeks ago, perhaps there are issues I am just unaware of.

Seems like a perfectly reasonable approach.

> I started looking into creating the new mover, but I was immediately overwhelmed by the task at hand. I am planning on using the Net::Amazon::S3,as David stated, but I am not the best perl programmer and it is going to take some time to get anything close to as sophisticated as the existing movers in place. I will keep the thread informed as I make progress though. It will help to motivate me to know that I am not the only one interested in this capability.

Thanks, but if you have something working, there's obviously no rush. Take your time, we can help you to get it right.

Best,

David

John, et. Al.,
I am *really* going to be doing this today, unless it has already been done.
2 questions, the first is please let me know if someone has already built this.
The second question, is there somewhere that features are being tracked in github? I see 27 issues there(seems like a small number) but not sure that features should go there. But if not there, then how do we know what features are out there and coordinate working on them?

Thanks for the advice,
Adam

From: John Durkin [mailto:john.durkin@gmail.com]
Sent: Monday, March 21, 2011 3:12 PM
To: users@lists.bricolage.cc
Cc: Adam Wilson
Subject: Re: New amazon S3 mover

Any progress with this? I'm going to be making one as well. Was trying to just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs can't handle the way perl is trying to save files (and create directory structure), so that's out.

best,
John Durkin
On Thu, Mar 10, 2011 at 9:04 AM, David E. Wheeler <david@kineticode.com<mailto:david@kineticode.com>> wrote:
On Mar 10, 2011, at 8:34 AM, Adam Wilson wrote:

> First question, anyone already built an Amazon S3 mover for Bricolage that they are willing to share?
>
> If not, I might give it a whirl. Looks like it is probably way over my head, but I used to say that about Perl, and now I can print a pretty mean "Hello World" page. I see the directions on how to add the mover here -> http://bricolagecms.org/docs/current/api/?Bric::Dist::Action::Mover, is there anywhere else I might look for additional tips and tricks?
Just look at the code in the other movers.

> Is there also a way to add additional parameters to either the Server Profile or to the Destination Profile? It would be really nice to be able to add a set of properties on the Server Profile page based on the type of Move Method specified in the Destination Profile.
> I suppose lacking this, I will also need to figure out how look up values in bricolage.conf.
Yes. See the email mover.

Best,

David

On Jul 13, 2011, at 8:02 AM, Adam Wilson wrote:

> John, et. Al.,
> I am *really* going to be doing this today, unless it has already been done.
> 2 questions, the first is please let me know if someone has already built this.

Not to my knowledge.

> The second question, is there somewhere that features are being tracked in github? I see 27 issues there(seems like a small number) but not sure that features should go there. But if not there, then how do we know what features are out there and coordinate working on them?

Lighthouse.

http://bricolage.lighthouseapp.com/

I don't see any open issues in the GitHub tracker, just a single pull request.

https://github.com/bricoleurs/bricolage/issues

That's how we coordinate pull requests, of course. See the "Getting the Source" section of https://github.com/bricoleurs/bricolage/wiki.

HTH,

David

On Jul 13, 2011, at 8:02 AM, Adam Wilson wrote:

> John, et. Al.,
> I am *really* going to be doing this today, unless it has already been done.
> 2 questions, the first is please let me know if someone has already built this.

Not to my knowledge.

> The second question, is there somewhere that features are being tracked in github? I see 27 issues there(seems like a small number) but not sure that features should go there. But if not there, then how do we know what features are out there and coordinate working on them?

Lighthouse.

http://bricolage.lighthouseapp.com/

I don't see any open issues in the GitHub tracker, just a single pull request.

https://github.com/bricoleurs/bricolage/issues

That's how we coordinate pull requests, of course. See the "Getting the Source" section of https://github.com/bricoleurs/bricolage/wiki.

HTH,

David

I built this - I have an EC2 image with a successful implementation of my S3
Mover... I did 99% of it - that last 1% - sharing it with anyone - that's
the part i gotta still do.

So - I'm spinning up that instance again (I'd stopped it, was not in use) -
I will refresh my memory as to the files I changed...

I am super busy, so Adam - if you contact me separately maybe we can figure
out a way for you to login to that instance and do the last 1% of the work?

JD

On Wed, Jul 13, 2011 at 9:13 AM, David E. Wheeler <david@kineticode.com>wrote:

> On Jul 13, 2011, at 8:02 AM, Adam Wilson wrote:
>
> > John, et. Al.,
> > I am *really* going to be doing this today, unless it has
> already been done.
> > 2 questions, the first is please let me know if someone
> has already built this.
>
> Not to my knowledge.
>
> > The second question, is there somewhere that features are
> being tracked in github? I see 27 issues there(seems like a small number)
> but not sure that features should go there. But if not there, then how do
> we know what features are out there and coordinate working on them?
>
> Lighthouse.
>
> http://bricolage.lighthouseapp.com/
>
> I don't see any open issues in the GitHub tracker, just a single pull
> request.
>
> https://github.com/bricoleurs/bricolage/issues
>
> That's how we coordinate pull requests, of course. See the "Getting the
> Source" section of https://github.com/bricoleurs/bricolage/wiki.
>
> HTH,
>
> David
>
>