Mailing List Archive

Announcing - the Assimilation monitoring system - a sub-project of Linux-HA
Hi,

Back last November or so, I started work on a new monitoring project -
for monitoring servers and services.

It's aims are:
- Scalable virtually without limit - tens of thousands of servers is
not a problem
- Easy installation and upkeep - includes integrated discovery of
servers, services
and switches - without setting off security alarms ;-)

This project isn't ready for a public release yet (it's in a fairly
early stage), but it seemed worthwhile to let others know that the
project exists, and start getting folks to read over the code, and
perhaps begin to play with it a bit as well.

The project has two arenas of operation:
nanoprobes - which run in (nearly) every monitored machine
Collective management - running in a central server (or HA cluster).

Current status:
- edge switch discovery code is complete
- scalable heartbeat code is complete
- Lots more to do :-D

Current code base is about 7K lines of C.

You can find documentation for the project here:
http://linux-ha.org/source-doc/assimilation/html/index.html

An overview of the architecture is found under the "System Goals and
Architecture" tab.

If you want to dive into the structure of the code in a sort-of top-down
way, you might explore under the "Modules" tab.

This documentation is high level (project aims), medium level (class
structure and descriptions), low level (APIs) and includes the code as
well. It's all done with Doxygen - which worked really well for this.

The code itself is available in the Linux-HA mercurial repository -
which you can find here:
http://hg.linux-ha.org/%7Cexperimental/assimilation/

This code includes all the documentation above - it's just not in quite
as pretty or organized a format.

My short term goal is to get the server monitoring completely up and
usable. Current thinking is that the central Collective management code
will be in Python, with the nanoprobes in 'C' (as they currently are).
It is expected that service monitoring will make use of the LRM - when
we get to that.

--
Alan Robertson<alanr@unix.sh>

"Openness is the foundation and preservative of friendship... Let me claim from you at all times your undisguised opinions." - William Wilberforce
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
Re: Announcing - the Assimilation monitoring system - a sub-project of Linux-HA [ In reply to ]
On Fri, Aug 12, 2011 at 03:11:36PM -0600, Alan Robertson wrote:
> Hi,
>
> Back last November or so, I started work on a new monitoring project -
> for monitoring servers and services.
>
> It's aims are:
> - Scalable virtually without limit - tens of thousands of servers is
> not a problem
> - Easy installation and upkeep - includes integrated discovery of
> servers, services
> and switches - without setting off security alarms ;-)
>
> This project isn't ready for a public release yet (it's in a fairly
> early stage), but it seemed worthwhile to let others know that the
> project exists, and start getting folks to read over the code, and
> perhaps begin to play with it a bit as well.
>
> The project has two arenas of operation:
> nanoprobes - which run in (nearly) every monitored machine

Why not matahari (http://matahariproject.org/)?

> Collective management - running in a central server (or HA cluster).

Quite simerlar to http://pacemaker-cloud.org/. Seems a
shame not to be working together.

-Angus
>
> Current status:
> - edge switch discovery code is complete
> - scalable heartbeat code is complete
> - Lots more to do :-D
>
> Current code base is about 7K lines of C.
>
> You can find documentation for the project here:
> http://linux-ha.org/source-doc/assimilation/html/index.html
>
> An overview of the architecture is found under the "System Goals and
> Architecture" tab.
>
> If you want to dive into the structure of the code in a sort-of top-down
> way, you might explore under the "Modules" tab.
>
> This documentation is high level (project aims), medium level (class
> structure and descriptions), low level (APIs) and includes the code as
> well. It's all done with Doxygen - which worked really well for this.
>
> The code itself is available in the Linux-HA mercurial repository -
> which you can find here:
> http://hg.linux-ha.org/%7Cexperimental/assimilation/
>
> This code includes all the documentation above - it's just not in quite
> as pretty or organized a format.
>
> My short term goal is to get the server monitoring completely up and
> usable. Current thinking is that the central Collective management code
> will be in Python, with the nanoprobes in 'C' (as they currently are).
> It is expected that service monitoring will make use of the LRM - when
> we get to that.
>
> --
> Alan Robertson<alanr@unix.sh>
>
> "Openness is the foundation and preservative of friendship... Let me claim from you at all times your undisguised opinions." - William Wilberforce
> _______________________________________________________
> Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
> Home Page: http://linux-ha.org/
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
Re: Announcing - the Assimilation monitoring system - a sub-project of Linux-HA [ In reply to ]
On 08/16/2011 05:08 PM, Angus Salkeld wrote:
> On Fri, Aug 12, 2011 at 03:11:36PM -0600, Alan Robertson wrote:
>> Hi,
>>
>> Back last November or so, I started work on a new monitoring project -
>> for monitoring servers and services.
>>
>> It's aims are:
>> - Scalable virtually without limit - tens of thousands of servers is
>> not a problem
>> - Easy installation and upkeep - includes integrated discovery of
>> servers, services
>> and switches - without setting off security alarms ;-)
>>
>> This project isn't ready for a public release yet (it's in a fairly
>> early stage), but it seemed worthwhile to let others know that the
>> project exists, and start getting folks to read over the code, and
>> perhaps begin to play with it a bit as well.
>>
>> The project has two arenas of operation:
>> nanoprobes - which run in (nearly) every monitored machine
> Why not matahari (http://matahariproject.org/)?
>
>> Collective management - running in a central server (or HA cluster).
> Quite simerlar to http://pacemaker-cloud.org/. Seems a
> shame not to be working together.
>
> -Angus
This is a set of ideas I've been working on for the last four years or
so. My most grandiose vision of it I called a Data Center Operating
System. This is about the same time that Amazon announced their first
cloud offering (unknown to me). There are a few hints about it a couple
of years ago in my blog.

I heard a little about Andrew's project when I announced this back in
November. Andrew has made it perfectly clear that he doesn't want to
work with me (really, absolutely, abundantly, perfectly, crystal clear)
and there is evidence that he doesn't work well with others besides me,
that's not a possibility.

In the short term I'm not specially concerned with clouds - just with
any collection of computers which range from 4 up to and above cloud
scale. That includes clouds of course - but we'll get a lot more users
at the small scale than we well at cloud scale.

There are several reasons for this approach:
- Existing monitoring software sucks.
- Many more collections of computers besides clouds exist and need
help - although this would work very well with clouds

This problem has dimensions that a cloud environment doesn't have. In a
cloud, all deployment is automated, so you can _know_ what is running
where. In a more conventional data center, having a way to discover
what's in your data center, and what's running on those servers is
important.

--
Alan Robertson<alanr@unix.sh>

"Openness is the foundation and preservative of friendship... Let me claim from you at all times your undisguised opinions." - William Wilberforce
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
Re: Announcing - the Assimilation monitoring system - a sub-project of Linux-HA [ In reply to ]
On Wed, Aug 17, 2011 at 09:51:14AM -0600, Alan Robertson wrote:
> On 08/16/2011 05:08 PM, Angus Salkeld wrote:
> > On Fri, Aug 12, 2011 at 03:11:36PM -0600, Alan Robertson wrote:
> >> Hi,
> >>
> >> Back last November or so, I started work on a new monitoring project -
> >> for monitoring servers and services.
> >>
> >> It's aims are:
> >> - Scalable virtually without limit - tens of thousands of servers is
> >> not a problem
> >> - Easy installation and upkeep - includes integrated discovery of
> >> servers, services
> >> and switches - without setting off security alarms ;-)
> >>
> >> This project isn't ready for a public release yet (it's in a fairly
> >> early stage), but it seemed worthwhile to let others know that the
> >> project exists, and start getting folks to read over the code, and
> >> perhaps begin to play with it a bit as well.
> >>
> >> The project has two arenas of operation:
> >> nanoprobes - which run in (nearly) every monitored machine
> > Why not matahari (http://matahariproject.org/)?
> >
> >> Collective management - running in a central server (or HA cluster).
> > Quite simerlar to http://pacemaker-cloud.org/. Seems a
> > shame not to be working together.
> >
> > -Angus
> This is a set of ideas I've been working on for the last four years or
> so. My most grandiose vision of it I called a Data Center Operating
> System. This is about the same time that Amazon announced their first
> cloud offering (unknown to me). There are a few hints about it a couple
> of years ago in my blog.
>
> I heard a little about Andrew's project when I announced this back in
> November. Andrew has made it perfectly clear that he doesn't want to
> work with me (really, absolutely, abundantly, perfectly, crystal clear)
> and there is evidence that he doesn't work well with others besides me,
> that's not a possibility.

Oops, seems like I have really stepped in it. Sorry for bringing this
up again.

>
> In the short term I'm not specially concerned with clouds - just with
> any collection of computers which range from 4 up to and above cloud
> scale. That includes clouds of course - but we'll get a lot more users
> at the small scale than we well at cloud scale.
>
> There are several reasons for this approach:
> - Existing monitoring software sucks.
> - Many more collections of computers besides clouds exist and need
> help - although this would work very well with clouds
>
> This problem has dimensions that a cloud environment doesn't have. In a
> cloud, all deployment is automated, so you can _know_ what is running
> where. In a more conventional data center, having a way to discover
> what's in your data center, and what's running on those servers is
> important.

Well technically this could be easily done in one project, but it seems
that's not going to happen (with everyone working on it).

-Angus

>
> --
> Alan Robertson<alanr@unix.sh>
>
> "Openness is the foundation and preservative of friendship... Let me claim from you at all times your undisguised opinions." - William Wilberforce
> _______________________________________________________
> Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
> Home Page: http://linux-ha.org/
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
Re: Announcing - the Assimilation monitoring system - a sub-project of Linux-HA [ In reply to ]
On 08/17/2011 07:16 PM, Angus Salkeld wrote:
> On Wed, Aug 17, 2011 at 09:51:14AM -0600, Alan Robertson wrote:
>> On 08/16/2011 05:08 PM, Angus Salkeld wrote:
>>> On Fri, Aug 12, 2011 at 03:11:36PM -0600, Alan Robertson wrote:
>>>> Hi,
>>>>
>>>> Back last November or so, I started work on a new monitoring project -
>>>> for monitoring servers and services.
>>>>
>>>> It's aims are:
>>>> - Scalable virtually without limit - tens of thousands of servers is
>>>> not a problem
>>>> - Easy installation and upkeep - includes integrated discovery of
>>>> servers, services
>>>> and switches - without setting off security alarms ;-)
>>>>
>>>> This project isn't ready for a public release yet (it's in a fairly
>>>> early stage), but it seemed worthwhile to let others know that the
>>>> project exists, and start getting folks to read over the code, and
>>>> perhaps begin to play with it a bit as well.
>>>>
>>>> The project has two arenas of operation:
>>>> nanoprobes - which run in (nearly) every monitored machine
>>> Why not matahari (http://matahariproject.org/)?
>>>
>>>> Collective management - running in a central server (or HA cluster).
>>> Quite simerlar to http://pacemaker-cloud.org/. Seems a
>>> shame not to be working together.
>>>
>>> -Angus
>> This is a set of ideas I've been working on for the last four years or
>> so. My most grandiose vision of it I called a Data Center Operating
>> System. This is about the same time that Amazon announced their first
>> cloud offering (unknown to me). There are a few hints about it a couple
>> of years ago in my blog.
>>
>> I heard a little about Andrew's project when I announced this back in
>> November. Andrew has made it perfectly clear that he doesn't want to
>> work with me (really, absolutely, abundantly, perfectly, crystal clear)
>> and there is evidence that he doesn't work well with others besides me,
>> that's not a possibility.
>
> Oops, seems like I have really stepped in it. Sorry for bringing this
> up again.
>
>>
>> In the short term I'm not specially concerned with clouds - just with
>> any collection of computers which range from 4 up to and above cloud
>> scale. That includes clouds of course - but we'll get a lot more users
>> at the small scale than we well at cloud scale.
>>
>> There are several reasons for this approach:
>> - Existing monitoring software sucks.
>> - Many more collections of computers besides clouds exist and need
>> help - although this would work very well with clouds
>>
>> This problem has dimensions that a cloud environment doesn't have. In a
>> cloud, all deployment is automated, so you can _know_ what is running
>> where. In a more conventional data center, having a way to discover
>> what's in your data center, and what's running on those servers is
>> important.
>
> Well technically this could be easily done in one project, but it seems
> that's not going to happen (with everyone working on it).
>
> -Angus

Linux is fairly described as an ecosystem. Differing branches and
methods of solving a given problem are tried, and the one with the most
backing and merit wins. It's part of what makes open-source what it is.
So, from my point of view, best of luck to both. :)

--
Digimer
E-Mail: digimer@alteeve.com
Freenode handle: digimer
Papers and Projects: http://alteeve.com
Node Assassin: http://nodeassassin.org
"At what point did we forget that the Space Shuttle was, essentially,
a program that strapped human beings to an explosion and tried to stab
through the sky with fire and math?"
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
Re: Announcing - the Assimilation monitoring system - a sub-project of Linux-HA [ In reply to ]
On 08/17/2011 09:15 PM, Digimer wrote:
>
> Linux is fairly described as an ecosystem. Differing branches and
> methods of solving a given problem are tried, and the one with the most
> backing and merit wins. It's part of what makes open-source what it is.
> So, from my point of view, best of luck to both. :)
Thanks!

And, I share your last point of view as well - I wish all parties good luck!

--
Alan Robertson<alanr@unix.sh>

"Openness is the foundation and preservative of friendship... Let me claim from you at all times your undisguised opinions." - William Wilberforce
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/