Mailing List Archive

On Sun, 2008-09-14 at 22:30 -0700, david at lang.hm wrote:
> my second project is to try and modify rsyslog to recieve logs from an
> application. the application sends the logs over TCP and expects an
> application-level handshake (very similar to relp). at the moment I want
> to try and avoid having to change the application (many different products
> with different release schedules), and instead teach rsyslog to deal with
> the existing log format.
>
> at the moment I am trying to understand the imtcp module, but I am getting
> lost in the callbacks. it looks like rsyslog is calling a routine in
> imtcp, which calls a routine in tcpsrv, which calls a routine somewhere
> else to actually recieve the log.

The imtcp module has a lot of history and too complex code. This all
stems back to various stages of GSSAPI integration. Things have been
simplified with the introduction of the transport stream layer, but the
imtcp module does not yet reflect this simplicity. So far, I am hesitant
to split these things, because we still do not have a clean gssapi
netstream driver.

I can tell you where you need to provide callbacks ... or you could
start from imrelp, which in regard means mostly from scratch, but
actually receiving tcp isn't rocket science, so it may be easier to
start with a proper template without tcp functionality and integrate
your tcp reception code into it. What do you (think ;)) you prefer?

> ideally what I would like to use is to take imtcp and replace the message
> recieving/parsing logic with my own, then have it submit the parsed
> message into the queue (which looks like it would be via the SubmitMsg()
> call).
>
> but at the moment I am lost in the twisty maze of function calls between
> source files, all of which look different.
>
> I'm also not clear on what fields inside of the Msg structure need to be
> populated. looking at Msg.c/h I see a lot of fields there, but it looks
> like many/most of them are optional.

look at imfile. This is what you need:

CHKiRet(msgConstruct(&pMsg));
MsgSetFlowControlType(pMsg, eFLOWCTL_FULL_DELAY);
MsgSetInputName(pMsg, "imfile");
MsgSetUxTradMsg(pMsg, (char*)rsCStrGetSzStr(cstrLine));
MsgSetRawMsg(pMsg, (char*)rsCStrGetSzStr(cstrLine));
MsgSetMSG(pMsg, (char*)rsCStrGetSzStr(cstrLine));
MsgSetHOSTNAME(pMsg, (char*)glbl.GetLocalHostName());
MsgSetTAG(pMsg, (char*)pInfo->pszTag);
pMsg->iFacility = LOG_FAC(pInfo->iFacility);
pMsg->iSeverity = LOG_PRI(pInfo->iSeverity);
pMsg->bParseHOSTNAME = 0;
datetime.getCurrTime(&(pMsg->tTIMESTAMP)); /* use the current time! */
CHKiRet(submitMsg(pMsg))

Also look at imtemplate, which is a template input module. Sometimes it
is a bit outdated, if you have a problem during compile, tell me ;)

Rainer
>
> any pointers?
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog

On Mon, 15 Sep 2008, Rainer Gerhards wrote:

> On Sun, 2008-09-14 at 22:30 -0700, david at lang.hm wrote:
>> my second project is to try and modify rsyslog to recieve logs from an
>> application. the application sends the logs over TCP and expects an
>> application-level handshake (very similar to relp). at the moment I want
>> to try and avoid having to change the application (many different products
>> with different release schedules), and instead teach rsyslog to deal with
>> the existing log format.
>>
>> at the moment I am trying to understand the imtcp module, but I am getting
>> lost in the callbacks. it looks like rsyslog is calling a routine in
>> imtcp, which calls a routine in tcpsrv, which calls a routine somewhere
>> else to actually recieve the log.
>
> The imtcp module has a lot of history and too complex code. This all
> stems back to various stages of GSSAPI integration. Things have been
> simplified with the introduction of the transport stream layer, but the
> imtcp module does not yet reflect this simplicity. So far, I am hesitant
> to split these things, because we still do not have a clean gssapi
> netstream driver.
>
> I can tell you where you need to provide callbacks ... or you could
> start from imrelp, which in regard means mostly from scratch, but
> actually receiving tcp isn't rocket science, so it may be easier to
> start with a proper template without tcp functionality and integrate
> your tcp reception code into it. What do you (think ;)) you prefer?

the simpler the better. I'm pretty rusty with my C and I never did much
network or threaded programming in the first place, so while I can debug
programs in any language, I'm currently not much beyond the cut-n-past
stage here.

it sounds like I should not start from imtcp.

however when I looked at imrelp it looked like everything in in the relp
libraries, and I was hoping to avoid diving into them for the moment.

are any of the other tcp-based transports in better shape? or should I
start from scratch with the imtemplate plugin?

David Lang

>> ideally what I would like to use is to take imtcp and replace the message
>> recieving/parsing logic with my own, then have it submit the parsed
>> message into the queue (which looks like it would be via the SubmitMsg()
>> call).
>>
>> but at the moment I am lost in the twisty maze of function calls between
>> source files, all of which look different.
>>
>> I'm also not clear on what fields inside of the Msg structure need to be
>> populated. looking at Msg.c/h I see a lot of fields there, but it looks
>> like many/most of them are optional.
>
> look at imfile. This is what you need:
>
> CHKiRet(msgConstruct(&pMsg));
> MsgSetFlowControlType(pMsg, eFLOWCTL_FULL_DELAY);
> MsgSetInputName(pMsg, "imfile");
> MsgSetUxTradMsg(pMsg, (char*)rsCStrGetSzStr(cstrLine));
> MsgSetRawMsg(pMsg, (char*)rsCStrGetSzStr(cstrLine));
> MsgSetMSG(pMsg, (char*)rsCStrGetSzStr(cstrLine));
> MsgSetHOSTNAME(pMsg, (char*)glbl.GetLocalHostName());
> MsgSetTAG(pMsg, (char*)pInfo->pszTag);
> pMsg->iFacility = LOG_FAC(pInfo->iFacility);
> pMsg->iSeverity = LOG_PRI(pInfo->iSeverity);
> pMsg->bParseHOSTNAME = 0;
> datetime.getCurrTime(&(pMsg->tTIMESTAMP)); /* use the current time! */
> CHKiRet(submitMsg(pMsg))
>
> Also look at imtemplate, which is a template input module. Sometimes it
> is a bit outdated, if you have a problem during compile, tell me ;)
>
> Rainer
>>
>> any pointers?
>>
>> David Lang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>

On Mon, 2008-09-15 at 00:16 -0700, david at lang.hm wrote:
> the simpler the better. I'm pretty rusty with my C and I never did much
> network or threaded programming in the first place, so while I can debug
> programs in any language, I'm currently not much beyond the cut-n-past
> stage here.
>
> it sounds like I should not start from imtcp.
>
> however when I looked at imrelp it looked like everything in in the relp
> libraries, and I was hoping to avoid diving into them for the moment.
>
> are any of the other tcp-based transports in better shape? or should I
> start from scratch with the imtemplate plugin?

Bear a bit with me (but ping me if I don't reply by mid-week). I'll have
a look and recommend the way I think is best given the constraints.

Rainer

On Mon, 15 Sep 2008, Rainer Gerhards wrote:

> On Mon, 2008-09-15 at 00:16 -0700, david at lang.hm wrote:
>> the simpler the better. I'm pretty rusty with my C and I never did much
>> network or threaded programming in the first place, so while I can debug
>> programs in any language, I'm currently not much beyond the cut-n-past
>> stage here.
>>
>> it sounds like I should not start from imtcp.
>>
>> however when I looked at imrelp it looked like everything in in the relp
>> libraries, and I was hoping to avoid diving into them for the moment.
>>
>> are any of the other tcp-based transports in better shape? or should I
>> start from scratch with the imtemplate plugin?
>
> Bear a bit with me (but ping me if I don't reply by mid-week). I'll have
> a look and recommend the way I think is best given the constraints.

Ok, in the meantime I'll forget about trying to untangle imtcp and see
what I can do seperatly.

David Lang