Mailing List Archive

[lvs-users] DTLS Load balancing
Hi,

  There is a draft[1] at the IETF about connection ID for DTLS . This
is a way to identify a "DTLS connection" by an ID instead of the
classical Ip address/port tuple. The objective is to reduce the need of
DTLS full handshake when client address/port change.

   I would like to know if it make sense to make load balancing based
on this connection ID.

   Here is the use case:
   You have a cluster of servers behind a unique IP address.
   You do load balancing using IP address.
   You use UDP/DTLS.
   Some clients are behind NAT and so theirs IP/port can change.
   DTLS connection states are store in each server and so are not shared.


   So if clients use same address/port, there is no issue as traffic
will be redirect always on the same server. Server has already a
connection for this peer, no need to full-handshake.
   If address/port change, 2 possibilities:
     - by chance load balancer, send traffic to the same server and
thanks to CID the server can reuse its connection, no-need to full-handshake
     - bad luck, traffic is redirect on server which does not know this
peer, so a new full-handshake is needed.

   It seems to me that doing load balancing on this connection ID could
solve the problem. [2]

   Does it make sense to you ? Is it a way to extends LVS to support
this kind of behavior ? I don't think so as LVS seems to be a 4-layers
load balancer, but I'm still interesting to know your opinions as your
seems to know better than me in load-balancing.

Thx

Simon

[1]https://tools.ietf.org/html/draft-rescorla-tls-dtls-connection-id-00
[2]https://www.ietf.org/mail-archive/web/tls/current/msg24619.html
<https://tools.ietf.org/html/draft-rescorla-tls-dtls-connection-id-00>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] DTLS Load balancing [ In reply to ]
Hello,

On Thu, 19 Oct 2017, Simon Bernard wrote:

> Hi,
>
>   There is a draft[1] at the IETF about connection ID for DTLS . This
> is a way to identify a "DTLS connection" by an ID instead of the
> classical Ip address/port tuple. The objective is to reduce the need of
> DTLS full handshake when client address/port change.
>
>    I would like to know if it make sense to make load balancing based
> on this connection ID.
>
>    Here is the use case:
>    You have a cluster of servers behind a unique IP address.
>    You do load balancing using IP address.
>    You use UDP/DTLS.
>    Some clients are behind NAT and so theirs IP/port can change.
>    DTLS connection states are store in each server and so are not shared.
>
>
>    So if clients use same address/port, there is no issue as traffic
> will be redirect always on the same server. Server has already a
> connection for this peer, no need to full-handshake.
>    If address/port change, 2 possibilities:
>      - by chance load balancer, send traffic to the same server and
> thanks to CID the server can reuse its connection, no-need to full-handshake
>      - bad luck, traffic is redirect on server which does not know this
> peer, so a new full-handshake is needed.
>
>    It seems to me that doing load balancing on this connection ID could
> solve the problem. [2]
>
>    Does it make sense to you ? Is it a way to extends LVS to support
> this kind of behavior ? I don't think so as LVS seems to be a 4-layers
> load balancer, but I'm still interesting to know your opinions as your
> seems to know better than me in load-balancing.

There is a similar example in IPVS: the SIP Persistence
Engine (PE in short), see ip_vs_pe_sip.c. The SIP PE details:

- every SIP message contains Call-ID that can be used
to schedule different packets in same UDP transport connection
to different real servers. Or many UDP connections with same
Call-ID to same real server.

- As IPVS first lookups for existing connection (by address/port)
and then if not found lookups for persistence template (which
forwards separate connections to specific real server by some
criteria defined by the PE) with the goal to create new
connection, the SIP PE does not keep the normal UDP connections
when OPS mode is selected (One-packet scheduling, the --ops option).
By this way, for every packet, the "ct_match" SIP PE method can
inspect the UDP payload and to maintain a context (persistence template
with pe_data) where the Call-ID is saved together with the real server
to be used for all packets with such Call-ID.

- if OPS mode is not set, same connection can not be used for
different Call-IDs.

If you want to go ahead with implementing ip_vs_pe_dtls.c
I can give you some hints to consider:

- if some DTLS protocol version does not contain the CID in
every packet, DTLS PE will have to fallback to address/port
identification. This should be decided for the first packet in
UDP connection because the dialog should be with the same
real server. I'm refering to this text in your [1] draft:

"In DTLS 1.2, connection ids are exchanged at the beginning of the
"DTLS session only."

So, we have 2 choices when first packet in UDP
connection is received:

1. Create and keep normal connection (without IP_VS_CONN_F_ONE_PACKET)
when CID can not be used or CID is present only early.
This means we can not inspect further this UDP connection for
CID until it is expired. The first lookup by addr/port will
find the existing connection for every following packet. Why we
need to keep the connection? Because we can use the initially
provided CID to get the needed real server but when next
packets without CID come, we should know where to forward them.
Or may be this can be handled by the "ct_match" PE method
and we can create only IP_VS_CONN_F_ONE_PACKET conns.

2. Create connection for the packet (IP_VS_CONN_F_ONE_PACKET)
but not based on svc->flags & IP_VS_SVC_F_ONEPACKET, this
time we should provide it from the "ct_match" PE method.
Connection is dropped after the packet is forwarded and
the next packets from same client port will come again to
the "ct_match" method. We will use this when CID will be
present in all packets from this client port.

As result, the PE should be able to maintain
persistence templates with or without pe_data (CID),
i.e. one per CID and additional without CID, all for the
same CIP:CPORT->VIP:VPORT->RIP:RPORT.

- According to RFC 6347 4.1.1. Transport Layer Mapping,
"Multiple DTLS records may be placed in a single datagram".
So, if multiple records contain different CIDs, this should
be considered. I can think for some fatal cases where
two records in one packet can be mapped to different
real servers.

Regards

--
Julian Anastasov <ja@ssi.bg>
Re: [lvs-users] DTLS Load balancing [ In reply to ]
Thx a lot Julian for this accurate response.

Just to let you know, this is a mid/long term idea, but more I think
about it more I like it.
CID is still a draft. Before to go to the LVS implementation we need to
have DTLS implementation which support DTLS CID.

All of this is really new for me. I have some questions before to start.
(newbie's questions :p)

If I well understand IPVS  is a module of the linux kernel.

So the source code is directly available in the kernel repository [1] ?

I should use ip_vs_pe_sip.c[2] as sample ?

I looked at internet resource for kernel development there is a lot ! 
Linux Kernel Newbies[1] seems good. Any advice about how to tests my
code ? Are you using virtualization or something like that ?

Thx again for you time.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/netfilter/ipvs

[2]
https://github.com/torvalds/linux/blob/master/net/netfilter/ipvs/ip_vs_pe_sip.c
[3] https://kernelnewbies.org/FirstKernelPatch

Le 21/10/2017 à 16:13, Julian Anastasov a écrit :
> Hello,
>
> On Thu, 19 Oct 2017, Simon Bernard wrote:
>
>> Hi,
>>
>>   There is a draft[1] at the IETF about connection ID for DTLS . This
>> is a way to identify a "DTLS connection" by an ID instead of the
>> classical Ip address/port tuple. The objective is to reduce the need of
>> DTLS full handshake when client address/port change.
>>
>>    I would like to know if it make sense to make load balancing based
>> on this connection ID.
>>
>>    Here is the use case:
>>    You have a cluster of servers behind a unique IP address.
>>    You do load balancing using IP address.
>>    You use UDP/DTLS.
>>    Some clients are behind NAT and so theirs IP/port can change.
>>    DTLS connection states are store in each server and so are not shared.
>>
>>
>>    So if clients use same address/port, there is no issue as traffic
>> will be redirect always on the same server. Server has already a
>> connection for this peer, no need to full-handshake.
>>    If address/port change, 2 possibilities:
>>      - by chance load balancer, send traffic to the same server and
>> thanks to CID the server can reuse its connection, no-need to full-handshake
>>      - bad luck, traffic is redirect on server which does not know this
>> peer, so a new full-handshake is needed.
>>
>>    It seems to me that doing load balancing on this connection ID could
>> solve the problem. [2]
>>
>>    Does it make sense to you ? Is it a way to extends LVS to support
>> this kind of behavior ? I don't think so as LVS seems to be a 4-layers
>> load balancer, but I'm still interesting to know your opinions as your
>> seems to know better than me in load-balancing.
> There is a similar example in IPVS: the SIP Persistence
> Engine (PE in short), see ip_vs_pe_sip.c. The SIP PE details:
>
> - every SIP message contains Call-ID that can be used
> to schedule different packets in same UDP transport connection
> to different real servers. Or many UDP connections with same
> Call-ID to same real server.
>
> - As IPVS first lookups for existing connection (by address/port)
> and then if not found lookups for persistence template (which
> forwards separate connections to specific real server by some
> criteria defined by the PE) with the goal to create new
> connection, the SIP PE does not keep the normal UDP connections
> when OPS mode is selected (One-packet scheduling, the --ops option).
> By this way, for every packet, the "ct_match" SIP PE method can
> inspect the UDP payload and to maintain a context (persistence template
> with pe_data) where the Call-ID is saved together with the real server
> to be used for all packets with such Call-ID.
>
> - if OPS mode is not set, same connection can not be used for
> different Call-IDs.
>
> If you want to go ahead with implementing ip_vs_pe_dtls.c
> I can give you some hints to consider:
>
> - if some DTLS protocol version does not contain the CID in
> every packet, DTLS PE will have to fallback to address/port
> identification. This should be decided for the first packet in
> UDP connection because the dialog should be with the same
> real server. I'm refering to this text in your [1] draft:
>
> "In DTLS 1.2, connection ids are exchanged at the beginning of the
> "DTLS session only."
>
> So, we have 2 choices when first packet in UDP
> connection is received:
>
> 1. Create and keep normal connection (without IP_VS_CONN_F_ONE_PACKET)
> when CID can not be used or CID is present only early.
> This means we can not inspect further this UDP connection for
> CID until it is expired. The first lookup by addr/port will
> find the existing connection for every following packet. Why we
> need to keep the connection? Because we can use the initially
> provided CID to get the needed real server but when next
> packets without CID come, we should know where to forward them.
> Or may be this can be handled by the "ct_match" PE method
> and we can create only IP_VS_CONN_F_ONE_PACKET conns.
>
> 2. Create connection for the packet (IP_VS_CONN_F_ONE_PACKET)
> but not based on svc->flags & IP_VS_SVC_F_ONEPACKET, this
> time we should provide it from the "ct_match" PE method.
> Connection is dropped after the packet is forwarded and
> the next packets from same client port will come again to
> the "ct_match" method. We will use this when CID will be
> present in all packets from this client port.
>
> As result, the PE should be able to maintain
> persistence templates with or without pe_data (CID),
> i.e. one per CID and additional without CID, all for the
> same CIP:CPORT->VIP:VPORT->RIP:RPORT.
>
> - According to RFC 6347 4.1.1. Transport Layer Mapping,
> "Multiple DTLS records may be placed in a single datagram".
> So, if multiple records contain different CIDs, this should
> be considered. I can think for some fatal cases where
> two records in one packet can be mapped to different
> real servers.
>
> Regards
>
> --
> Julian Anastasov <ja@ssi.bg>


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] DTLS Load balancing [ In reply to ]
Hello,

On Tue, 31 Oct 2017, Simon Bernard wrote:

> Thx a lot Julian for this accurate response.
>
> Just to let you know, this is a mid/long term idea, but more I think about it
> more I like it.
> CID is still a draft. Before to go to the LVS implementation we need to have
> DTLS implementation which support DTLS CID.
>
> All of this is really new for me. I have some questions before to start.
> (newbie's questions :p)
>
> If I well understand IPVS  is a module of the linux kernel.

Yes, there is a core module (ip_vs), modules for schedulers,
for apps (ip_vs_ftp), persistence engines (ip_vs_pe_sip).

> So the source code is directly available in the kernel repository [1] ?

Yes but you can also use latest kernel from:

https://www.kernel.org/

You can also read some files from the kernel sources:

# When coding:
Documentation/process/coding-style.rst
# How to send patches in raw form:
Documentation/process/email-clients.rst
Documentation/process/submitting-patches.rst
# The final patch should be sent to addresses specified with
# 'M' and 'L' for 'IPVS' here:
MAINTAINERS

Knowing 'git' to create a patch is not required,
using such command to create your patch should be enough:

diff -urpN linux.orig linux

For user-space tool:

# get fresh ipvsadm sources:
git clone git://git.kernel.org/pub/scm/utils/kernel/ipvsadm/ipvsadm.git

to use --pe dtls

> I should use ip_vs_pe_sip.c[2] as sample ?

Yes. But more changes can be needed.

> I looked at internet resource for kernel development there is a lot !  Linux
> Kernel Newbies[1] seems good. Any advice about how to tests my code ? Are you
> using virtualization or something like that ?

Yes, you can use such software. But initially you can
start with local real server and local client. IPVS supports all
4 cases:

1. local client + local real server
2. local client + remote real server (NAT)
3. remote client + local real server
4. remote client + remote real server

> Thx again for you time.
>
> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/netfilter/ipvs
> [2]
> https://github.com/torvalds/linux/blob/master/net/netfilter/ipvs/ip_vs_pe_sip.c
> [3] https://kernelnewbies.org/FirstKernelPatch

Regards

--
Julian Anastasov <ja@ssi.bg>