Mailing List Archive

Ottomata has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/404870 )

Change subject: Update Kafka to 1.0 with SSL support
......................................................................


Update Kafka to 1.0 with SSL support

This will make testing Mediawiki integration with Kafka and SSL easier

Bug: T126494
Change-Id: I93d7c7cb98664e3e41b5a383ba8f9976a0b09099
---
M puppet/modules/kafka/files/kafka.profile.sh
M puppet/modules/kafka/files/kafka.sh
D puppet/modules/kafka/files/server.properties
A puppet/modules/kafka/files/ssl/kafka_broker/ca.crt.pem
A puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.crt.pem
A puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.csr.pem
A puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.private.pem
A puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.public.pem
A puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.jks
A puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.p12
A puppet/modules/kafka/files/ssl/kafka_broker/truststore.jks
A puppet/modules/kafka/files/ssl/local_ca/ca.crt.pem
A puppet/modules/kafka/files/ssl/local_ca/local_ca.crt.pem
A puppet/modules/kafka/files/ssl/local_ca/local_ca.csr.pem
A puppet/modules/kafka/files/ssl/local_ca/local_ca.key.private.pem
A puppet/modules/kafka/files/ssl/local_ca/local_ca.key.public.pem
A puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.jks
A puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.p12
A puppet/modules/kafka/files/ssl/local_ca/truststore.jks
A puppet/modules/kafka/files/ssl/test0/ca.crt.pem
A puppet/modules/kafka/files/ssl/test0/test0.crt.pem
A puppet/modules/kafka/files/ssl/test0/test0.csr.pem
A puppet/modules/kafka/files/ssl/test0/test0.key.private.pem
A puppet/modules/kafka/files/ssl/test0/test0.key.public.pem
A puppet/modules/kafka/files/ssl/test0/test0.keystore.jks
A puppet/modules/kafka/files/ssl/test0/test0.keystore.p12
A puppet/modules/kafka/files/ssl/test0/truststore.jks
M puppet/modules/kafka/manifests/init.pp
A puppet/modules/kafka/templates/server.properties.erb
M puppet/modules/kafka/templates/systemd/kafka.erb
M puppet/modules/role/settings/kafka.yaml
31 files changed, 421 insertions(+), 119 deletions(-)

Approvals:
Ottomata: Verified; Looks good to me, approved
BryanDavis: Looks good to me, but someone else must approve



diff --git a/puppet/modules/kafka/files/kafka.profile.sh b/puppet/modules/kafka/files/kafka.profile.sh
index ab3ed80..f1f2a8a 100644
--- a/puppet/modules/kafka/files/kafka.profile.sh
+++ b/puppet/modules/kafka/files/kafka.profile.sh
@@ -3,5 +3,6 @@
# These environment variables are used by the kafka CLI
# so that you don't have to provide them as arguments
# every time you use it.
-export ZOOKEEPER_URL=localhost:2181
-export BROKER_LIST=localhost:9092
+export KAFKA_ZOOKEEPER_URL=localhost:2181/kafka
+export KAFKA_BOOTSTRAP_SERVERS=localhost:9092
+export KAFKA_JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
diff --git a/puppet/modules/kafka/files/kafka.sh b/puppet/modules/kafka/files/kafka.sh
index e7db2bb..e2c1c8b 100755
--- a/puppet/modules/kafka/files/kafka.sh
+++ b/puppet/modules/kafka/files/kafka.sh
@@ -1,5 +1,7 @@
#!/bin/bash

+# NOTE: This file is managed by Puppet.
+
SCRIPT_NAME=$(basename "$0")

commands=$(ls /usr/bin/kafka-* | xargs -n 1 basename | sed 's@kafka-@ @g')
@@ -8,9 +10,9 @@
$SCRIPT_NAME <command> [options]

Handy wrapper around various kafka-* scripts. Set the environment variables
-ZOOKEEPER_URL and BROKER_LIST so you don't have to keep typing
---zookeeper-connect or --broker-list each time you want to use a kafka-*
-script.
+KAFKA_ZOOKEEPER_URL, KAFKA_BOOTSTRAP_SERVERS so you don't have to keep typing
+--zookeeper-connect, --broker-list or --bootstrap-server each time you want to
+use a kafka-* script.

Usage:

@@ -20,11 +22,18 @@
$commands

Environment Variables:
- ZOOKEEPER_URL - If this is set, any commands that take a --zookeeper flag will be given this value.
- BROKER_LIST - If this is set, any commands that take a --broker-list flag will be given this value.
+ KAFKA_JAVA_HOME - Value of JAVA_HOME to use for invoking Kafka commands.
+ KAFKA_ZOOKEEPER_URL - If this is set, any commands that take a --zookeeper
+ flag will be given this value.
+ KAFKA_BOOTSTRAP_SERVERS - If this is set, any commands that take a --broker-list or
+ --bootstrap-server flag will be given this value.
+ Also any command that take a --authorizer-properties
+ will get the correct zookeeper.connect value.
+
"

-if [ -z "${1}" -o ${1:0:1} == '-' ]; then
+# Print usage if no <command> given, or $1 starts with '-'
+if [ -z "${1}" -o "${1:0:1}" == '-' ]; then
echo "${USAGE}"
exit 1
fi
@@ -33,43 +42,77 @@
command="kafka-${1}"
shift

+# Export JAVA_HOME as KAFKA_JAVA_HOME if it is set.
+# This makes kafka-run-class use the preferred JAVA_HOME for Kafka.
+if [ -n "${KAFKA_JAVA_HOME}" ]; then
+ : ${JAVA_HOME="$KAFKA_JAVA_HOME"}
+ export JAVA_HOME
+fi
+
# Set ZOOKEEPER_OPT if ZOOKEEPER_URL is set and --zookeeper has not
# also been passed in as a CLI arg. This will be included
# in command functions that take a --zookeeper argument.
-if [ -n "${ZOOKEEPER_URL}" -a -z "$(echo $@ | grep -- --zookeeper)" ]; then
- ZOOKEEPER_OPT="--zookeeper ${ZOOKEEPER_URL}"
+if [ -n "${KAFKA_ZOOKEEPER_URL}" -a -z "$(echo $@ | grep -- --zookeeper)" ]; then
+ ZOOKEEPER_OPT="--zookeeper ${KAFKA_ZOOKEEPER_URL}"
fi

-# Set BROKER_LIST_OPT if BROKER_LIST is set and --broker-list has not
+# Set BROKER_LIST_OPT if KAFKA_BOOTSTRAP_SERVERS is set and --broker-list has not
# also been passed in as a CLI arg. This will be included
# in command functions that take a --broker-list argument.
-if [ -n "${BROKER_LIST}" -a -z "$(echo $@ | grep -- --broker-list)" ]; then
- BROKER_LIST_OPT="--broker-list ${BROKER_LIST}"
+if [ -n "${KAFKA_BOOTSTRAP_SERVERS}" -a -z "$(echo $@ | grep -- --broker-list)" ]; then
+ BROKER_LIST_OPT="--broker-list ${KAFKA_BOOTSTRAP_SERVERS}"
fi

-# Each of these lists signifies that either --broker-list or --zookeeper needs
-# to be given to the $command. If $command matches one of these, then we
-# will add the opt if it is not provided already in $@.
+# Set BOOTSTRAP_SERVER_OPT if KAFKA_BOOTSTRAP_SERVERS is set and --bootstrap-server has not
+# also been passed in as a CLI arg. This will be included
+# in command functions that take a --bootstrap-server argument.
+if [ -n "${KAFKA_BOOTSTRAP_SERVERS}" -a -z "$(echo $@ | grep -- --bootstrap-server)" ]; then
+ BOOTSTRAP_SERVER_OPT="--bootstrap-server ${KAFKA_BOOTSTRAP_SERVERS}"
+fi
+
+# Set ZOOKEEPER_CONNECT_OPT if KAFKA_ZOOKEEPER_URL is set and '--authorizer-properties zookeeper.connect'
+# has not also been passed in as a CLI arg. This will be included
+# in command functions that take '--authorizer-properties zookeeper.connect' argument.
+if [ -n "${KAFKA_ZOOKEEPER_URL}" -a -z "$(echo $@ | egrep -- '--authorizer-properties\ *zookeeper\.connect')" ]; then
+ ZOOKEEPER_CONNECT_OPT="--authorizer-properties zookeeper.connect=${KAFKA_ZOOKEEPER_URL}"
+fi
+
+# Each of these lists signifies that either --broker-list, --bootstrap-server,
+# or --zookeeper needs to be given to the $command. If $command matches one of these,
+# then we will add the opt if it is not provided already in $@.
+# Until https://issues.apache.org/jira/browse/KAFKA-4307 is available, there are
+# inconsistencies in broker CLI parameters. Some use --bootstrap-server, others
+# use --broker-list, so we have to support both for now.
+# --broker-list should be removed in later versions in favor of --bootstrap-server
broker_list_commands="kafka-console-producer "\
"kafka-consumer-perf-test "\
+"kafka-replay-log-producer "\
"kafka-replica-verification "\
"kafka-simple-consumer-shell "\
"kafka-verifiable-consumer "\
"kafka-verifiable-producer"

+bootstrap_server_commands="kafka-console-consumer "\
+"kafka-broker-api-versions "\
+"kafka-consumer-groups "
+
zookeeper_commands="kafka-configs "\
-"kafka-console-consumer "\
-"kafka-consumer-groups "\
-"kafka-consumer-perf-test "\
+"kafka-consumer-offset-checker.sh "\
"kafka-preferred-replica-election "\
"kafka-reassign-partitions "\
"kafka-replay-log-producer "\
"kafka-topics"

+zookeeper_connect_commands="kafka-acls"
+
EXTRA_OPTS=""
echo "${broker_list_commands}" | /bin/grep -q "${command}" && EXTRA_OPTS="${BROKER_LIST_OPT} "
+echo "${bootstrap_server_commands}" | /bin/grep -q "${command}" && EXTRA_OPTS="${EXTRA_OPTS}${BOOTSTRAP_SERVER_OPT} "
echo "${zookeeper_commands}" | /bin/grep -q "${command}" && EXTRA_OPTS="${EXTRA_OPTS}${ZOOKEEPER_OPT} "
+echo "${zookeeper_connect_commands}" | /bin/grep -q "${command}" && EXTRA_OPTS="${EXTRA_OPTS}${ZOOKEEPER_CONNECT_OPT} "

# Print out the command we are about to exec, and then run it
-echo "${command} ${EXTRA_OPTS}$@"
-${command} ${EXTRA_OPTS}$@
+# set -f to not expand wildcards in command, e.g. --topic '*'
+set -f
+echo ${command} ${EXTRA_OPTS}"$@"
+${command} ${EXTRA_OPTS}"$@"
diff --git a/puppet/modules/kafka/files/server.properties b/puppet/modules/kafka/files/server.properties
deleted file mode 100644
index a64c8cd..0000000
--- a/puppet/modules/kafka/files/server.properties
+++ /dev/null
@@ -1,86 +0,0 @@
-# NOTE: This file is managed by Puppet.
-
-############################# Server Basics #############################
-
-# The id of the broker. This must be set to a unique integer for each broker.
-broker.id=0
-
-############################# Socket Server Settings #############################
-
-listeners=PLAINTEXT://:9092
-
-# The port the socket server listens on
-#port=9092
-
-# The number of threads handling network requests
-num.network.threads=3
-
-# The number of threads doing disk I/O
-num.io.threads=8
-
-# The send buffer (SO_SNDBUF) used by the socket server
-socket.send.buffer.bytes=102400
-
-# The receive buffer (SO_RCVBUF) used by the socket server
-socket.receive.buffer.bytes=102400
-
-# The maximum size of a request that the socket server will accept (protection against OOM)
-socket.request.max.bytes=104857600
-
-
-############################# Log Basics #############################
-
-# A comma seperated list of directories under which to store log files
-log.dirs=/var/lib/kafka
-
-# The default number of log partitions per topic. More partitions allow greater
-# parallelism for consumption, but this will also result in more files across
-# the brokers.
-num.partitions=1
-
-# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
-# This value is recommended to be increased for installations with data dirs located in RAID array.
-num.recovery.threads.per.data.dir=1
-
-############################# Log Retention Policy #############################
-
-# The following configurations control the disposal of log segments. The policy can
-# be set to delete segments after a period of time, or after a given size has accumulated.
-# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
-# from the end of the log.
-
-# The minimum age of a log file to be eligible for deletion
-log.retention.hours=168
-
-# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining
-# segments don't drop below log.retention.bytes.
-#log.retention.bytes=1073741824
-
-# The maximum size of a log segment file. When this size is reached a new log segment will be created.
-log.segment.bytes=1073741824
-
-# The interval at which log segments are checked to see if they can be deleted according
-# to the retention policies
-log.retention.check.interval.ms=300000
-
-# Allow topic deletion
-delete.topic.enable=true
-
-############################# Zookeeper #############################
-
-# Zookeeper connection string (see zookeeper docs for details).
-# This is a comma separated host:port pairs, each corresponding to a zk
-# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
-# You can also append an optional chroot string to the urls to specify the
-# root directory for all kafka znodes.
-zookeeper.connect=localhost:2181
-
-# Timeout in ms for connecting to zookeeper
-zookeeper.connection.timeout.ms=6000
-
-##################### Confluent Proactive Support ######################
-
-# If set to true, then the feature to collect and report support metrics
-# ("Metrics") is enabled. If set to false, the feature is disabled.
-#
-confluent.support.metrics.enable=false
diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/ca.crt.pem b/puppet/modules/kafka/files/ssl/kafka_broker/ca.crt.pem
new file mode 100644
index 0000000..5b15145
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/kafka_broker/ca.crt.pem
@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC6DCCAdCgAwIBAgIUB4c5VpnNmpxVHmwDd433tjPQ/FEwDQYJKoZIhvcNAQEL
+BQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwIBcNMTgwMTE3MjEzNzU1WhgPMjExNzEy
+MjQyMTM3NTVaMBMxETAPBgNVBAMMCGxvY2FsX2NhMIIBIjANBgkqhkiG9w0BAQEF
+AAOCAQ8AMIIBCgKCAQEAsMsG+Y+N2BUmivLvKjxD6/YhS8q0lkn5XdGdDwI/LHVW
+iWu3NKYYjtwBKOF6QORrZBatiUiFAo7WlcjHBPdg+bd7CD4+rV6By3O2d2wJuvpO
+Oex2xT1iLOM4vuZFc/GbOtGWSe7+fJVef8IDwUcMq9U2BrV1nKV4Eb1coJBMifBT
+b36zl0zgRDkajpxKaj7LdLTeCU/Vn+bE9BO0uAAQqGBu4Eyj6GqXX/IIWoxQ1nxR
+GaIBhM2pNEzoBza5lX1MgYjitK0rC/LAa54bE154hd40fTN5sfnGFjSK+jsiL9w1
+g3myH+Vlt7Y+3yGls7Km3sWAhuN/vDeAKmUslV0oWQIDAQABozIwMDAPBgNVHRMB
+Af8EBTADAQH/MB0GA1UdDgQWBBRwYbjt7osV+1HSg6q0ey/JcljdgDANBgkqhkiG
+9w0BAQsFAAOCAQEAqQGzKDl0x0LU/Q+YMfFY5499COmJ2ATp5I1Ou/GngrxhkdFk
+ZlliPouVMNpqm+Xpr4vSHm5wlis09xxm508z8JAA/PzKo847tGMQBnLwmdW67f4H
+njUsJsFBOfakfc64f0LUfHRgHL4EXF6hbC4W4PhZH09cbt91v7CjY0KGMdHThe2Y
+1y5/QQmVH7tPyDeOnDUrCiXmdwtWe2KOyu04e36NYTx9hQzC26WXmdz4rbI/MEQ1
+K9SJs96pIz/X3MyQ/JfF8ThOplhJ1ACUkUyzxmjmzXc5evUHF3Ho3fAIGpl8z1fE
+REWPT9f0L3p4mBuXyJN3KPfyd9ylec3wuSdPQw==
+-----END CERTIFICATE-----
diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.crt.pem b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.crt.pem
new file mode 100644
index 0000000..c7df18b
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.crt.pem
@@ -0,0 +1,15 @@
+-----BEGIN CERTIFICATE-----
+MIICPTCCASWgAwIBAgIUdgK/B0WTHrDyaitpTAgd+MajusMwDQYJKoZIhvcNAQEL
+BQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwHhcNMTgwMTE3MjEzODQ0WhcNMTkwMTE3
+MjEzODQ0WjAXMRUwEwYDVQQDDAxrYWZrYV9icm9rZXIwWTATBgcqhkjOPQIBBggq
+hkjOPQMBBwNCAASGYVywLuFgffqmD0TswEHeyALZecYSpbp1qe6kgH7fXN71t3+l
+SQ2f0maLq+vNqbDVGJGnkq+gJylMt+9h/UW7o1AwTjAMBgNVHRMBAf8EAjAAMB0G
+A1UdDgQWBBQnM27LGtQA4rV6ct2I6G6ke8IMQTAfBgNVHSMEGDAWgBRwYbjt7osV
++1HSg6q0ey/JcljdgDANBgkqhkiG9w0BAQsFAAOCAQEApjV43RBvCGny/l725hck
+zv1AkZiX3o1BajiLWqa8LTTGOr5u01f7FmCPZ9c0sZCSfkweRzAbj3uwKg/4fJab
+dh4BCMsfbEU/azZ1dosKT14hdEWHawYKyRbdInmSB7u1cNGbDXToQg+wv/tsB6M0
+jA56sERp2FfdXC1sR5/LO26VXN0S8oDwCSb/nLbz/FBZA31rnitOJL2HzKnMfh/5
+3KdBngsVC17DA9Q7mKd11K8G6mSpM5aD+a8+SF755Jr/rGTme1lbJ3yVNohMRM7k
+rEbWW1ZUMN1kNd1kwqb8l54DtPxWXpYkC3bDZm1qCbOAuC1z72E1UbMok/ZHqeeB
+Rw==
+-----END CERTIFICATE-----
diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.csr.pem b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.csr.pem
new file mode 100644
index 0000000..789e93b
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.csr.pem
@@ -0,0 +1,9 @@
+-----BEGIN CERTIFICATE REQUEST-----
+MIIBMTCB2AIBADAXMRUwEwYDVQQDDAxrYWZrYV9icm9rZXIwWTATBgcqhkjOPQIB
+BggqhkjOPQMBBwNCAASGYVywLuFgffqmD0TswEHeyALZecYSpbp1qe6kgH7fXN71
+t3+lSQ2f0maLq+vNqbDVGJGnkq+gJylMt+9h/UW7oF8wXQYJKoZIhvcNAQkOMVAw
+TjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBQnM27LGtQA4rV6ct2I6G6ke8IMQTAf
+BgNVHSMEGDAWgBRwYbjt7osV+1HSg6q0ey/JcljdgDAKBggqhkjOPQQDAgNIADBF
+AiEAnMLETBbG4OCajAiKQcOPxstu1c8aRv7N4lEs1STPTW4CICwkzCuhzsLQ7E+V
+mDLyUNhNeDxJ7YIKeY0Atl8EherX
+-----END CERTIFICATE REQUEST-----
diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.private.pem b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.private.pem
new file mode 100644
index 0000000..e508c8a
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.private.pem
@@ -0,0 +1,7 @@
+-----BEGIN ENCRYPTED PRIVATE KEY-----
+MIHsMFcGCSqGSIb3DQEFDTBKMCkGCSqGSIb3DQEFDDAcBAjRb3lYmKm2BwICCAAw
+DAYIKoZIhvcNAgkFADAdBglghkgBZQMEASoEEIOlTfYrDLSN+s7dW9dKBPwEgZCe
+oOFlEcPasrxHqF+p8vVZrgVacxco0+4Si1UipaNNTocJsxngOU4CUzOq+yZuOydx
+7YJ+nTbn/rNmGtIeCpxrJ2SaCx0/U5XafaWY5jRjCi5NEWwkT3au7aamcmsRcaZN
+gBb/R0P995nCzPgSZ4oHPFj8BEppDde8BYHfviLjxJdOYrw9kBa5c6+q+tfEuB8=
+-----END ENCRYPTED PRIVATE KEY-----
diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.public.pem b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.public.pem
new file mode 100644
index 0000000..4a4921b
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.public.pem
@@ -0,0 +1,4 @@
+-----BEGIN PUBLIC KEY-----
+MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEhmFcsC7hYH36pg9E7MBB3sgC2XnG
+EqW6danupIB+31ze9bd/pUkNn9Jmi6vrzamw1RiRp5KvoCcpTLfvYf1Fuw==
+-----END PUBLIC KEY-----
diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.jks b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.jks
new file mode 100644
index 0000000..8e46fb9
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.jks
Binary files differ
diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.p12 b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.p12
new file mode 100644
index 0000000..44c9b0e
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.p12
Binary files differ
diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/truststore.jks b/puppet/modules/kafka/files/ssl/kafka_broker/truststore.jks
new file mode 100644
index 0000000..df7ec3f
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/kafka_broker/truststore.jks
Binary files differ
diff --git a/puppet/modules/kafka/files/ssl/local_ca/ca.crt.pem b/puppet/modules/kafka/files/ssl/local_ca/ca.crt.pem
new file mode 100644
index 0000000..5b15145
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/local_ca/ca.crt.pem
@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC6DCCAdCgAwIBAgIUB4c5VpnNmpxVHmwDd433tjPQ/FEwDQYJKoZIhvcNAQEL
+BQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwIBcNMTgwMTE3MjEzNzU1WhgPMjExNzEy
+MjQyMTM3NTVaMBMxETAPBgNVBAMMCGxvY2FsX2NhMIIBIjANBgkqhkiG9w0BAQEF
+AAOCAQ8AMIIBCgKCAQEAsMsG+Y+N2BUmivLvKjxD6/YhS8q0lkn5XdGdDwI/LHVW
+iWu3NKYYjtwBKOF6QORrZBatiUiFAo7WlcjHBPdg+bd7CD4+rV6By3O2d2wJuvpO
+Oex2xT1iLOM4vuZFc/GbOtGWSe7+fJVef8IDwUcMq9U2BrV1nKV4Eb1coJBMifBT
+b36zl0zgRDkajpxKaj7LdLTeCU/Vn+bE9BO0uAAQqGBu4Eyj6GqXX/IIWoxQ1nxR
+GaIBhM2pNEzoBza5lX1MgYjitK0rC/LAa54bE154hd40fTN5sfnGFjSK+jsiL9w1
+g3myH+Vlt7Y+3yGls7Km3sWAhuN/vDeAKmUslV0oWQIDAQABozIwMDAPBgNVHRMB
+Af8EBTADAQH/MB0GA1UdDgQWBBRwYbjt7osV+1HSg6q0ey/JcljdgDANBgkqhkiG
+9w0BAQsFAAOCAQEAqQGzKDl0x0LU/Q+YMfFY5499COmJ2ATp5I1Ou/GngrxhkdFk
+ZlliPouVMNpqm+Xpr4vSHm5wlis09xxm508z8JAA/PzKo847tGMQBnLwmdW67f4H
+njUsJsFBOfakfc64f0LUfHRgHL4EXF6hbC4W4PhZH09cbt91v7CjY0KGMdHThe2Y
+1y5/QQmVH7tPyDeOnDUrCiXmdwtWe2KOyu04e36NYTx9hQzC26WXmdz4rbI/MEQ1
+K9SJs96pIz/X3MyQ/JfF8ThOplhJ1ACUkUyzxmjmzXc5evUHF3Ho3fAIGpl8z1fE
+REWPT9f0L3p4mBuXyJN3KPfyd9ylec3wuSdPQw==
+-----END CERTIFICATE-----
diff --git a/puppet/modules/kafka/files/ssl/local_ca/local_ca.crt.pem b/puppet/modules/kafka/files/ssl/local_ca/local_ca.crt.pem
new file mode 100644
index 0000000..5b15145
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/local_ca/local_ca.crt.pem
@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC6DCCAdCgAwIBAgIUB4c5VpnNmpxVHmwDd433tjPQ/FEwDQYJKoZIhvcNAQEL
+BQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwIBcNMTgwMTE3MjEzNzU1WhgPMjExNzEy
+MjQyMTM3NTVaMBMxETAPBgNVBAMMCGxvY2FsX2NhMIIBIjANBgkqhkiG9w0BAQEF
+AAOCAQ8AMIIBCgKCAQEAsMsG+Y+N2BUmivLvKjxD6/YhS8q0lkn5XdGdDwI/LHVW
+iWu3NKYYjtwBKOF6QORrZBatiUiFAo7WlcjHBPdg+bd7CD4+rV6By3O2d2wJuvpO
+Oex2xT1iLOM4vuZFc/GbOtGWSe7+fJVef8IDwUcMq9U2BrV1nKV4Eb1coJBMifBT
+b36zl0zgRDkajpxKaj7LdLTeCU/Vn+bE9BO0uAAQqGBu4Eyj6GqXX/IIWoxQ1nxR
+GaIBhM2pNEzoBza5lX1MgYjitK0rC/LAa54bE154hd40fTN5sfnGFjSK+jsiL9w1
+g3myH+Vlt7Y+3yGls7Km3sWAhuN/vDeAKmUslV0oWQIDAQABozIwMDAPBgNVHRMB
+Af8EBTADAQH/MB0GA1UdDgQWBBRwYbjt7osV+1HSg6q0ey/JcljdgDANBgkqhkiG
+9w0BAQsFAAOCAQEAqQGzKDl0x0LU/Q+YMfFY5499COmJ2ATp5I1Ou/GngrxhkdFk
+ZlliPouVMNpqm+Xpr4vSHm5wlis09xxm508z8JAA/PzKo847tGMQBnLwmdW67f4H
+njUsJsFBOfakfc64f0LUfHRgHL4EXF6hbC4W4PhZH09cbt91v7CjY0KGMdHThe2Y
+1y5/QQmVH7tPyDeOnDUrCiXmdwtWe2KOyu04e36NYTx9hQzC26WXmdz4rbI/MEQ1
+K9SJs96pIz/X3MyQ/JfF8ThOplhJ1ACUkUyzxmjmzXc5evUHF3Ho3fAIGpl8z1fE
+REWPT9f0L3p4mBuXyJN3KPfyd9ylec3wuSdPQw==
+-----END CERTIFICATE-----
diff --git a/puppet/modules/kafka/files/ssl/local_ca/local_ca.csr.pem b/puppet/modules/kafka/files/ssl/local_ca/local_ca.csr.pem
new file mode 100644
index 0000000..ffd4f66
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/local_ca/local_ca.csr.pem
@@ -0,0 +1,16 @@
+-----BEGIN CERTIFICATE REQUEST-----
+MIICmTCCAYECAQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwggEiMA0GCSqGSIb3DQEB
+AQUAA4IBDwAwggEKAoIBAQCwywb5j43YFSaK8u8qPEPr9iFLyrSWSfld0Z0PAj8s
+dVaJa7c0phiO3AEo4XpA5GtkFq2JSIUCjtaVyMcE92D5t3sIPj6tXoHLc7Z3bAm6
++k457HbFPWIs4zi+5kVz8Zs60ZZJ7v58lV5/wgPBRwyr1TYGtXWcpXgRvVygkEyJ
+8FNvfrOXTOBEORqOnEpqPst0tN4JT9Wf5sT0E7S4ABCoYG7gTKPoapdf8ghajFDW
+fFEZogGEzak0TOgHNrmVfUyBiOK0rSsL8sBrnhsTXniF3jR9M3mx+cYWNIr6OyIv
+3DWDebIf5WW3tj7fIaWzsqbexYCG43+8N4AqZSyVXShZAgMBAAGgQTA/BgkqhkiG
+9w0BCQ4xMjAwMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFHBhuO3uixX7UdKD
+qrR7L8lyWN2AMA0GCSqGSIb3DQEBCwUAA4IBAQAWmdBr9XyKaL1Oe0XMJr23yjq+
+ouj5pLf/Z2zmb540HPDgpDJQIAZNhtUOdFzbyrP3TcHFZK1birMYB9lsyRzaXBlw
+lIqD1eCxvqki2u+0t9/Qqs4kTYU9LQEW9T4oj1sbA+mbz6/F3FCmCgVG/XW0kGOq
+CuiIRONDnF/QUs2NBb3eWbhnwGF5pW+dWWC3m4U+B8hqOOHBM4PHbUq0nBjA99u6
+LG3/bxog5pEE2wrp31HEizu8Ou3/dt7zhBAjwqSLq9rn6GNF08UVfENzJrYM2kGf
+ST+wZFL8NzLw/HEmUlRSOOt4sa+fzJ1AD3lMHECf+34Hg4T14EGgM6iEMw3S
+-----END CERTIFICATE REQUEST-----
diff --git a/puppet/modules/kafka/files/ssl/local_ca/local_ca.key.private.pem b/puppet/modules/kafka/files/ssl/local_ca/local_ca.key.private.pem
new file mode 100644
index 0000000..5a2a9a3
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/local_ca/local_ca.key.private.pem
@@ -0,0 +1,30 @@
+-----BEGIN ENCRYPTED PRIVATE KEY-----
+MIIFLTBXBgkqhkiG9w0BBQ0wSjApBgkqhkiG9w0BBQwwHAQIp5YiQej3JXQCAggA
+MAwGCCqGSIb3DQIJBQAwHQYJYIZIAWUDBAEqBBB1TdS3A2ynG4wnIrrHGG6uBIIE
+0AjVrEAkvTtEemcBqCbGYa/IiwR9A7CaTpyCLufPfvBxW4euhIN6J9ucrYD4dS3m
+tgrz7mXqUxs0HAapQttHpx1kEX2eSTfRMAhkEW3VKzFuxixUcpokKTMEf/9NVBd5
+C2aw2kXfi94plJPo9FX49Gst/nWBLzHin1Wh+Ferb3cxU5Rw53uBOvGGU04AgJ9Y
+3Hbe5VCeUPl6yGuNn3iaFbTjOJuZwPFpOU7DLEL/kgl0oGjJKfjD9pcB04DJADB2
+grns1xsYqIscC0rRK5Cuv82KHtYvpRCT+68UW5ZySQpMRLRTwBu10z2FF20X092p
+5PLV9OMqvBjN1Fv4QGCkjGIhIlHyHTiNMrozsN3upjvocG7y1TnUgT51FVUS4hYf
+mFJvAZQuAP0dParpuva7YDJ9xEoYaz/mXj1rrvC7kZ3iPG5y+A20mhRR2gDo2aHU
+W3ZZe+XHo6NV3ULLcgr6dykrmXldnmNzP5gXF1tZTSqXlu6Q+x+CaJtQAESEHTqV
+7nk25DSM2U1/+Lw+wT0mgonAaLf2gQaYFjCeLUfFAm4EfFLSHZsYLc11oVmSJRFp
+CU5afpCj6Ygc/FT68hclakutDwGTRii4tNN635pMknYh178sn5QKqv1uLN2PWt09
+OL4Z37Ma6YXtT/t7ruM745JbY2pcFt0BPYiHRCnm/fzjAJydqhM6eEl23+H1xgQO
+BtFsqVs1W8UfnG/O+SdoSPRmzxcQ8GxOXZhllGaaNHionlvfqzFKtIVlz/2zPjld
+TvELkyKuxnz1qDZDJs5PFC5DL7DR1yi/Aezsn+cRg0bj03M1qAJxZ7cihZqZc0IZ
+rV+U2mlfbEIfndt79GpdQ5Vzgbxxk9BzAHN+tyMY6//CyeZqYUdYyH+mFipdyF95
+/019rxXEYy+JnFIAg26vAVGDgORVVuHphM53LOpk/AqA8ravFGhTO4ABHafXV3Gv
+VAj4hu69tIkoX91xZPm6VMnDAeP76IfuKk/uje4JrlpJ3Hoqv6Qi0r4TVaiGwx1N
+v93vgJ9CYKOKHwS+UaIwHzqLrEbQ+Q1bIGabvQXAmcTQJP0tPe/Xv8oOKL46tzhA
+i1gihluu8REGf5/09+ibqb6ktj0X2hGDU17/FAowmPkFQEee03fsg3V41bBG1OL/
+YwQMD4EhvcxHnWfli+PtoM55GoCA1jhD4ucfDGCAiMEoUYfcj2KDlqhHGA+24LL1
+Ge6Dm4sbGfV1LnNkHKysG3cRsNzxZ1VHFnslB/5UShYNd4Bft8UgZ+VRustnbwiq
+F0ZZYiNA8ENwjA6bida92CvZmq7gXug87FhKiFlHREjYtuqvkEWl1C7UjxbZUcx3
+X+fBg/fJo3M4FZ3wlrAYFi23szdtyf43UNe/dSuE9mJO85DJjTapyFEESnkwFoNQ
+w2JhX7Wzv3OdwwNqfnpVXLdALBh0svqu2t7nRL3oOENpFU3AkWqLafTNV4vfFQsU
++U7Io1hHmuM3Tjcy+lGUTeuexSBN7EGpdoAeTlf1l+CruCOWYvbQ9NzyRUskDeMD
+mdCmh41EhwjuwT6YcjQLWOD9uXPdUnHyUy7c3nmRrUBAgQd2A9of0x3fvzrwRWd6
+MDoQdYYasRJ8Q9Lub0jvW7Y8EPocA7ewZAv+SijxCB/w
+-----END ENCRYPTED PRIVATE KEY-----
diff --git a/puppet/modules/kafka/files/ssl/local_ca/local_ca.key.public.pem b/puppet/modules/kafka/files/ssl/local_ca/local_ca.key.public.pem
new file mode 100644
index 0000000..8e9daab
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/local_ca/local_ca.key.public.pem
@@ -0,0 +1,9 @@
+-----BEGIN PUBLIC KEY-----
+MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsMsG+Y+N2BUmivLvKjxD
+6/YhS8q0lkn5XdGdDwI/LHVWiWu3NKYYjtwBKOF6QORrZBatiUiFAo7WlcjHBPdg
++bd7CD4+rV6By3O2d2wJuvpOOex2xT1iLOM4vuZFc/GbOtGWSe7+fJVef8IDwUcM
+q9U2BrV1nKV4Eb1coJBMifBTb36zl0zgRDkajpxKaj7LdLTeCU/Vn+bE9BO0uAAQ
+qGBu4Eyj6GqXX/IIWoxQ1nxRGaIBhM2pNEzoBza5lX1MgYjitK0rC/LAa54bE154
+hd40fTN5sfnGFjSK+jsiL9w1g3myH+Vlt7Y+3yGls7Km3sWAhuN/vDeAKmUslV0o
+WQIDAQAB
+-----END PUBLIC KEY-----
diff --git a/puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.jks b/puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.jks
new file mode 100644
index 0000000..c63da93
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.jks
Binary files differ
diff --git a/puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.p12 b/puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.p12
new file mode 100644
index 0000000..e88c4ef
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.p12
Binary files differ
diff --git a/puppet/modules/kafka/files/ssl/local_ca/truststore.jks b/puppet/modules/kafka/files/ssl/local_ca/truststore.jks
new file mode 100644
index 0000000..7dea6a4
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/local_ca/truststore.jks
Binary files differ
diff --git a/puppet/modules/kafka/files/ssl/test0/ca.crt.pem b/puppet/modules/kafka/files/ssl/test0/ca.crt.pem
new file mode 100644
index 0000000..5b15145
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/test0/ca.crt.pem
@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC6DCCAdCgAwIBAgIUB4c5VpnNmpxVHmwDd433tjPQ/FEwDQYJKoZIhvcNAQEL
+BQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwIBcNMTgwMTE3MjEzNzU1WhgPMjExNzEy
+MjQyMTM3NTVaMBMxETAPBgNVBAMMCGxvY2FsX2NhMIIBIjANBgkqhkiG9w0BAQEF
+AAOCAQ8AMIIBCgKCAQEAsMsG+Y+N2BUmivLvKjxD6/YhS8q0lkn5XdGdDwI/LHVW
+iWu3NKYYjtwBKOF6QORrZBatiUiFAo7WlcjHBPdg+bd7CD4+rV6By3O2d2wJuvpO
+Oex2xT1iLOM4vuZFc/GbOtGWSe7+fJVef8IDwUcMq9U2BrV1nKV4Eb1coJBMifBT
+b36zl0zgRDkajpxKaj7LdLTeCU/Vn+bE9BO0uAAQqGBu4Eyj6GqXX/IIWoxQ1nxR
+GaIBhM2pNEzoBza5lX1MgYjitK0rC/LAa54bE154hd40fTN5sfnGFjSK+jsiL9w1
+g3myH+Vlt7Y+3yGls7Km3sWAhuN/vDeAKmUslV0oWQIDAQABozIwMDAPBgNVHRMB
+Af8EBTADAQH/MB0GA1UdDgQWBBRwYbjt7osV+1HSg6q0ey/JcljdgDANBgkqhkiG
+9w0BAQsFAAOCAQEAqQGzKDl0x0LU/Q+YMfFY5499COmJ2ATp5I1Ou/GngrxhkdFk
+ZlliPouVMNpqm+Xpr4vSHm5wlis09xxm508z8JAA/PzKo847tGMQBnLwmdW67f4H
+njUsJsFBOfakfc64f0LUfHRgHL4EXF6hbC4W4PhZH09cbt91v7CjY0KGMdHThe2Y
+1y5/QQmVH7tPyDeOnDUrCiXmdwtWe2KOyu04e36NYTx9hQzC26WXmdz4rbI/MEQ1
+K9SJs96pIz/X3MyQ/JfF8ThOplhJ1ACUkUyzxmjmzXc5evUHF3Ho3fAIGpl8z1fE
+REWPT9f0L3p4mBuXyJN3KPfyd9ylec3wuSdPQw==
+-----END CERTIFICATE-----
diff --git a/puppet/modules/kafka/files/ssl/test0/test0.crt.pem b/puppet/modules/kafka/files/ssl/test0/test0.crt.pem
new file mode 100644
index 0000000..af99729
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/test0/test0.crt.pem
@@ -0,0 +1,14 @@
+-----BEGIN CERTIFICATE-----
+MIICNjCCAR6gAwIBAgIUVyTo+2/NNL7zi60tsX/mGhTtABkwDQYJKoZIhvcNAQEL
+BQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwHhcNMTgwMTE3MjEzOTA4WhcNMTkwMTE3
+MjEzOTA4WjAQMQ4wDAYDVQQDDAV0ZXN0MDBZMBMGByqGSM49AgEGCCqGSM49AwEH
+A0IABDeSxfTQlf3w8Bizm3tXQJO/T+4ekZKr7BDEMaO9vaf4/aqJQTZ9UkMIlIKi
+6wswg+JPmZcoZhAQgbt0drPrPw2jUDBOMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYE
+FL4eoGQL6YYQqNYbC3fStuDI13sgMB8GA1UdIwQYMBaAFHBhuO3uixX7UdKDqrR7
+L8lyWN2AMA0GCSqGSIb3DQEBCwUAA4IBAQCZOzbMhYRjgzAeGWfrYuAhFhKYL2G6
+sQue4XsNSJoiSeqxZ82dMhmmtUigQOMoxzFgQWZ0imPCwf7rNhA1B4Ucy6QCFXIs
+91O+DTjw7AqUBIEkhDNLbO6lwusJg+UfUbmW6djg8ruqVd6BULVX4KsJIXz/j6fH
+2lnH9PnjDny39sBFU8jk/f/iH4ieW3nkpd+b0hgme1HL7oNgPO+OHtq9UtAsG5s3
+/7leFfpVhhXn+VIBgO8jyq10gat804hiXzm+m1R7pBzlwYoHk2bwa83VM4L9BB7s
+dQKiTBGE+y4uxw/VK/pa6VzlSigDFsQU80JqRnXCUQ52hbXDB3PDt2Hw
+-----END CERTIFICATE-----
diff --git a/puppet/modules/kafka/files/ssl/test0/test0.csr.pem b/puppet/modules/kafka/files/ssl/test0/test0.csr.pem
new file mode 100644
index 0000000..217739b
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/test0/test0.csr.pem
@@ -0,0 +1,9 @@
+-----BEGIN CERTIFICATE REQUEST-----
+MIIBKjCB0QIBADAQMQ4wDAYDVQQDDAV0ZXN0MDBZMBMGByqGSM49AgEGCCqGSM49
+AwEHA0IABDeSxfTQlf3w8Bizm3tXQJO/T+4ekZKr7BDEMaO9vaf4/aqJQTZ9UkMI
+lIKi6wswg+JPmZcoZhAQgbt0drPrPw2gXzBdBgkqhkiG9w0BCQ4xUDBOMAwGA1Ud
+EwEB/wQCMAAwHQYDVR0OBBYEFL4eoGQL6YYQqNYbC3fStuDI13sgMB8GA1UdIwQY
+MBaAFHBhuO3uixX7UdKDqrR7L8lyWN2AMAoGCCqGSM49BAMCA0gAMEUCIQCOipDe
+9zhSGYuqF6XIVRE8KIBsaIsFshzuc1JJGaFIzgIgM/CqEwEMirOgri3pw6BcItFO
+rj0Ij4yBa1Phy5NU+bo=
+-----END CERTIFICATE REQUEST-----
diff --git a/puppet/modules/kafka/files/ssl/test0/test0.key.private.pem b/puppet/modules/kafka/files/ssl/test0/test0.key.private.pem
new file mode 100644
index 0000000..f35b4dc
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/test0/test0.key.private.pem
@@ -0,0 +1,7 @@
+-----BEGIN ENCRYPTED PRIVATE KEY-----
+MIHsMFcGCSqGSIb3DQEFDTBKMCkGCSqGSIb3DQEFDDAcBAiWP25ydPgoiwICCAAw
+DAYIKoZIhvcNAgkFADAdBglghkgBZQMEASoEECBJbSrJH/pwbuc1sSUiM34EgZB9
++MLs/LExw2621Yk6PQjOXvbKUPdZnyXvmGzTe4OmsSuboVY9SRIbQcrsYgoAbrpC
+ya030PPOvGGjQBl2mvei7Maz8EUQZKdROPaQyNbpJfUrzAx6V8A9q6ZwJS2CttRu
+3siVNO8/xN89oyqTT0At+rC3aa4kyXar3nWwyUkCK8SrD7x07xpFivCfZivVZ7Q=
+-----END ENCRYPTED PRIVATE KEY-----
diff --git a/puppet/modules/kafka/files/ssl/test0/test0.key.public.pem b/puppet/modules/kafka/files/ssl/test0/test0.key.public.pem
new file mode 100644
index 0000000..b62fef4
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/test0/test0.key.public.pem
@@ -0,0 +1,4 @@
+-----BEGIN PUBLIC KEY-----
+MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEN5LF9NCV/fDwGLObe1dAk79P7h6R
+kqvsEMQxo729p/j9qolBNn1SQwiUgqLrCzCD4k+ZlyhmEBCBu3R2s+s/DQ==
+-----END PUBLIC KEY-----
diff --git a/puppet/modules/kafka/files/ssl/test0/test0.keystore.jks b/puppet/modules/kafka/files/ssl/test0/test0.keystore.jks
new file mode 100644
index 0000000..65e238d
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/test0/test0.keystore.jks
Binary files differ
diff --git a/puppet/modules/kafka/files/ssl/test0/test0.keystore.p12 b/puppet/modules/kafka/files/ssl/test0/test0.keystore.p12
new file mode 100644
index 0000000..d64712e
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/test0/test0.keystore.p12
Binary files differ
diff --git a/puppet/modules/kafka/files/ssl/test0/truststore.jks b/puppet/modules/kafka/files/ssl/test0/truststore.jks
new file mode 100644
index 0000000..453ff2c
--- /dev/null
+++ b/puppet/modules/kafka/files/ssl/test0/truststore.jks
Binary files differ
diff --git a/puppet/modules/kafka/manifests/init.pp b/puppet/modules/kafka/manifests/init.pp
index c7e48d7..4068dce 100644
--- a/puppet/modules/kafka/manifests/init.pp
+++ b/puppet/modules/kafka/manifests/init.pp
@@ -1,14 +1,15 @@
# == Class: Kafka
#
-class kafka {
+class kafka(
+ $ssl_enabled = true,
+) {
require ::service
require ::mediawiki::ready_service
- require ::kafka::repository
+ require kafka::repository

- $kafka_package = 'confluent-kafka-2.11'
- require_package('openjdk-8-jdk')
+ require_package('openjdk-8-jre')
require_package('zookeeperd')
- require_package($kafka_package)
+ require_package('confluent-kafka-2.11')
require_package('kafkacat')

$logdir = '/var/log/kafka'
@@ -16,7 +17,7 @@
group { 'kafka':
ensure => 'present',
system => true,
- require => Package[$kafka_package],
+ require => Package['confluent-kafka-2.11'],
}
# Kafka system user
user { 'kafka':
@@ -42,18 +43,29 @@
source => 'puppet:///modules/kafka/kafka.profile.sh',
}

+ if $ssl_enabled {
+ file { '/etc/kafka/ssl':
+ ensure => 'directory',
+ source => 'puppet:///modules/kafka/ssl',
+ recurse => true,
+ owner => 'root',
+ group => 'root',
+ mode => '0755',
+ }
+ }
+
file { '/etc/kafka/server.properties':
ensure => 'present',
- source => 'puppet:///modules/kafka/server.properties',
+ content => template('kafka/server.properties.erb'),
mode => '0444',
- require => Package[$kafka_package],
+ require => Package['confluent-kafka-2.11'],
}

file { '/etc/kafka/log4j.properties':
ensure => 'present',
content => template('kafka/log4j.properties.erb'),
mode => '0444',
- require => Package[$kafka_package],
+ require => Package['confluent-kafka-2.11'],
}

file { [$logdir, '/var/lib/kafka']:
@@ -61,7 +73,7 @@
owner => 'kafka',
group => 'kafka',
mode => '0755',
- require => Package[$kafka_package],
+ require => Package['confluent-kafka-2.11'],
}

service { 'zookeeper':
@@ -76,7 +88,7 @@
require => [
User['kafka'],
Service['zookeeper'],
- Package[$kafka_package],
+ Package['confluent-kafka-2.11'],
],
subscribe => [
File['/etc/kafka/server.properties'],
diff --git a/puppet/modules/kafka/templates/server.properties.erb b/puppet/modules/kafka/templates/server.properties.erb
new file mode 100644
index 0000000..f2c9fd4
--- /dev/null
+++ b/puppet/modules/kafka/templates/server.properties.erb
@@ -0,0 +1,133 @@
+# NOTE: This file is managed by Puppet.
+
+############################# Server Basics #############################
+
+# The id of the broker. This must be set to a unique integer for each broker.
+broker.id=0
+
+# Always require a static broker id.
+broker.id.generation.enable=false
+
+
+<% if @ssl_enabled -%>
+listeners=PLAINTEXT://:9092,SSL://:9093
+<% else -%>
+listeners=PLAINTEXT://:9092
+<% end -%>
+
+# Define whether the timestamp in the message is message create time or log append time.
+# The value should be either `CreateTime` or `LogAppendTime`
+log.message.timestamp.type=LogAppendTime
+
+######################### Socket Server Settings ########################
+<% if @ssl_enabled -%>
+security.inter.broker.protocol=SSL
+
+ssl.keystore.location=/etc/kafka/ssl/kafka_broker/kafka_broker.keystore.jks
+ssl.keystore.password=qwerty
+ssl.key.password=qwerty
+ssl.truststore.location=/etc/kafka/ssl/kafka_broker/truststore.jks
+ssl.truststore.password=qwerty
+ssl.enabled.protocols=TLSv1.2
+ssl.cipher.suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
+
+ssl.client.auth=requested
+
+<% end -%>
+
+# The number of threads doing disk I/O
+num.io.threads=1
+
+# The send buffer (SO_SNDBUF) used by the socket server
+socket.send.buffer.bytes=1048576
+
+# The receive buffer (SO_RCVBUF) used by the socket server
+socket.receive.buffer.bytes=1048576
+
+############################# Log Basics #############################
+
+# A comma seperated list of directories under which to store log files
+log.dirs=/var/lib/kafka
+
+# The default number of log partitions per topic. More partitions allow greater
+# parallelism for consumption, but this will also result in more files across
+# the brokers.
+num.partitions=1
+
+# The default replication factor for automatically created topics.
+# Default to the number of brokers in this cluster.
+default.replication.factor=1
+
+# Enables topic deletion
+delete.topic.enable=true
+
+# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
+# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
+offsets.topic.replication.factor=1
+
+# Enable auto creation of topic on the server. If this is set to true
+# then attempts to produce, consume, or fetch metadata for a non-existent
+# topic will automatically create it with the default replication factor
+# and number of partitions.
+auto.create.topics.enable=true
+
+# If this is enabled the controller will automatically try to balance
+# leadership for partitions among the brokers by periodically returning
+# leadership to the "preferred" replica for each partition if it is available.
+auto.leader.rebalance.enable=true
+
+# Number of threads used to replicate messages from leaders. Increasing this
+# value can increase the degree of I/O parallelism in the follower broker.
+# This is useful to temporarily increase if you have a broker that needs
+# to catch up on messages to get back into the ISR.
+num.replica.fetchers=1
+
+############################# Log Retention Policy #############################
+
+# The following configurations control the disposal of log segments. The policy
+# can be set to delete segments after a period of time, or after a given size
+# has accumulated. A segment will be deleted whenever *either* of these
+# criteria are met. Deletion always happens from the end of the log.
+
+# The minimum age of a log file to be eligible for deletion due to age
+log.retention.hours=168
+
+# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining
+# segments don't drop below log.retention.bytes. Functions independently of log.retention.hours.
+log.retention.bytes=268435456
+
+# Log retention window in minutes for offsets topic. If an offset
+# commit for a consumer group has not been recieved in this amount of
+# time, Kafka will drop the offset commit and consumers in the group
+# will have to start a new. This can be overridden in an offset commit
+# request.
+offsets.retention.minutes=10080
+
+############################# Zookeeper #############################
+
+# Zookeeper connection string (see zookeeper docs for details).
+# This is a comma separated host:port pairs, each corresponding to a zk
+# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
+# You can also append an optional chroot string to the urls to specify the
+# root directory for all kafka znodes.
+zookeeper.connect=localhost:2181/kafka
+
+
+##################### Confluent Proactive Support ######################
+# If set to true, and confluent-support-metrics package is installed
+# then the feature to collect and report support metrics
+confluent.support.metrics.enable=false
+
+# The customer ID under which support metrics will be collected and
+# reported.
+#
+# When the customer ID is set to "anonymous" (the default), then only a
+# reduced set of metrics is being collected and reported.
+#
+# Confluent customers
+# -------------------
+# If you are a Confluent customer, then you should replace the default
+# value with your actual Confluent customer ID. Doing so will ensure
+# that additional support metrics will be collected and reported.
+#
+confluent.support.customer.id=anonymous
diff --git a/puppet/modules/kafka/templates/systemd/kafka.erb b/puppet/modules/kafka/templates/systemd/kafka.erb
index d533515..ffcfb19 100644
--- a/puppet/modules/kafka/templates/systemd/kafka.erb
+++ b/puppet/modules/kafka/templates/systemd/kafka.erb
@@ -6,7 +6,7 @@
[Service]
User=kafka
Group=kafka
-Environment="KAFKA_HEAP_OPTS=-Xmx164m -Xmx164m"
+Environment="JAVA_OPTS=-Djava.awt.headless=true KAFKA_HEAP_OPTS=-Xmx164m -Xmx164m"
ExecStart=/usr/bin/kafka-server-start /etc/kafka/server.properties
Restart=always

diff --git a/puppet/modules/role/settings/kafka.yaml b/puppet/modules/role/settings/kafka.yaml
index fab9f85..19f8ea9 100644
--- a/puppet/modules/role/settings/kafka.yaml
+++ b/puppet/modules/role/settings/kafka.yaml
@@ -1 +1,4 @@
vagrant_ram: 128
+forward_ports:
+ 9092: 9092
+ 9093: 9093

--
To view, visit https://gerrit.wikimedia.org/r/404870
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I93d7c7cb98664e3e41b5a383ba8f9976a0b09099
Gerrit-PatchSet: 3
Gerrit-Project: mediawiki/vagrant
Gerrit-Branch: master
Gerrit-Owner: Ottomata <aotto@wikimedia.org>
Gerrit-Reviewer: BryanDavis <bdavis@wikimedia.org>
Gerrit-Reviewer: Dduvall <dduvall@wikimedia.org>
Gerrit-Reviewer: Ottomata <aotto@wikimedia.org>
Gerrit-Reviewer: jenkins-bot <>

_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits