好习惯:结论放最前
CPP+CPP
Python+Java
kafka的文档只有Java的API,因此一开始考虑在MXnet的Python API跨语言调用Kafka的Java API。大致思路是Python下运行Java的虚拟机。
参考:https://www.cnblogs.com/mumuli/p/5806963.html
Python+Python
后来找到第三方做Kafka的Python API的库,根据更新情况和文档情况,选择了Kafka-python。
根据文档在virtualenv
中安装pip install kafka-python
(之前已经安装MXNet)。
写了个test文件
from kafka import KafkaProducer
from kafka.client import KafkaClient
if __name__ == "__main__":
topic = KafkaClient().add_topic("test")
producer = KafkaProducer(bootstrap_servers='localhost:9092')
for _ in range(100):
producer.send('test', b'this is a message')
根据之前的介绍的单机启动
部分,启动本地kafka,并开启了一个consumer观察。
测试成功:
CPP+CPP
librdkafka的代码
基于librdkafka的high level wrapper API cppkafka代码
centos:
yum install libgsasl-devel.x86_64
yum install openssl-devel
yum install zlib zlib-devel
git clone git@github.com:edenhill/librdkafka.git
mkdir librdkafka-build
cd librdkafka
./configure --prefix=../librdkafka-build
make
sudo make install
# install protobuf: https://github.com/google/protobuf/blob/master/src/README.md
# compile the cppkafka: cmake and boost are needed
git clone git@github.com:mfontanini/cppkafka.git
mkdir cppkafka-build
cd cppkafka-build
# .so :
# cmake ../cppkafka -DRDKAFKA_ROOT_DIR=../librdkafka-build
# .a :
cmake ../cppkafka -DCPPKAFKA_BUILD_SHARED=0 -DRDKAFKA_ROOT_DIR=../librdkafka-build
make
make install
-- Install configuration: ""
-- Installing: /usr/local/lib/libcppkafka.a
-- Installing: /usr/local/include/cppkafka/message_builder.h
-- Installing: /usr/local/include/cppkafka/configuration_base.h
-- Installing: /usr/local/include/cppkafka/producer.h
-- Installing: /usr/local/include/cppkafka/error.h
-- Installing: /usr/local/include/cppkafka/configuration_option.h
-- Installing: /usr/local/include/cppkafka/topic.h
-- Installing: /usr/local/include/cppkafka/message.h
-- Installing: /usr/local/include/cppkafka/metadata.h
-- Installing: /usr/local/include/cppkafka/clonable_ptr.h
-- Installing: /usr/local/include/cppkafka/kafka_handle_base.h
-- Installing: /usr/local/include/cppkafka/configuration.h
-- Installing: /usr/local/include/cppkafka/group_information.h
-- Installing: /usr/local/include/cppkafka/topic_partition_list.h
-- Installing: /usr/local/include/cppkafka/topic_partition.h
-- Installing: /usr/local/include/cppkafka/buffer.h
-- Installing: /usr/local/include/cppkafka/exceptions.h
-- Installing: /usr/local/include/cppkafka/topic_configuration.h
-- Installing: /usr/local/include/cppkafka/macros.h
-- Installing: /usr/local/include/cppkafka/consumer.h
-- Installing: /usr/local/include/cppkafka/utils/consumer_dispatcher.h
-- Installing: /usr/local/include/cppkafka/utils/backoff_performer.h
-- Installing: /usr/local/include/cppkafka/utils/compacted_topic_processor.h
-- Installing: /usr/local/include/cppkafka/utils/backoff_committer.h
-- Installing: /usr/local/include/cppkafka/utils/buffered_producer.h
CPP+Java
JNI