Storm & Kafka 整合问题笔记

使用版本

<storm.version>1.1.0</storm.version>
<storm.kafka.version>0.8.2.1</storm.kafka.version>
<storm.kafka.artifact.id>kafka_2.10</storm.kafka.artifact.id>

<storm.kafka.client.version>0.10.0.0</storm.kafka.client.version>

问题1

本地启动报错：java.lang.NoClassDefFoundError: kafka/api/OffsetRequest

解决方案

pom.xml 中根据使用当前依赖可以解决，参照github storm源码范例

        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-kafka</artifactId>
            <version>${storm.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-kafka-client</artifactId>
            <version>${storm.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>${storm.kafka.artifact.id}</artifactId>
            <version>${storm.kafka.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.zookeeper</groupId>
                    <artifactId>zookeeper</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>${storm.kafka.client.version}</version>
        </dependency>

2017-08-07 15:38:42,385:ERROR Thread-26-spout-transferOpenTsdbTrident-executor[5 5] (NO_SOURCE_FILE:0) - 
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topic_metadata': Error reading array of size 554085, only 166 bytes available
    at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73) ~[kafka-clients-0.10.0.0.jar:?]