org.apache.spark.streaming.kafka

Class KafkaUtils

Object

org.apache.spark.streaming.kafka.KafkaUtils

public class KafkaUtils
extends Object

Constructor Summary

Constructors

Constructor and Description
KafkaUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static <K,V,U extends kafka.serializer.Decoder<?>,T extends kafka.serializer.Decoder<?>> JavaPairReceiverInputDStream<K,V>	createStream(JavaStreamingContext jssc, Class<K> keyTypeClass, Class<V> valueTypeClass, Class<U> keyDecoderClass, Class<T> valueDecoderClass, java.util.Map<String,String> kafkaParams, java.util.Map<String,Integer> topics, StorageLevel storageLevel) Create an input stream that pulls messages form a Kafka Broker.
static JavaPairReceiverInputDStream<String,String>	createStream(JavaStreamingContext jssc, String zkQuorum, String groupId, java.util.Map<String,Integer> topics) Create an input stream that pulls messages form a Kafka Broker.
static JavaPairReceiverInputDStream<String,String>	createStream(JavaStreamingContext jssc, String zkQuorum, String groupId, java.util.Map<String,Integer> topics, StorageLevel storageLevel) Create an input stream that pulls messages form a Kafka Broker.
static <K,V,U extends kafka.serializer.Decoder<?>,T extends kafka.serializer.Decoder<?>> ReceiverInputDStream<scala.Tuple2<K,V>>	createStream(StreamingContext ssc, scala.collection.immutable.Map<String,String> kafkaParams, scala.collection.immutable.Map<String,Object> topics, StorageLevel storageLevel, scala.reflect.ClassTag<K> evidence$1, scala.reflect.ClassTag<V> evidence$2, scala.reflect.Manifest<U> evidence$3, scala.reflect.Manifest<T> evidence$4) Create an input stream that pulls messages from a Kafka Broker.
static ReceiverInputDStream<scala.Tuple2<String,String>>	createStream(StreamingContext ssc, String zkQuorum, String groupId, scala.collection.immutable.Map<String,Object> topics, StorageLevel storageLevel) Create an input stream that pulls messages from a Kafka Broker.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KafkaUtils

public KafkaUtils()

Method Detail

createStream

public static ReceiverInputDStream<scala.Tuple2<String,String>> createStream(StreamingContext ssc,
                                                             String zkQuorum,
                                                             String groupId,
                                                             scala.collection.immutable.Map<String,Object> topics,
                                                             StorageLevel storageLevel)

Create an input stream that pulls messages from a Kafka Broker.

Parameters:

ssc - StreamingContext object

zkQuorum - Zookeeper quorum (hostname:port,hostname:port,..)

groupId - The group id for this consumer

topics - Map of (topic_name -> numPartitions) to consume. Each partition is consumed in its own thread

storageLevel - Storage level to use for storing the received objects (default: StorageLevel.MEMORY_AND_DISK_SER_2)

createStream

public static <K,V,U extends kafka.serializer.Decoder<?>,T extends kafka.serializer.Decoder<?>> ReceiverInputDStream<scala.Tuple2<K,V>> createStream(StreamingContext ssc,
                                                                                                                                     scala.collection.immutable.Map<String,String> kafkaParams,
                                                                                                                                     scala.collection.immutable.Map<String,Object> topics,
                                                                                                                                     StorageLevel storageLevel,
                                                                                                                                     scala.reflect.ClassTag<K> evidence$1,
                                                                                                                                     scala.reflect.ClassTag<V> evidence$2,
                                                                                                                                     scala.reflect.Manifest<U> evidence$3,
                                                                                                                                     scala.reflect.Manifest<T> evidence$4)

Create an input stream that pulls messages from a Kafka Broker.

Parameters:

ssc - StreamingContext object

kafkaParams - Map of kafka configuration parameters, see http://kafka.apache.org/08/configuration.html

topics - Map of (topic_name -> numPartitions) to consume. Each partition is consumed in its own thread.

storageLevel - Storage level to use for storing the received objects

createStream

public static JavaPairReceiverInputDStream<String,String> createStream(JavaStreamingContext jssc,
                                                       String zkQuorum,
                                                       String groupId,
                                                       java.util.Map<String,Integer> topics)

Create an input stream that pulls messages form a Kafka Broker. Storage level of the data will be the default StorageLevel.MEMORY_AND_DISK_SER_2.

Parameters:

jssc - JavaStreamingContext object

zkQuorum - Zookeeper quorum (hostname:port,hostname:port,..)

groupId - The group id for this consumer

topics - Map of (topic_name -> numPartitions) to consume. Each partition is consumed in its own thread

createStream

public static JavaPairReceiverInputDStream<String,String> createStream(JavaStreamingContext jssc,
                                                       String zkQuorum,
                                                       String groupId,
                                                       java.util.Map<String,Integer> topics,
                                                       StorageLevel storageLevel)

Create an input stream that pulls messages form a Kafka Broker.

Parameters:

jssc - JavaStreamingContext object

zkQuorum - Zookeeper quorum (hostname:port,hostname:port,..).

groupId - The group id for this consumer.

topics - Map of (topic_name -> numPartitions) to consume. Each partition is consumed in its own thread.

storageLevel - RDD storage level.

createStream

public static <K,V,U extends kafka.serializer.Decoder<?>,T extends kafka.serializer.Decoder<?>> JavaPairReceiverInputDStream<K,V> createStream(JavaStreamingContext jssc,
                                                                                                                               Class<K> keyTypeClass,
                                                                                                                               Class<V> valueTypeClass,
                                                                                                                               Class<U> keyDecoderClass,
                                                                                                                               Class<T> valueDecoderClass,
                                                                                                                               java.util.Map<String,String> kafkaParams,
                                                                                                                               java.util.Map<String,Integer> topics,
                                                                                                                               StorageLevel storageLevel)

Create an input stream that pulls messages form a Kafka Broker.

Parameters:

jssc - JavaStreamingContext object

keyTypeClass - Key type of RDD

valueTypeClass - value type of RDD

keyDecoderClass - Type of kafka key decoder

valueDecoderClass - Type of kafka value decoder

kafkaParams - Map of kafka configuration parameters, see http://kafka.apache.org/08/configuration.html

topics - Map of (topic_name -> numPartitions) to consume. Each partition is consumed in its own thread

storageLevel - RDD storage level.