Class KafkaProducer<K,V>
- java.lang.Object
-
- org.apache.kafka.clients.producer.KafkaProducer<K,V>
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
,Producer<K,V>
public class KafkaProducer<K,V> extends Object implements Producer<K,V>
A Kafka client that publishes records to the Kafka cluster.The producer is thread safe and sharing a single producer instance across threads will generally be faster than having multiple instances.
Here is a simple example of using the producer to send records with strings containing sequential numbers as the key/value pairs.
Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("linger.ms", 1); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); Producer<String, String> producer = new KafkaProducer<>(props); for (int i = 0; i < 100; i++) producer.send(new ProducerRecord<String, String>("my-topic", Integer.toString(i), Integer.toString(i))); producer.close();
The producer consists of a pool of buffer space that holds records that haven't yet been transmitted to the server as well as a background I/O thread that is responsible for turning these records into requests and transmitting them to the cluster. Failure to close the producer after use will leak these resources.
The
send()
method is asynchronous. When called, it adds the record to a buffer of pending record sends and immediately returns. This allows the producer to batch together individual records for efficiency.The
acks
config controls the criteria under which requests are considered complete. The default setting "all" will result in blocking on the full commit of the record, the slowest but most durable setting.If the request fails, the producer can automatically retry. The
retries
setting defaults toInteger.MAX_VALUE
, and it's recommended to usedelivery.timeout.ms
to control retry behavior, instead ofretries
.The producer maintains buffers of unsent records for each partition. These buffers are of a size specified by the
batch.size
config. Making this larger can result in more batching, but requires more memory (since we will generally have one of these buffers for each active partition).By default a buffer is available to send immediately even if there is additional unused space in the buffer. However if you want to reduce the number of requests you can set
linger.ms
to something greater than 0. This will instruct the producer to wait up to that number of milliseconds before sending a request in hope that more records will arrive to fill up the same batch. This is analogous to Nagle's algorithm in TCP. For example, in the code snippet above, likely all 100 records would be sent in a single request since we set our linger time to 1 millisecond. However this setting would add 1 millisecond of latency to our request waiting for more records to arrive if we didn't fill up the buffer. Note that records that arrive close together in time will generally batch together even withlinger.ms=0
. So, under heavy load, batching will occur regardless of the linger configuration; however setting this to something larger than 0 can lead to fewer, more efficient requests when not under maximal load at the cost of a small amount of latency.The
buffer.memory
controls the total amount of memory available to the producer for buffering. If records are sent faster than they can be transmitted to the server then this buffer space will be exhausted. When the buffer space is exhausted additional send calls will block. The threshold for time to block is determined bymax.block.ms
after which it throws a TimeoutException.The
key.serializer
andvalue.serializer
instruct how to turn the key and value objects the user provides with theirProducerRecord
into bytes. You can use the includedByteArraySerializer
orStringSerializer
for simple byte or string types.From Kafka 0.11, the KafkaProducer supports two additional modes: the idempotent producer and the transactional producer. The idempotent producer strengthens Kafka's delivery semantics from at least once to exactly once delivery. In particular producer retries will no longer introduce duplicates. The transactional producer allows an application to send messages to multiple partitions (and topics!) atomically.
From Kafka 3.0, the
enable.idempotence
configuration defaults to true. When enabling idempotence,retries
config will default toInteger.MAX_VALUE
and theacks
config will default toall
. There are no API changes for the idempotent producer, so existing applications will not need to be modified to take advantage of this feature.To take advantage of the idempotent producer, it is imperative to avoid application level re-sends since these cannot be de-duplicated. As such, if an application enables idempotence, it is recommended to leave the
retries
config unset, as it will be defaulted toInteger.MAX_VALUE
. Additionally, if asend(ProducerRecord)
returns an error even with infinite retries (for instance if the message expires in the buffer before being sent), then it is recommended to shut down the producer and check the contents of the last produced message to ensure that it is not duplicated. Finally, the producer can only guarantee idempotence for messages sent within a single session.To use the transactional producer and the attendant APIs, you must set the
transactional.id
configuration property. If thetransactional.id
is set, idempotence is automatically enabled along with the producer configs which idempotence depends on. Further, topics which are included in transactions should be configured for durability. In particular, thereplication.factor
should be at least3
, and themin.insync.replicas
for these topics should be set to 2. Finally, in order for transactional guarantees to be realized from end-to-end, the consumers must be configured to read only committed messages as well.The purpose of the
transactional.id
is to enable transaction recovery across multiple sessions of a single producer instance. It would typically be derived from the shard identifier in a partitioned, stateful, application. As such, it should be unique to each producer instance running within a partitioned application.All the new transactional APIs are blocking and will throw exceptions on failure. The example below illustrates how the new APIs are meant to be used. It is similar to the example above, except that all 100 messages are part of a single transaction.
Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("transactional.id", "my-transactional-id"); Producer<String, String> producer = new KafkaProducer<>(props, new StringSerializer(), new StringSerializer()); producer.initTransactions(); try { producer.beginTransaction(); for (int i = 0; i < 100; i++) producer.send(new ProducerRecord<>("my-topic", Integer.toString(i), Integer.toString(i))); producer.commitTransaction(); } catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) { // We can't recover from these exceptions, so our only option is to close the producer and exit. producer.close(); } catch (KafkaException e) { // For all other exceptions, just abort the transaction and try again. producer.abortTransaction(); } producer.close();
As is hinted at in the example, there can be only one open transaction per producer. All messages sent between the
beginTransaction()
andcommitTransaction()
calls will be part of a single transaction. When thetransactional.id
is specified, all messages sent by the producer must be part of a transaction.The transactional producer uses exceptions to communicate error states. In particular, it is not required to specify callbacks for
By callingproducer.send()
or to call.get()
on the returned Future: aKafkaException
would be thrown if any of theproducer.send()
or transactional calls hit an irrecoverable error during a transaction. See thesend(ProducerRecord)
documentation for more details about detecting errors from a transactional send.producer.abortTransaction()
upon receiving aKafkaException
we can ensure that any successful writes are marked as aborted, hence keeping the transactional guarantees.This client can communicate with brokers that are version 0.10.0 or newer. Older or newer brokers may not support certain client features. For instance, the transactional APIs need broker versions 0.11.0 or later. You will receive an
UnsupportedVersionException
when invoking an API that is not available in the running broker version.
-
-
Field Summary
Fields Modifier and Type Field Description static String
NETWORK_THREAD_PREFIX
static String
PRODUCER_METRIC_GROUP_NAME
-
Constructor Summary
Constructors Constructor Description KafkaProducer(Map<String,Object> configs)
A producer is instantiated by providing a set of key-value pairs as configuration.KafkaProducer(Map<String,Object> configs, Serializer<K> keySerializer, Serializer<V> valueSerializer)
A producer is instantiated by providing a set of key-value pairs as configuration, a key and a valueSerializer
.KafkaProducer(Properties properties)
A producer is instantiated by providing a set of key-value pairs as configuration.KafkaProducer(Properties properties, Serializer<K> keySerializer, Serializer<V> valueSerializer)
A producer is instantiated by providing a set of key-value pairs as configuration, a key and a valueSerializer
.
-
Method Summary
All Methods Instance Methods Concrete Methods Deprecated Methods Modifier and Type Method Description void
abortTransaction()
Not supported in MapR Producer.void
beginTransaction()
Not supported in MapR Producer.void
close()
Close this producer.void
close(Duration timeout)
This method waits up totimeout
for the producer to complete the sending of all incomplete requests.void
commitTransaction()
Not supported in MapR Producer.void
flush()
Invoking this method makes all buffered records immediately available to send (even iflinger.ms
is greater than 0) and blocks on the completion of the requests associated with these records.void
initTransactions()
Not supported in MapR Producer.Map<MetricName,? extends Metric>
metrics()
Get the full set of internal metrics maintained by the producer.List<PartitionInfo>
partitionsFor(String topic)
Get the partition metadata for the given topic.Future<RecordMetadata>
send(ProducerRecord<K,V> record)
Asynchronously send a record to a topic.Future<RecordMetadata>
send(ProducerRecord<K,V> record, Callback callback)
Asynchronously send a record to a topic and invoke the provided callback when the send has been acknowledged.void
sendOffsetsToTransaction(Map<TopicPartition,OffsetAndMetadata> offsets, String consumerGroupId)
Deprecated.Since 3.0.0, please usesendOffsetsToTransaction(Map, ConsumerGroupMetadata)
instead.void
sendOffsetsToTransaction(Map<TopicPartition,OffsetAndMetadata> offsets, ConsumerGroupMetadata groupMetadata)
Not supported in MapR Producer.
-
-
-
Field Detail
-
NETWORK_THREAD_PREFIX
public static final String NETWORK_THREAD_PREFIX
- See Also:
- Constant Field Values
-
PRODUCER_METRIC_GROUP_NAME
public static final String PRODUCER_METRIC_GROUP_NAME
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
KafkaProducer
public KafkaProducer(Map<String,Object> configs)
A producer is instantiated by providing a set of key-value pairs as configuration. Valid configuration strings are documented here. Values can be either strings or Objects of the appropriate type (for example a numeric configuration would accept either the string "42" or the integer 42).Note: after creating a
KafkaProducer
you must alwaysclose()
it to avoid resource leaks.- Parameters:
configs
- The producer configs
-
KafkaProducer
public KafkaProducer(Map<String,Object> configs, Serializer<K> keySerializer, Serializer<V> valueSerializer)
A producer is instantiated by providing a set of key-value pairs as configuration, a key and a valueSerializer
. Valid configuration strings are documented here. Values can be either strings or Objects of the appropriate type (for example a numeric configuration would accept either the string "42" or the integer 42).Note: after creating a
KafkaProducer
you must alwaysclose()
it to avoid resource leaks.- Parameters:
configs
- The producer configskeySerializer
- The serializer for key that implementsSerializer
. The configure() method won't be called in the producer when the serializer is passed in directly.valueSerializer
- The serializer for value that implementsSerializer
. The configure() method won't be called in the producer when the serializer is passed in directly.
-
KafkaProducer
public KafkaProducer(Properties properties)
A producer is instantiated by providing a set of key-value pairs as configuration. Valid configuration strings are documented here.Note: after creating a
KafkaProducer
you must alwaysclose()
it to avoid resource leaks.- Parameters:
properties
- The producer configs
-
KafkaProducer
public KafkaProducer(Properties properties, Serializer<K> keySerializer, Serializer<V> valueSerializer)
A producer is instantiated by providing a set of key-value pairs as configuration, a key and a valueSerializer
. Valid configuration strings are documented here.Note: after creating a
KafkaProducer
you must alwaysclose()
it to avoid resource leaks.- Parameters:
properties
- The producer configskeySerializer
- The serializer for key that implementsSerializer
. The configure() method won't be called in the producer when the serializer is passed in directly.valueSerializer
- The serializer for value that implementsSerializer
. The configure() method won't be called in the producer when the serializer is passed in directly.
-
-
Method Detail
-
initTransactions
public void initTransactions()
Not supported in MapR Producer. Needs to be called before any other methods when thetransactional.id
is set in the configuration. This method does the following:- Ensures any transactions initiated by previous instances of the producer with the same
transactional.id
are completed. If the previous instance had failed with a transaction in progress, it will be aborted. If the last transaction had begun completion, but not yet finished, this method awaits its completion. - Gets the internal producer id and epoch, used in all future transactional messages issued by the producer.
TimeoutException
if the transactional state cannot be initialized before expiration ofmax.block.ms
. Additionally, it will raiseInterruptException
if interrupted. It is safe to retry in either case, but once the transactional state has been successfully initialized, this method should no longer be used.- Specified by:
initTransactions
in interfaceProducer<K,V>
- Throws:
IllegalStateException
- if notransactional.id
has been configuredUnsupportedVersionException
- fatal error indicating the broker does not support transactions (i.e. if its version is lower than 0.11.0.0)AuthorizationException
- error indicating that the configured transactional.id is not authorized, or the idempotent producer id is unavailable. See the exception for more details. User may retry this function call after fixing the permission.KafkaException
- if the producer has encountered a previous fatal error or for any other unexpected errorTimeoutException
- if the time taken for initialize the transaction has surpassedmax.block.ms
.InterruptException
- if the thread is interrupted while blocked
- Ensures any transactions initiated by previous instances of the producer with the same
-
beginTransaction
public void beginTransaction() throws ProducerFencedException
Not supported in MapR Producer. Should be called before the start of each new transaction. Note that prior to the first invocation of this method, you must invokeinitTransactions()
exactly one time.- Specified by:
beginTransaction
in interfaceProducer<K,V>
- Throws:
IllegalStateException
- if notransactional.id
has been configured or ifinitTransactions()
has not yet been invokedProducerFencedException
- if another producer with the same transactional.id is activeInvalidProducerEpochException
- if the producer has attempted to produce with an old epoch to the partition leader. See the exception for more detailsUnsupportedVersionException
- fatal error indicating the broker does not support transactions (i.e. if its version is lower than 0.11.0.0)AuthorizationException
- fatal error indicating that the configuredtransactional.id
is not authorized. See the exception for more detailsKafkaException
- if the producer has encountered a previous fatal error or for any other unexpected error
-
sendOffsetsToTransaction
@Deprecated public void sendOffsetsToTransaction(Map<TopicPartition,OffsetAndMetadata> offsets, String consumerGroupId) throws ProducerFencedException
Deprecated.Since 3.0.0, please usesendOffsetsToTransaction(Map, ConsumerGroupMetadata)
instead.Not supported in MapR Producer. Sends a list of specified offsets to the consumer group coordinator, and also marks those offsets as part of the current transaction. These offsets will be considered committed only if the transaction is committed successfully. The committed offset should be the next message your application will consume, i.e. lastProcessedMessageOffset + 1.This method should be used when you need to batch consumed and produced messages together, typically in a consume-transform-produce pattern. Thus, the specified
consumerGroupId
should be the same as config parametergroup.id
of the usedconsumer
. Note, that the consumer should haveenable.auto.commit=false
and should also not commit offsets manually (viasync
orasync
commits).This method is a blocking call that waits until the request has been received and acknowledged by the consumer group coordinator; but the offsets are not considered as committed until the transaction itself is successfully committed later (via the
commitTransaction()
call).- Specified by:
sendOffsetsToTransaction
in interfaceProducer<K,V>
- Throws:
IllegalStateException
- if no transactional.id has been configured, no transaction has been startedProducerFencedException
- fatal error indicating another producer with the same transactional.id is activeUnsupportedVersionException
- fatal error indicating the broker does not support transactions (i.e. if its version is lower than 0.11.0.0)UnsupportedForMessageFormatException
- fatal error indicating the message format used for the offsets topic on the broker does not support transactionsAuthorizationException
- fatal error indicating that the configured transactional.id is not authorized, or the consumer group id is not authorized.InvalidProducerEpochException
- if the producer has attempted to produce with an old epoch to the partition leader. See the exception for more detailsTimeoutException
- if the time taken for sending the offsets has surpassedmax.block.ms
.KafkaException
- if the producer has encountered a previous fatal or abortable error, or for any other unexpected error
-
sendOffsetsToTransaction
public void sendOffsetsToTransaction(Map<TopicPartition,OffsetAndMetadata> offsets, ConsumerGroupMetadata groupMetadata) throws ProducerFencedException
Not supported in MapR Producer. Sends a list of specified offsets to the consumer group coordinator, and also marks those offsets as part of the current transaction. These offsets will be considered committed only if the transaction is committed successfully. The committed offset should be the next message your application will consume, i.e. lastProcessedMessageOffset + 1.This method should be used when you need to batch consumed and produced messages together, typically in a consume-transform-produce pattern. Thus, the specified
groupMetadata
should be extracted from the usedconsumer
viaKafkaConsumer.groupMetadata()
to leverage consumer group metadata. This will provide stronger fencing than just supplying theconsumerGroupId
and passing innew ConsumerGroupMetadata(consumerGroupId)
, however note that the full set of consumer group metadata returned byKafkaConsumer.groupMetadata()
requires the brokers to be on version 2.5 or newer to understand.This method is a blocking call that waits until the request has been received and acknowledged by the consumer group coordinator; but the offsets are not considered as committed until the transaction itself is successfully committed later (via the
commitTransaction()
call).Note, that the consumer should have
enable.auto.commit=false
and should also not commit offsets manually (viasync
orasync
commits). This method will raiseTimeoutException
if the producer cannot send offsets before expiration ofmax.block.ms
. Additionally, it will raiseInterruptException
if interrupted.- Specified by:
sendOffsetsToTransaction
in interfaceProducer<K,V>
- Throws:
IllegalStateException
- if no transactional.id has been configured or no transaction has been started.ProducerFencedException
- fatal error indicating another producer with the same transactional.id is activeUnsupportedVersionException
- fatal error indicating the broker does not support transactions (i.e. if its version is lower than 0.11.0.0) or the broker doesn't support the latest version of transactional API with all consumer group metadata (i.e. if its version is lower than 2.5.0).UnsupportedForMessageFormatException
- fatal error indicating the message format used for the offsets topic on the broker does not support transactionsAuthorizationException
- fatal error indicating that the configured transactional.id is not authorized, or the consumer group id is not authorized.CommitFailedException
- if the commit failed and cannot be retried (e.g. if the consumer has been kicked out of the group). Users should handle this by aborting the transaction.FencedInstanceIdException
- if this producer instance gets fenced by broker due to a mis-configured consumer instance id within group metadata.InvalidProducerEpochException
- if the producer has attempted to produce with an old epoch to the partition leader. See the exception for more detailsKafkaException
- if the producer has encountered a previous fatal or abortable error, or for any other unexpected errorTimeoutException
- if the time taken for sending the offsets has surpassedmax.block.ms
.InterruptException
- if the thread is interrupted while blocked
-
commitTransaction
public void commitTransaction() throws ProducerFencedException
Not supported in MapR Producer. Commits the ongoing transaction. This method will flush any unsent records before actually committing the transaction.Further, if any of the
send(ProducerRecord)
calls which were part of the transaction hit irrecoverable errors, this method will throw the last received exception immediately and the transaction will not be committed. So allsend(ProducerRecord)
calls in a transaction must succeed in order for this method to succeed.If the transaction is committed successfully and this method returns without throwing an exception, it is guaranteed that all
callbacks
for records in the transaction will have been invoked and completed. Note that exceptions thrown by callbacks are ignored; the producer proceeds to commit the transaction in any case.Note that this method will raise
TimeoutException
if the transaction cannot be committed before expiration ofmax.block.ms
, but this does not mean the request did not actually reach the broker. In fact, it only indicates that we cannot get the acknowledgement response in time, so it's up to the application's logic to decide how to handle timeouts. Additionally, it will raiseInterruptException
if interrupted. It is safe to retry in either case, but it is not possible to attempt a different operation (such as abortTransaction) since the commit may already be in the progress of completing. If not retrying, the only option is to close the producer.- Specified by:
commitTransaction
in interfaceProducer<K,V>
- Throws:
IllegalStateException
- if no transactional.id has been configured or no transaction has been startedProducerFencedException
- fatal error indicating another producer with the same transactional.id is activeUnsupportedVersionException
- fatal error indicating the broker does not support transactions (i.e. if its version is lower than 0.11.0.0)AuthorizationException
- fatal error indicating that the configured transactional.id is not authorized. See the exception for more detailsInvalidProducerEpochException
- if the producer has attempted to produce with an old epoch to the partition leader. See the exception for more detailsKafkaException
- if the producer has encountered a previous fatal or abortable error, or for any other unexpected errorTimeoutException
- if the time taken for committing the transaction has surpassedmax.block.ms
.InterruptException
- if the thread is interrupted while blocked
-
abortTransaction
public void abortTransaction() throws ProducerFencedException
Not supported in MapR Producer. Aborts the ongoing transaction. Any unflushed produce messages will be aborted when this call is made. This call will throw an exception immediately if any priorsend(ProducerRecord)
calls failed with aProducerFencedException
or an instance ofAuthorizationException
.Note that this method will raise
TimeoutException
if the transaction cannot be aborted before expiration ofmax.block.ms
, but this does not mean the request did not actually reach the broker. In fact, it only indicates that we cannot get the acknowledgement response in time, so it's up to the application's logic to decide how to handle timeouts. Additionally, it will raiseInterruptException
if interrupted. It is safe to retry in either case, but it is not possible to attempt a different operation (such ascommitTransaction()
) since the abort may already be in the progress of completing. If not retrying, the only option is to close the producer.- Specified by:
abortTransaction
in interfaceProducer<K,V>
- Throws:
IllegalStateException
- if no transactional.id has been configured or no transaction has been startedProducerFencedException
- fatal error indicating another producer with the same transactional.id is activeInvalidProducerEpochException
- if the producer has attempted to produce with an old epoch to the partition leader. See the exception for more detailsUnsupportedVersionException
- fatal error indicating the broker does not support transactions (i.e. if its version is lower than 0.11.0.0)AuthorizationException
- fatal error indicating that the configured transactional.id is not authorized. See the exception for more detailsKafkaException
- if the producer has encountered a previous fatal error or for any other unexpected errorTimeoutException
- if the time taken for aborting the transaction has surpassedmax.block.ms
.InterruptException
- if the thread is interrupted while blocked
-
send
public Future<RecordMetadata> send(ProducerRecord<K,V> record)
Asynchronously send a record to a topic. Equivalent tosend(record, null)
. Seesend(ProducerRecord, Callback)
for details.
-
send
public Future<RecordMetadata> send(ProducerRecord<K,V> record, Callback callback)
Asynchronously send a record to a topic and invoke the provided callback when the send has been acknowledged.The send is asynchronous and this method will return immediately once the record has been stored in the buffer of records waiting to be sent. This allows sending many records in parallel without blocking to wait for the response after each one.
The result of the send is a
RecordMetadata
specifying the partition the record was sent to, the offset it was assigned and the timestamp of the record. If the producer is configured with acks = 0, theRecordMetadata
will have offset = -1 because the producer does not wait for the acknowledgement from the broker. IfCreateTime
is used by the topic, the timestamp will be the user provided timestamp or the record send time if the user did not specify a timestamp for the record. IfLogAppendTime
is used for the topic, the timestamp will be the Kafka broker local time when the message is appended.Since the send call is asynchronous it returns a
Future
for theRecordMetadata
that will be assigned to this record. Invokingget()
on this future will block until the associated request completes and then return the metadata for the record or throw any exception that occurred while sending the record.If you want to simulate a simple blocking call you can call the
get()
method immediately:byte[] key = "key".getBytes(); byte[] value = "value".getBytes(); ProducerRecord<byte[],byte[]> record = new ProducerRecord<byte[],byte[]>("my-topic", key, value) producer.send(record).get();
Fully non-blocking usage can make use of the
Callback
parameter to provide a callback that will be invoked when the request is complete.ProducerRecord<byte[],byte[]> record = new ProducerRecord<byte[],byte[]>("the-topic", key, value); producer.send(myRecord, new Callback() { public void onCompletion(RecordMetadata metadata, Exception e) { if(e != null) { e.printStackTrace(); } else { System.out.println("The offset of the record we just sent is: " + metadata.offset()); } } });
callback1
is guaranteed to execute beforecallback2
:producer.send(new ProducerRecord<byte[],byte[]>(topic, partition, key1, value1), callback1); producer.send(new ProducerRecord<byte[],byte[]>(topic, partition, key2, value2), callback2);
When used as part of a transaction, it is not necessary to define a callback or check the result of the future in order to detect errors from
send
. If any of the send calls failed with an irrecoverable error, the finalcommitTransaction()
call will fail and throw the exception from the last failed send. When this happens, your application should callabortTransaction()
to reset the state and continue to send data.Some transactional send errors cannot be resolved with a call to
abortTransaction()
. In particular, if a transactional send finishes with aProducerFencedException
, aOutOfOrderSequenceException
, aUnsupportedVersionException
, or anAuthorizationException
, then the only option left is to callclose()
. Fatal errors cause the producer to enter a defunct state in which future API calls will continue to raise the same underlying error wrapped in a newKafkaException
.It is a similar picture when idempotence is enabled, but no
transactional.id
has been configured. In this case,UnsupportedVersionException
andAuthorizationException
are considered fatal errors. However,ProducerFencedException
does not need to be handled. Additionally, it is possible to continue sending after receiving anOutOfOrderSequenceException
, but doing so can result in out of order delivery of pending messages. To ensure proper ordering, you should close the producer and create a new instance.If the message format of the destination topic is not upgraded to 0.11.0.0, idempotent and transactional produce requests will fail with an
UnsupportedForMessageFormatException
error. If this is encountered during a transaction, it is possible to abort and continue. But note that future sends to the same topic will continue receiving the same exception until the topic is upgraded.Note that callbacks will generally execute in the I/O thread of the producer and so should be reasonably fast or they will delay the sending of messages from other threads. If you want to execute blocking or computationally expensive callbacks it is recommended to use your own
Executor
in the callback body to parallelize processing.- Specified by:
send
in interfaceProducer<K,V>
- Parameters:
record
- The record to sendcallback
- A user-supplied callback to execute when the record has been acknowledged by the server (null indicates no callback)- Throws:
AuthenticationException
- if authentication fails. See the exception for more detailsAuthorizationException
- fatal error indicating that the producer is not allowed to writeIllegalStateException
- if a transactional.id has been configured and no transaction has been started, or when send is invoked after producer has been closed.InterruptException
- If the thread is interrupted while blockedSerializationException
- If the key or value are not valid objects given the configured serializersTimeoutException
- If the record could not be appended to the send buffer due to memory unavailable or missing metadata withinmax.block.ms
.KafkaException
- If a Kafka related error occurs that does not belong to the public API exceptions.
-
flush
public void flush()
Invoking this method makes all buffered records immediately available to send (even iflinger.ms
is greater than 0) and blocks on the completion of the requests associated with these records. The post-condition offlush()
is that any previously sent record will have completed (e.g.Future.isDone() == true
). A request is considered completed when it is successfully acknowledged according to theacks
configuration you have specified or else it results in an error.Other threads can continue sending records while one thread is blocked waiting for a flush call to complete, however no guarantee is made about the completion of records sent after the flush call begins.
This method can be useful when consuming from some input system and producing into Kafka. The
flush()
call gives a convenient way to ensure all previously sent messages have actually completed.This example shows how to consume from one Kafka topic and produce to another Kafka topic:
for(ConsumerRecord<String, String> record: consumer.poll(100)) producer.send(new ProducerRecord("my-topic", record.key(), record.value()); producer.flush(); consumer.commitSync();
retries=<large_number>
in our config.Applications don't need to call this method for transactional producers, since the
commitTransaction()
will flush all buffered records before performing the commit. This ensures that all thesend(ProducerRecord)
calls made since the previousbeginTransaction()
are completed before the commit.- Specified by:
flush
in interfaceProducer<K,V>
- Throws:
InterruptException
- If the thread is interrupted while blocked
-
partitionsFor
public List<PartitionInfo> partitionsFor(String topic)
Get the partition metadata for the given topic. This can be used for custom partitioning.- Specified by:
partitionsFor
in interfaceProducer<K,V>
- Throws:
AuthenticationException
- if authentication fails. See the exception for more detailsAuthorizationException
- if not authorized to the specified topic. See the exception for more detailsInterruptException
- if the thread is interrupted while blockedTimeoutException
- if metadata could not be refreshed withinmax.block.ms
KafkaException
- for all Kafka-related exceptions, including the case where this method is called after producer close
-
metrics
public Map<MetricName,? extends Metric> metrics()
Get the full set of internal metrics maintained by the producer. For MapR Producer,ProducerConfig.METRICS_ENABLED_CONFIG
config must be set to true, otherwise it returns an empty map. See MapR Kafka metrics documentation for details.
-
close
public void close()
Close this producer. This method blocks until all previously sent requests complete. This method is equivalent toclose(Long.MAX_VALUE, TimeUnit.MILLISECONDS)
.If close() is called from
Callback
, a warning message will be logged and close(0, TimeUnit.MILLISECONDS) will be called instead. We do this because the sender thread would otherwise try to join itself and block forever.- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Specified by:
close
in interfaceProducer<K,V>
- Throws:
InterruptException
- If the thread is interrupted while blocked.KafkaException
- If an unexpected error occurs while trying to close the client, this error should be treated as fatal and indicate the client is no longer usable.
-
close
public void close(Duration timeout)
This method waits up totimeout
for the producer to complete the sending of all incomplete requests.If the producer is unable to complete all requests before the timeout expires, this method will fail any unsent and unacknowledged records immediately. It will also abort the ongoing transaction if it's not already completing.
If invoked from within a
Callback
this method will not block and will be equivalent toclose(Duration.ofMillis(0))
. This is done since no further sending will happen while blocking the I/O thread of the producer.- Specified by:
close
in interfaceProducer<K,V>
- Parameters:
timeout
- The maximum time to wait for producer to complete any pending requests. The value should be non-negative. Specifying a timeout of zero means do not wait for pending send requests to complete.- Throws:
InterruptException
- If the thread is interrupted while blocked.KafkaException
- If an unexpected error occurs while trying to close the client, this error should be treated as fatal and indicate the client is no longer usable.IllegalArgumentException
- If thetimeout
is negative.
-
-