Differences between HPE Ezmeral Data Fabric Streams and Apache Kafka Configuration
Describes the HPE Ezmeral Data Fabric Streams supportability of Apache Kafka configuration parameters for producers and consumers.
| Name | Description | Supported for producers in HPE Ezmeral Data Fabric Streams? |
|---|---|---|
bootstrap.servers |
A list of host/port pairs to use for establishing the initial
connection to the Kafka cluster. The client will make use of all servers
irrespective of which servers are specified here for bootstrapping—this list only
impacts the initial hosts used to discover the full set of servers. This list should
be in the form host1:port1,host2:port2,.... Since these servers are
just used for the initial connection to discover the full cluster membership (which
may change dynamically), this list need not contain the full set of servers (you may
want more than one, though, in case a server is down). |
No. Cluster details are discovered from the file
mapr-clusters.conf. |
key.serializer |
Serializer class for key that implements the
Serializer interface. |
Yes |
value.serializer |
Serializer class for value that implements the
Serializer interface. |
Yes |
acks |
The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. The following settings are common: | Ignored, all writes in HPE Ezmeral Data Fabric Streams are synchronous, and number of replicas is determined at the volume level, with a default of 3. |
acks=0
If set to zero then the producer will not wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the retries configuration will not take effect (as the client won't generally know of any failures). The offset given back for each record will always be set to -1. |
||
acks=1
This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgement from all followers. In this case should the leader fail immediately after acknowledging the record but before the followers have replicated it then the record will be lost. |
||
acks=all
This means the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee. |
||
buffer.memory |
The total bytes of memory the producer can use to buffer records
waiting to be sent to the server. If records are sent faster than they can be
delivered to the server the producer will either block or throw an exception based
on the preference specified by block.on.buffer.full. |
Yes |
| This setting should correspond roughly to the total memory the producer will use, but is not a hard bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests. | ||
compression.type |
The compression type for all data generated by the producer. The
default is none (i.e. no compression). Valid values are none,
gzip, snappy, or lz4.
Compression is of full batches of data, so the efficacy of batching will also impact
the compression ratio (more batching means better compression). |
Ignored. Compression is configured per stream. |
retries |
Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries will potentially change the ordering of records because if two records are sent to a single partition, and the first fails and is retried but the second succeeds, then the second record may appear first. | Ignored. HPE Ezmeral Data Fabric Streams always does automatic retries on transient errors. |
ssl.key.password |
The password of the private key in the key store file. This is optional for client. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
ssl.keystore.location |
The location of the key store file. This is optional for client and can be used for two-way authentication for client. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
ssl.keystore.password |
The store password for the key store file.This is optional for client
and only needed if ssl.keystore.location is configured. |
Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
ssl.truststore.location |
The location of the trust store file. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
ssl.truststore.password |
The password for the trust store file. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
batch.size |
The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. | Ignored. HPE Ezmeral Data Fabric always batches records for optimal performance. |
| No attempt will be made to batch records larger than this size. | ||
| Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent. | ||
| A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records. | ||
client.id |
An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging. | Yes |
connections.max.idle.ms |
Close idle connections after the number of milliseconds specified by this config. | Ignored. |
linger.ms |
The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay—that is, rather than immediately sending out a record the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle's algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get batch.size worth of records for a partition it will be sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will 'linger' for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting linger.ms=5, for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absense of load. | Ignored. |
max.block.ms |
The configuration controls how long {@link KafkaProducer#send()} and {@link KafkaProducer#partitionsFor} will block. These methods can be blocked for multiple reasons. For e.g: buffer full, metadata unavailable.This configuration imposes maximum limit on the total time spent in fetching metadata, serialization of key and value, partitioning and allocation of buffer memory when doing a send(). In case of partitionsFor(), this configuration imposes a maximum time threshold on waiting for metadata | Ignored. HPE Ezmeral Data Fabric Streams has a similar parameter:
streams.rpc.timeout.ms |
max.request.size |
The maximum size of a request. This is also effectively a cap on the maximum record size. Note that the server has its own cap on record size which may be different from this. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. | Ignored. |
partitioner.class |
Partitioner class that implements the Partitioner interface. | Use the Kafka StreamsPartitioner interface. |
receive.buffer.bytes |
The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. | Ignored. |
request.timeout.ms |
The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. | Ignored. HPE Ezmeral Data Fabric Streams has a similar parameter:
streams.rpc.timeout.ms |
sasl.kerberos.service.name |
The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
security.protocol |
Protocol used to communicate with brokers. Currently only PLAINTEXT and SSL are supported. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
send.buffer.bytes |
The size of the TCP send buffer (SO_SNDBUF) to use when sending data. | Ignored. |
ssl.enabled.protocols |
The list of protocols enabled for SSL connections. TLSv1.2, TLSv1.1 and TLSv1 are enabled by default. | Ignored. |
ssl.keystore.type |
The file format of the key store file. This is optional for client. Default value is JKS | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
ssl.protocol |
The SSL protocol used to generate the SSLContext. Default setting is TLS, which is fine for most cases. Allowed values in recent JVMs are TLS, TLSv1.1 and TLSv1.2. SSL, SSLv2 and SSLv3 may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
ssl.provider |
The name of the security provider used for SSL connections. Default value is the default security provider of the JVM. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
ssl.truststore.type |
The file format of the trust store file. Default value is JKS. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
timeout.ms |
The configuration controls the maximum amount of time the server will wait for acknowledgments from followers to meet the acknowledgment requirements the producer has specified with the acks configuration. If the requested number of acknowledgments are not met when the timeout elapses an error will be returned. This timeout is measured on the server side and does not include the network latency of the request. | Ignored. HPE Ezmeral Data Fabric Streams has a similar parameter:
streams.rpc.timeout.ms |
block.on.buffer.full |
When our memory buffer is exhausted we must either stop accepting new records (block) or throw errors. By default this setting is true and we block, however in some scenarios blocking is not desirable and it is better to immediately give an error. Setting this to false will accomplish that: the producer will throw a BufferExhaustedException if a recrord is sent and the buffer space is full. | Ignored. |
max.in.flight.requests.per.connection |
The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled). | Ignored. HPE Ezmeral Data Fabric Streams has a similar parameter:
streams.parallel.flushers.per.partition |
metadata.fetch.timeout.ms |
The first time data is sent to a topic we must fetch metadata about that topic to know which servers host the topic's partitions. This fetch to succeed before throwing an exception back to the client. | Ignored. |
metadata.max.age.ms |
The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions. | Yes |
metric.reporters |
A list of classes to use as metrics reporters. Implementing the MetricReporter interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics. | No |
metrics.num.samples |
The number of samples maintained to compute metrics. | No. |
metrics.sample.window.ms |
The number of samples maintained to compute metrics. | No. |
reconnect.backoff.ms |
The amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all requests sent by the consumer to the broker. | Ignored. HPE Ezmeral Data Fabric Streams has a similar parameter:
streams.rpc.timeout.ms |
retry.backoff.ms |
The amount of time to wait before attempting to retry a failed fetch request to a given topic partition. This avoids repeated fetching-and-failing in a tight loop. | Ignored. HPE Ezmeral Data Fabric Streams has a similar parameter:
streams.rpc.timeout.ms |
sasl.kerberos.kinit.cmd |
Kerberos kinit command path. Default is /usr/bin/kinit | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
sasl.kerberos.min.time.before.relogin |
Login thread sleep time between refresh attempts. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
sasl.kerberos.ticket.renew.jitter |
Percentage of random jitter added to the renewal time. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
sasl.kerberos.ticket.renew.window.factor |
Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
ssl.cipher.suites |
A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol.By default all the available cipher suites are supported. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
ssl.endpoint.identification.algorithm |
The endpoint identification algorithm to validate server hostname using server certificate. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
ssl.keymanager.algorithm |
The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
ssl.trustmanager.algorithm |
The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| Name | Description | Supported for consumers in HPE Ezmeral Data Fabric Streams? |
|---|---|---|
| bootstrap.servers | A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down). | No. Cluster details are discovered from the file
mapr-clusters.conf. |
| key.deserializer | Deserializer class for key that implements the Deserializer interface. | Yes |
| value.deserializer | Deserializer class for value that implements the Deserializer interface. | Yes |
| fetch.min.bytes | The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency. | Yes |
| fetch.max.bytes | The maximum amount of data the server should return for a fetch request. If the
first record batch in the first non-empty partition of the fetch is larger than this
configuration, the record batch is still returned to ensure that the consumer can
make progress. NOTE This is new as of MapR 6.0.1. |
Yes, as of MapR 6.0.1. |
| group.id | A unique string that identifies the consumer group this consumer belongs to. This property is required if the consumer uses either the group management functionality by using subscribe(topic) or the Kafka-based offset management strategy. | Yes |
| heartbeat.interval.ms | The expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances. | No |
| max.poll.records | Places an upper bound on the number of records returned from each call.
NOTE This parameter is new as of MapR 6.0.1. |
Yes, as of MapR 6.0.1. |
| max.partition.fetch.bytes | The maximum amount of data per-partition the server will return. The
maximum total memory used for a request will be #partitions *
max.partition.fetch.bytes. This size must be at least as large as the maximum
message size the server allows or else it is possible for the producer to send
messages larger than the consumer can fetch. If that happens, the consumer can get
stuck trying to fetch a large message on a certain partition. If the first record
batch in the first non-empty partition of the fetch is larger than this
configuration, the record batch is still returned to ensure that the consumer can
make progress. NOTE This is a behavior change as of MapR
6.0.1. |
Yes |
| session.timeout.ms | The timeout used to detect failures when using Kafka's group management facilities. | Ignored |
| ssl.key.password | The password of the private key in the key store file. This is optional for client. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| ssl.keystore.location | The location of the key store file. This is optional for client and can be used for two-way authentication for client. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| ssl.keystore.password | The store password for the key store file.This is optional for client and only needed if ssl.keystore.location is configured. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| ssl.truststore.location | The location of the trust store file. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| ssl.truststore.password | The password for the trust store file. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| auto.offset.reset |
What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):
|
Yes |
| connections.max.idle.ms | Close idle connections after the number of milliseconds specified by this config. | Ignored |
| enable.auto.commit | If true the consumer's offset will be periodically committed in the background. | Yes |
| partition.assignment.strategy | The class name of the partition assignment strategy that the client will use to distribute partition ownership amongst consumer instances when group management is used | Ignored. HPE Ezmeral Data Fabric Streams distributes partitions equally among the consumers in a group. |
| receive.buffer.bytes | The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. | Ignored. |
| request.timeout.ms | The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. | Ignored. HPE Ezmeral Data Fabric Streams has a similar parameter:
streams.rpc.timeout.ms |
| sasl.kerberos.service.name | The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| security.protocol | Protocol used to communicate with brokers. Currently only PLAINTEXT and SSL are supported. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| send.buffer.bytes | The size of the TCP send buffer (SO_SNDBUF) to use when sending data. | Ignored. |
| ssl.enabled.protocols | The list of protocols enabled for SSL connections. TLSv1.2, TLSv1.1 and TLSv1 are enabled by default. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| ssl.keystore.type | The file format of the key store file. This is optional for client. Default value is JKS | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| ssl.protocol | The SSL protocol used to generate the SSLContext. Default setting is TLS, which is fine for most cases. Allowed values in recent JVMs are TLS, TLSv1.1 and TLSv1.2. SSL, SSLv2 and SSLv3 may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| ssl.provider | The name of the security provider used for SSL connections. Default value is the default security provider of the JVM. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| ssl.truststore.type | The file format of the trust store file. Default value is JKS. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| auto.commit.interval.ms | The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if enable.auto.commit is set to true. | Yes |
| check.crcs | Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance. |
Ignored. HPE Ezmeral Data Fabric Streams always does end-to-end crc computation and verification. |
| client.id | An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging. | Yes |
| fetch.max.wait.ms | The maximum amount of time the server will block before answering the fetch request if there isn't sufficient data to immediately satisfy the requirement given by fetch.min.bytes. | No |
| metadata.max.age.ms | The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions. | Yes |
| metric.reporters | A list of classes to use as metrics reporters. Implementing the MetricReporter interface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics. | No |
| metrics.num.samples | The number of samples maintained to compute metrics. | No |
| metrics.sample.window.ms | The number of samples maintained to compute metrics. | No |
| reconnect.backoff.ms | The amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all requests sent by the consumer to the broker. | Ignored. HPE Ezmeral Data Fabric Streams has a similar parameter:
streams.rpc.timeout.ms |
| retry.backoff.ms | The amount of time to wait before attempting to retry a failed fetch request to a given topic partition. This avoids repeated fetching-and-failing in a tight loop. | Ignored. HPE Ezmeral Data Fabric Streams has a similar parameter:
streams.rpc.timeout.ms |
| sasl.kerberos.kinit.cmd | Kerberos kinit command path. Default is /usr/bin/kinit | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| sasl.kerberos.min.time.before.relogin | Login thread sleep time between refresh attempts. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| sasl.kerberos.ticket.renew.jitter | Percentage of random jitter added to the renewal time. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| sasl.kerberos.ticket.renew.window.factor | Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| ssl.cipher.suites | A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol.By default all the available cipher suites are supported. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| ssl.endpoint.identification.algorithm | The endpoint identification algorithm to validate server hostname using server certificate. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| ssl.keymanager.algorithm | The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
| ssl.trustmanager.algorithm | The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine. | Ignored. Authentication and authorization are handled through HPE Ezmeral Data Fabric security. See Security for more information. |
The following HPE Ezmeral Data Fabric Streams parameters are for the Admin API.
| HPE Ezmeral Data Fabric Streams Parameter | Description | Kafka Parameter Replaced |
|---|---|---|
streams.rpc.timeout.ms |
Specifies the length of time in milliseconds to wait for a response from
the HPE Ezmeral Data Fabric Streams server if soft mount is configured (fs.mapr.hardmount is set to
false). Default: 120000 Minimum: 30000 NOTE Applicable as of MapR 6.0.1, is
used instead of fs.mapr.rpc.timeout
|
|
The following HPE Ezmeral Data Fabric Streams parameter are for the Producer API:
| HPE Ezmeral Data Fabric Streams Parameter | Description | Kafka Parameter Replaced |
|---|---|---|
streams.buffer.max.time.ms |
Messages are buffered in the producer for at most the specified time. A thread will flush all the messages that have been buffered for more than the time specified. Default: 3 * 1000 msec |
linger.ms |
streams.parallel.flushers.per.partition |
If enabled, producer may have multiple parallel send requests to the server for each topic partition. If this setting is set to true, it is possible for messages to be sent out of order. Default: true |
max.in.flight.requests.per.connection |
streams.partitioner.class |
The class that implements the StreamsPartitioner
interface. This interface lets you write custom algorithms for determining which
topic and partition to use for messages that match specific criteria. Use this
configuration parameter only for producers that are written in Java. |
Not applicable. |
streams.producer.default.stream |
Specifies the stream that the producer will use by default if the producer does not provide the name of a stream when specifying a topic to write to. For example, the producer can specify the name of a stream together with the name of a topic to write to, like this: /<stream>/<topic> However, if the stream is not specified, the value of this configuration parameter is assumed to be the stream in which the topic is located. If the producer specifies the name of a topic without also providing the path and name of the stream, and there is no value for this configuration parameter, HPE Ezmeral Data Fabric Streams assumes that the topic specified is in Apache Kafka and does nothing. |
Not applicable. |
streams.rpc.timeout.ms |
Specifies the length of time in milliseconds to wait for a response from the
HPE Ezmeral Data Fabric Streams server if soft mount is configured (fs.mapr.hardmount is set to false).
Default: 30000 Minimum: 30000 If the time expires while a producer waits for a
response from the HPE Ezmeral Data Fabric Streams server, and the producer used the
KafkaProducer.send(ProducerRecord<K,V> record, Callback
callback) method, the callback is invoked with the error
EAGAIN, which means "Resource temporarily unavailable."
NOTE Applicable as of MapR 6.0.1, is used instead of
fs.mapr.rpc.timeout
|
|
The following HPE Ezmeral Data Fabric Streams parameters are for the Consumer API:
| HPE Ezmeral Data Fabric Streams Parameter | Description | Kafka Parameter Replaced |
|---|---|---|
streams.consumer.default.stream |
Specifies the path and name of the stream that the consumer subscribes to if, when subscribing to a topic, the consumer does not specify a stream. This default value is also used for the KafkaConsumer.listTopics() method. |
Not applicable. |
streams.rpc.timeout.ms |
Specifies the length of time in milliseconds to wait for a response from the
HPE Ezmeral Data Fabric Streams server if a soft mount is configured (fs.mapr.hardmount is set to false).
Default: 305000 Minimum: 300000 NOTE Applicable as of MapR 6.0.1, is used instead
of fs.mapr.rpc.timeout
|
|