com.rpl.rama.integration

Interface ExternalDepot

All Superinterfaces:

java.lang.AutoCloseable, java.io.Closeable, RamaSerializable, java.io.Serializable, TaskGlobalObject

public interface ExternalDepot
extends TaskGlobalObject

This interface is used to integrate external queue systems as depots inside Rama. This allows Rama topologies to consume those external queue systems directly. This interface exposes everything necessary to support the complete API for both stream and microbatch topologies. An ExternalDepot can be dynamically upscaled or downscaled, and Rama will automatically detect this to start consuming from new partitions or stop consuming from obsolete partitions. Rama references individual partitions of an external queue via a "partition index". It's critical that a partition index always identity the same partition of data (including after upscaling or downscaling) since Rama stores progress state for stream and microbatch topologies by partition index. Individual records on a partition are identified by an "offset". Rama expects offsets for records to monotonically increase by one for each record added. All methods on this interface return asynchronously via CompletableFuture. These methods cannot block the thread on which they're called, so if they must perform blocking operations, they should do so on a separate thread. This thread can be created and managed in the TaskGlobalObject.prepareForTask(int, com.rpl.rama.integration.TaskGlobalContext) method.

See Also:

Integration documentation

Method Summary

Modifier and Type	Method and Description
java.util.concurrent.CompletableFuture<java.lang.Long>	endOffset(int partitionIndex) Asynchronously returns the end offset of the given partition index (exclusive).
java.util.concurrent.CompletableFuture<java.util.List>	fetchFrom(int partitionIndex, long startOffset) Asynchronously returns a chunk of records starting at the given offset.
java.util.concurrent.CompletableFuture<java.util.List>	fetchFrom(int partitionIndex, long startOffset, long endOffset) Asynchronously returns the sequence of records between the given range of offsets.
java.util.concurrent.CompletableFuture<java.lang.Integer>	getNumPartitions() Asynchronously returns the number of partitions in the external queue.
java.util.concurrent.CompletableFuture<java.lang.Long>	offsetAfterTimestampMillis(int partitionIndex, long millis) Asynchronously returns the first offset of the given partition index after the given timestamp.
java.util.concurrent.CompletableFuture<java.lang.Long>	startOffset(int partitionIndex) Asynchronously returns the start offset of the given partition index (inclusive).

Methods inherited from interface com.rpl.rama.integration.TaskGlobalObject

gainedLeadership, prepareForTask

Methods inherited from interface java.io.Closeable

close

Method Detail

getNumPartitions

java.util.concurrent.CompletableFuture<java.lang.Integer> getNumPartitions()

Asynchronously returns the number of partitions in the external queue.

startOffset

java.util.concurrent.CompletableFuture<java.lang.Long> startOffset(int partitionIndex)

Asynchronously returns the start offset of the given partition index (inclusive). This method does not need to be implemented if consuming topologies do not use startFromBeginning() in their source options.

endOffset

java.util.concurrent.CompletableFuture<java.lang.Long> endOffset(int partitionIndex)

Asynchronously returns the end offset of the given partition index (exclusive).

offsetAfterTimestampMillis

java.util.concurrent.CompletableFuture<java.lang.Long> offsetAfterTimestampMillis(int partitionIndex,
                                                                                  long millis)

Asynchronously returns the first offset of the given partition index after the given timestamp. If no such offset exists, this method should asynchronously return null. This method does not need to be implemented if consuming topologies do not use startFromOffsetAfterTimestamp(long millis) in their source options.

fetchFrom

java.util.concurrent.CompletableFuture<java.util.List> fetchFrom(int partitionIndex,
                                                                 long startOffset,
                                                                 long endOffset)

Asynchronously returns the sequence of records between the given range of offsets.

Parameters:

partitionIndex - Partition to query

startOffset - Start offset of range (inclusive)

endOffset - End offset of range (exclusive)

fetchFrom

java.util.concurrent.CompletableFuture<java.util.List> fetchFrom(int partitionIndex,
                                                                 long startOffset)

Asynchronously returns a chunk of records starting at the given offset. The implementation should decide what is a reasonable amount of data to fetch in a single call. This method is used by stream topologies to poll for new data or to catch up when far behind.

Parameters:

partitionIndex - Partition to query

startOffset - Start offset of range (inclusive)