Streams API
Service Information:
Version: 1.0
Availability: Development
Release Date: (planned) August 2018
Framework: AITS RESTful Data API
Description: Provides a RESTful interface to the Event Streams service. For additional information about the Event Stream service, please see the Event Streams service guide.
Source System: Various
Data Format: JSON
Operations:
poll
GET /{resource}/stream/poll/{senderAppId}/{topic}
Retrieves all events that occurred since the last committed offset (last call to the service) for a given senderAppId and topic. This service transparently manages offsets for, and consumes from all partitions that exist for the topic being polled.
NOTE: The Kafka platform will persist a consumer's offset values for up to seven days of inactivity since the last "poll" execution. After that time, the consumer will automatically seek to the end of the topic with the next "poll" execution. The Kafka platform also has configured a max.poll.records (max number of records retrieved with each "poll" execution) of 500. This can be exceeded occasionally as the platform attempts to budget the max over topics having numerous partitions.
| Parameter | Description | Type | Required |
|---|---|---|---|
| resource | AITS Web Services resource (e.g. PersonWS) associated with the topic of interest. | String | Yes |
| senderAppId | Authorized sender application ID. | String | Yes |
| topic | Event stream topic name | String | Yes |
Sample Request
GET /personWS/stream/poll/mySenderAppId/TestEvent HTTP/1.1 Host: webservices-dev.admin.uillinois.edu
Sample Response
[ { "object": "TestEvent", "topic": "avro-testEvent", "partition": 1, "offset": 0, "record": { ... } }, { "object": "TestEvent", "topic": "avro-testEvent", "partition": 1, "offset": 1, "record": { ... } }
Object Detail
| Path | Type | Name | Occurrence | Description |
|---|---|---|---|---|
| object | Element | object | Exactly One | Name of object that transports data on a topic |
| topic | Element | topic | Exactly One | Name of the topic being polled |
| partition | Element | partition | Exactly One | Number of partition from which an event was read |
| offset | Element | offset | Exactly One | Immutable ID of the specific event |
| record | Object-Array | record | Exactly One | Object housing the business data and metadata pretinent to a streams event. For detailed object definitions, please see the Events Streams Service Guide |
read
GET /{resource}/stream/read/{senderAppId}/{topic}/{partition}/{offset}
Provides an array of events as are avaiable for a specified topic and partition, starting from the specified offset.
| Parameter | Description | Type | Required |
|---|---|---|---|
| resource | AITS Web Services resource (e.g. PersonWS) associated with the topic of interest. | String | Yes |
| senderAppId | Authorized sender application ID. | String | Yes |
| topic | Event stream topic name | String | Yes |
| partition | Specific partition from which to read from | Int | Yes |
| offset | Earliest offset to attempt reading from | Int | Yes |
Sample Request
GET /personWS/stream/read/mySenderAppId/TestEvent/1/20 HTTP/1.1 Host: webservices-dev.admin.uillinois.edu
Sample Response
[ { "object": "TestEvent", "topic": "avro-testEvent", "partition": 1, "offset": 0, "record": { ... } }, { "object": "TestEvent", "topic": "avro-testEvent", "partition": 1, "offset": 1, "record": { ... } }
Object Detail
| Path | Type | Name | Occurrence | Description |
|---|---|---|---|---|
| object | Element | object | Exactly One | Name of object that transports data on a topic |
| topic | Element | topic | Exactly One | Name of the topic being polled |
| partition | Element | partition | Exactly One | Number of partition from which an event was read |
| offset | Element | offset | Exactly One | Immutable ID of the specific event |
| record | Object-Array | record | Exactly One | Object housing the business data and metadata pretinent to a streams event. For detailed object definitions, please see the Events Streams Service Guide |
stats
GET /{resource}/stream/stats/{senderAppId}/{topic}
Provides a count of messages currently on a topic, along with a listing of partitions and the current start/end offset for each.
| Parameter | Description | Type | Required |
|---|---|---|---|
| resource | AITS Web Services resource (e.g. PersonWS) associated with the topic of interest. | String | Yes |
| senderAppId | Authorized sender application ID. | String | Yes |
| topic | Event stream topic name | String | Yes |
Sample Request
GET /personWS/stream/stats/mySenderAppId/testEvent HTTP/1.1 Host: webservices-dev.admin.uillinois.edu
Sample Response
{ "object": "testEvent", "topic": "testTopic", "total": 362, "countData": [ { "partition": 0, "beginOffset": 0, "endOffset": 112, "total": 112 }, { "partition": 1, "beginOffset": 0, "endOffset": 134, "total": 134 }, { "partition": 2, "beginOffset": 0, "endOffset": 116, "total": 116 } ] }
Object Detail
| Path | Type | Name | Occurrence | Description |
|---|---|---|---|---|
| object | element | object | Exactly One | Name of object that transports data on a topic |
| topic | element | topic | Exactly One | Name of the topic |
| total | element | total | Exactly One | Total number of events on a topic |
| countData | Object-Array | countData | Exactly One | Child object containing topic detail and counts |
| countData.partition | Object-Array | partition | Exactly One | Partition number |
| countData.beginOffset | Object-Array | beginOffset | Exactly One | Earliest offset present for the associated partition of a topic |
| countData.endOffset | Object-Array | endOffset | Exactly One | Latest offset present for the associated partition of a topic |
| countData.total | Object-Array | total | Exactly One | Total nubmer of events present on the associated partition of a topic |
lag
GET /{resource}/stream/lag/{senderAppId}/{topic}
Provides "stats" data along with lag data detailing a consumer group's place in a topic.
| Parameter | Description | Type | Required |
|---|---|---|---|
| resource | AITS Web Services resource (e.g. PersonWS) associated with the topic of interest. | String | Yes |
| senderAppId | Authorized sender application ID. | String | Yes |
| topic | Event stream topic name | String | Yes |
Sample Request
GET /personWS/stream/lag/mySenderAppId/testEvent HTTP/1.1 Host: webservices-dev.admin.uillinois.edu
Sample Response
{ "object": "Object", "topic": "Topic", "groupName": "ConsumerGroupName", "total": 671, "lag": 0, "lagPercent": 0.0, "countData": [ { "partition": 0, "beginOffset": 0, "endOffset": 640, "total": 640 }, { "partition": 1, "beginOffset": 0, "endOffset": 6, "total": 6 }, { "partition": 2, "beginOffset": 0, "endOffset": 25, "total": 25 } ], "lagData": [ { "partition": 0, "currentOffset": 640, "endOffset": 640, "lag": 0 }, { "partition": 1, "currentOffset": 6, "endOffset": 6, "lag": 0 }, { "partition": 2, "currentOffset": 25, "endOffset": 25, "lag": 0 } ] }
Object Detail
| Path | Type | Name | Occurrence | Description |
|---|---|---|---|---|
| object | element | object | Exactly One | Name of object that transports data on a topic |
| topic | element | topic | Exactly One | Name of the topic |
| total | element | total | Exactly One | Total number of events on a topic |
| lag | element | lag | Exactly One | Total number of queued events for a subscriber on a topic |
| lagPercent | element | lagPercent | Exactly One | Percentage of total events on the topic that have not yet been consumed by a subscriber |
| countData | Object-Array | countData | Exactly One | Child object containing topic detail and counts |
| countData.partition | Object-Array | partition | Exactly One | Partition number |
| countData.beginOffset | Object-Array | beginOffset | Exactly One | Earliest offset present for the associated partition of a topic |
| countData.endOffset | Object-Array | endOffset | Exactly One | Latest offset present for the associated partition of a topic |
| countData.total | Object-Array | total | Exactly One | Total number of events present on the associated partition of a topic |
| lagData | Object-Array | lagData | Exactly One | Child object containing topic detail and counts for the subscribed consumer |
| lagData.partition | Object-Array | partition | Exactly One | Partition number |
| lagData.beginOffset | Object-Array | currentOffset | Exactly One | Last consumed offset for the associated consumer group on a given partition of a topic |
| lagData.endOffset | Object-Array | endOffset | Exactly One | Latest offset present for the associated partition of a topic |
| lagData.lag | Object-Array | total | Exactly One | Total number of queued events present on the associated partition of a topic waiting to be retrieved by the subscribed consumer |