HTTP Feeds

HTTP Feeds

Vanic April 4, 2022

Asynchronous occasion streaming and data replication with easy HTTP APIs.

HTTP feeds is a minimal specification for polling occasions over HTTP:

  • An HTTP feed presents a HTTP GET endpoint
  • that returns a chronological sequence (!) of occasions
  • serialized in CloudEvents occasion layout
  • in batched responses the exhaust of the media kind utility/cloudevents-batch+json
  • and respects the lastEventId query parameter to scroll through additional devices
  • to present a enhance to limitless polling for staunch-time feed subscriptions.

HTTP feeds will most definitely be outmoded to decouple programs asynchronously without message brokers, much like Kafka or RabbitMQ.

Instance

GET /inventory HTTP/1.1
Host:  https://instance.http-feeds.org

HTTP/1.1 200 OK
Train-Form:  utility/cloudevents-batch+json

[{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "1c6b8c6e-d8d0-4a91-b51c-1f56bd04c758",
  "time" : "2021-01-01T00:00:01Z",
  "subject" : "9521234567899",
  "data" : {
    "sku": "9521234567899",
    "updated": "2022-01-01T00:00:01Z",
    "quantity": 5
  }
},{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "292042fb-ab04-4653-af90-19a24032bffe",
  "time" : "2021-12-01T00:00:15Z",
  "subject" : "9521234512349",
  "data" : {
    "sku": "9521234512349",
    "updated": "2022-01-01T00:00:12Z",
    "quantity": 0
  }
},{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "fa3e2a22-398c-4d02-ad08-9415e43178e6",
  "time" : "2021-01-01T00:00:22Z",
  "subject" : "9521234567899",
  "data" : {
    "sku": "9521234567899",
    "updated": "2022-01-01T00:00:21Z",
    "quantity": 4
  }
}]

Shopper calls all over again with the final processed occasion identification.

GET /inventory?lastEventId=fa3e2a22-398c-4d02-advert08-9415e43178e6 HTTP/1.1
Host:  https://instance.http-feeds.org

HTTP/1.1 200 OK
Train-Form:  utility/cloudevents-batch+json

[]

An empty array signals the finish of the feed.

Polling

A consumer can proceed polling in a first-rate loop to subscribe to a feed.

Easy Polling

A consumer calls the endpoint with the final identified occasion identification as lastEventId query parameter in an never-ending loop.
If the response is an empty array, the patron reached the finish of the circulate and waits a while to manufacture any other call to salvage occasions that occured in the duration in-between.

Pseudocode: 

endpoint = "https://instance.http-feeds.org/inventory"
lastEventId = null

while factual: 
  strive: 
    response = GET endpoint + "?lastEventId=" + lastEventId 
    for occasion in response: 
      process occasion
      lastEventId = occasion.identification
    if response is empty: 
      wait N seconds 
  as a change of: 
    wait N seconds

Shopper must persist the identification of the final processed occasion as lastEventId for additional fetches.

The patron’s occasion processing must be idempotent (at-least-once provide semantic).
The identification would possibly per chance furthermore be outmoded for idempotency checks.

Long Polling

The server would possibly per chance furthermore furthermore give a enhance to prolonged polling for decrease latency.
The patron adds a timeout query parameter to specify the max duration of milliseconds to support for an response.

Shopper Pseudocode: 

endpoint = "https://instance.http-feeds.org/inventory"
lastEventId = null
timeout = 5000 // 5000 milliseconds is a factual timeout duration for prolonged polling

while factual: 
  strive: 
    response = GET endpoint + "?lastEventId=" + lastEventId + "&timeout=" + timeout
    for occasion in response: 
      process occasion
      lastEventId = occasion.identification
    // no consumer wait step within the loop
  as a change of: 
    // Extend the next quiz handiest in case of a server error
    wait N seconds

If there are no more moderen occasions accessible, the server keeps the connection open except new occasions arrive or the defined duration timed out.
The server then sends the response (with the brand new occasions or an empty array) and the patron can at once produce any other call.

The latency will most definitely be improved, because the server can react to new occasions successfully by imposing an internal occasion notification, change data rob triggers, or performing a excessive-frequency polling to the database.

The rate of prolonged polling is that the server wants to tackle more open connections concurrently.
This would furthermore turn out to be a anxiousness with bigger than 10K connections.

Tournament ID

The occasion.identification is outmoded as lastEventId to scroll through additional occasions.
This means that occasions have to be strongly ordered to retrieve subsequent occasions.

Events would possibly per chance furthermore furthermore salvage deleted from the feed, e.g. through Compaction and Deletion.
The server must unexcited respect the fashioned advise and ship handiest more moderen occasions, even when an occasion with the lastEventId used to be deleted.

One approach to enact right here’s to exhaust an time-ordered UUIDv6 as occasion ID (stare IETF draft and Java-Library).
This a viable risk, especially if handiest one server appends new occasions to the feed, nonetheless will most definitely be a anxiousness with more than one servers when the clocks are no longer in-sync.

An different is to exhaust a database sequence that is outmoded as allotment of the occasion.identification and interpreted when querying the database for the next batch. Instance: The occasion.identification 0000001000001::5f8de8ff-30d8-4fab-8f5a-c32f326d6f26 incorporates a database sequence 0000001000001 and a random UUID.

Tournament Feeds

HTTP feeds will most definitely be outmoded to produce an API for publishing immutable domain occasions to other programs.

It can probably be very frequent that one http feed endpoint entails assorted occasion kinds that belong to the an analogous bounded context.

Aggregate Feeds

HTTP feeds will most definitely be outmoded to produce an API for data collections of mutable objects (aka aggregates, master data) to other programs.

An aggregate is diagnosed through its self-discipline.
An aggregate feed must like every aggregate as a minimum once.
Each created aggregate and each update ends in an appended feed entry with the paunchy unique advise.

Feed customers can subscribe an aggregate feed to produce finish to staunch-time data synchronization to fabricate local read fashions and to trigger actions when new or up thus a long way data is purchased.
A feed consumer has a fixed advise when reaching the finish of the feed.

The server have to put into effect Compaction and would possibly per chance furthermore put into effect Deletion in step with industry requirements.

Compaction

Each aggregate update ends in an extra entry in the feed.
It’s a long way factual note to support the feed shrimp to permit a immediate synchronization of unique customers.
When feed devices consist of the paunchy unique advise of the resource, older feed devices for the an analogous aggregate would possibly per chance furthermore be out of date.

Clients that read the feed from the origin will subsequently occur an inconsistent advise for a immediate time.
To mitigate, entries would possibly per chance furthermore be deleted from the feed when any other entry used to be appended to the feed with the an analogous self-discipline.

Instance:
There would possibly per chance be an update for self-discipline 9521234567899.

HTTP/1.1 200 OK
Train-Form:  utility/cloudevents-batch+json

[{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "1c6b8c6e-d8d0-4a91-b51c-1f56bd04c758",
  "time" : "2021-01-01T00:00:01Z",
  "subject" : "9521234567899",
  "data" : {
    "sku": "9521234567899",
    "updated": "2022-01-01T00:00:01Z",
    "quantity": 5
  }
},{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "292042fb-ab04-4653-af90-19a24032bffe",
  "time" : "2021-12-01T00:00:15Z",
  "subject" : "9521234512349",
  "data" : {
    "sku": "9521234512349",
    "updated": "2022-01-01T00:00:12Z",
    "quantity": 0
  }
},{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "fa3e2a22-398c-4d02-ad08-9415e43178e6",
  "time" : "2021-01-01T00:00:22Z",
  "subject" : "9521234567899",
  "data" : {
    "sku": "9521234567899",
    "updated": "2022-01-01T00:00:21Z",
    "quantity": 4
  }
}]

After a compaction speed, the first entry is gone: 

HTTP/1.1 200 OK
Train-Form:  utility/cloudevents-batch+json

[{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "292042fb-ab04-4653-af90-19a24032bffe",
  "time" : "2021-12-01T00:00:15Z",
  "subject" : "9521234512349",
  "data" : {
    "sku": "9521234512349",
    "updated": "2022-01-01T00:00:12Z",
    "quantity": 0
  }
},{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "fa3e2a22-398c-4d02-ad08-9415e43178e6",
  "time" : "2021-01-01T00:00:22Z",
  "subject" : "9521234567899",
  "data" : {
    "sku": "9521234567899",
    "updated": "2022-01-01T00:00:21Z",
    "quantity": 4
  }
}]

Deletion

Some aggregates have to be deleted, e. g. by regulatory requirements.

Aggregate feeds exhaust a plot self-discipline with cost DELETE to signal the deletion of an self-discipline to customers that built a neighborhood read mannequin before.

When aggregate used to be deleted, the server must append a DELETE entry with the self-discipline to delete and no data yell.

{
  "specversion" :  "1.0",
  "kind" :  "org.http-feeds.instance.inventory",
  "source" :  "https://instance.http-feeds.org/inventory",
  "identification" :  "06b13630-e4c3-4d85-a669-ce66fc4daa75",
  "time" :  "2021-12-31T00: 00: 01Z",
  "self-discipline" :  "9521234567899",
  "plot":  "DELETE"
}

Clients must delete this aggregate or in any other case tackle the removal.

The server have to open a compaction speed afterwards to delete old entries for the an analogous aggregate.

Data Model

An HTTP feed endpoint must give a enhance to these query parameters:

Inquire of Parameter Form Distinguished Description
lastEventId String Now not most critical The final occasion identification that used to be processed. Aged to scroll through additional occasions. Will seemingly be missing or null to open from the origin of the feed.
timeout Number Now not most critical A timeout is determined, when prolonged-polling ought to be outmoded and is supported by the server. Max waiting time in milliseconds for prolonged-polling, after which the server must ship a response. A frequent cost is 5000.

The response physique contains an array of occasions that vary to the CloudEvents Specification in the utility/cloudevents-batch+json layout.

Discipline Form Tournament Feed Aggregate Feed Description
specversion String Distinguished Distinguished The currently supported CloudEvents specification model.
identification String Distinguished Distinguished A irregular cost (much like a UUID) for this occasion. It will seemingly be outmoded to put into effect deduplication/idempotency going through in downstream programs. It’s a long way outmoded because the lastEventId in subsequent calls.
kind String Distinguished Distinguished The diversity of the occasion. Will seemingly be outmoded to specify and deserialize the payload. A feed would possibly per chance furthermore like assorted occasion kinds. It SHOULD be prefixed with a reverse-DNS name.
source String Distinguished Distinguished The source design that created the occasion. Ought to be a URI of the design.
time String Distinguished Distinguished The occasion addition timestamp. ISO 8601 UTC date and time layout.
self-discipline String n/a Distinguished Key to call the industry object. It doesn’t have to be irregular internal the feed. This would signify a industry key much like an state quantity or sku. Aged for compaction and deletion, if implemented.
plot String n/a Now not most critical The HTTP identical plot kind that the feed item performs on the self-discipline. PUT means that the self-discipline used to be created or up thus a long way. DELETE means that the self-discipline used to be deleted. Defaults to PUT.
datacontenttype String Now not most critical Now not most critical Defaults to utility/json.
data Object Distinguished Now not most critical The payload of the object. Defaults to JSON. Will seemingly be missing, e.g. when the vogue used to be DELETE.

Additional metadata would possibly per chance furthermore be added, e.g. for traceability.

Authentication

HTTP feeds would possibly per chance furthermore be friendly with HTTP authentication.

The most frequent authentication schemes are Total and Bearer.

The server would possibly per chance furthermore filter feed devices in step with the fundamental.
When filtering is utilized, caching would possibly per chance furthermore be unfeasible.

Caching

Servers would possibly per chance furthermore region appropriate response headers, much like Cache-Alter: public, max-age=31536000, when a batch is paunchy and would possibly per chance no longer be modified anymore.

Libraries and Examples

About

This situation is maintained by Jochen Christ.
HTTP feeds has stepped forward from relaxation-feeds.
Contributions are extremely liked, e.g. libraries or examples in assorted languages or frameworks again so much.

Realized an error or something is missing?
Please elevate a anxiousness or produce a pull quiz.

This specification is revealed below CC BY 4.0.

Read More

Categories: Featured / Important
Tags: ,
Vanic
“Simplicity, patience, compassion. These three are your greatest treasures. Simple in actions and thoughts, you return to the source of being. Patient with both friends and enemies, you accord with the way things are. Compassionate toward yourself, you reconcile all beings in the world.” ― Lao Tzu, Tao Te ChingBio: About:

Related Articles

What’s recent in Emacs 28.1?

What’s recent in Emacs 28.1?

By Mickey Petersen It’s that time again: there’s a new major version of Emacs and, with it, a treasure trove of new features and changes.Notable features include the formal inclusion of native compilation, a technique that will greatly speed up your Emacs experience.A critical issue surrounding the use of ligatures also fixed; without it, you…

Knowasiak April 9, 2022
On limitless ethics

On limitless ethics

And for all this, nature is never spent… — Gerard Manley Hopkins Summary: Infinite ethics (i.e., ethical theory that considers infinite worlds) is important – both in theory and in practice. Infinite ethics puts serious pressure on various otherwise-plausible ethical principles (including some that underlie common arguments for “longtermism”). We know, from impossibility results, that…

Vanic March 25, 2022
Xbox 360 Architecture

Xbox 360 Architecture

Supporting imageryModelMotherboardDiagramA quick introductionReleased a year before its main competitor, the Xbox 360 was already claiming technological superiority against the yet-to-be-seen Playstation 3. But while the Xbox 360 might be the flagship of the 7th generation, it will need to fight strongly once Nintendo and Sony take up retail space. This new entry of the…

Ava Chan June 10, 2022
0 Comments

Stack Overflow Developer Survey 2022

Overview Developer Profile Technology Work Community Professional Developers Methodology The questions we ask in our annual survey help us improve the Stack Overflow community and the platform that serves them. The challenge and opportunity for us is to continue expanding and improving our ability to help all developers and to make them feel welcome in…

Vanic June 23, 2022
0 Comments

Report

Harassment or bullying behavior
Contains mature or sensitive content
Contains misleading or false information
Contains abusive or derogatory content
Contains spam, fake content or potential malware

Block Member?

Please confirm you want to block this member.

You will no longer be able to:

  • See blocked member's posts
  • Mention this member in posts
  • Invite this member to groups
  • Message this member
  • Add this member as a connection

Please note: This action will also remove this member from your connections and send a report to the site admin. Please allow a few minutes for this process to complete.

Report

You have already reported this .