Documentation › Subscribers

Introduction

Superfeedr allows you to subscribe to content on the web, and receive push notifications in real-time when new content is published. It also allows you to retrieve past content, and we’ve also got a range of normalization options that make for easier consumption.

What can you subscribe to

You can use the Superfeedr feed API to subscribe to anything that has a URL. The key caveat to this is that the content needs to be publicly accessible by our servers – the content can’t be in a private network or behind a firewall.

While URLs can include an authentication element, but Superfeedr will not treat these URLs with any specific security concern. For this reason, we strongly discourage including URLs with authentication mechanisms.

XML based feeds

Our historical use case is to allow you to subscribe to RSS or Atom feeds and then have that content pushed to you in real-time. These are also normalized, to make consumption easier for you (see the schema section). Superfeedr supports RSS, Atom and RDF, as well as the most popular namespaces.

An update is detected when one or more item is added to the feed you are subscribed to. The real-time push notification includes the new item(s) and some key attributes from the feed, like the title.

Each entry in a feed is mapped with a unique identifier - the id element in our schema. We will send you a notification for every new unique identifier we detect, whatever the rest of the entry is.We don’t rely on the title, links or even the signature of the entry’s content to detect updates.

This behaviour is compliant with both RSS and Atom specifications. If we’re not able to find a unique ID, Superfeedr generates one, using complex feed-specific rules. (We indicate when the <id> has been generated by Superfeedr using the generated_ids field in the status).

Updates

If an entry is updated, they are not propagated by default. This is because we want to avoid creating numerous false positives.

There is, however, one exception: if a new entry contains a valid, updated element, and the update is very recent (within three minutes), you will receive a notification. This means that we will usually propagate updates for feeds if we receive a ping from the publisher.

Errors

If a feed goes into an error state (say, if it’s putting up HTTP error codes, or we’ve been unable to parse content successfully), you’ll also receive a notification.

This will allow you to monitor the situation, and decide whether you want to maintain the subscription to that feed. These notifications will only include the status part of the schema.

JSON feeds

It is obviously possible to subscribe to any kind of JSON document. In order to identify new content, we compute a hash signature of the whole document. If that signature changes between two fetches, Superfeedr propagates the change by sending the full JSON document to your endpoint.

You will also receive a notification if a resource goes into an HTTP error state. These notifications will only include the status part of the schema.

HTML pages and fragments

Superfeedr also allows you to subscribe to HTML fragments within an HTML page. A fragment is a portion of an HTML document, defined by a CSS path. You can subscribe to DOM elements inside a page as long as there is a CSS path that leads to them. You will append the URL-escaped CSS path to the document URL, using the fragment part of the URL. If there are multiple elements matching your CSS path, Superfeedr will concatenate each of them.

For example, if you want to subscribe to the .h-entry element of http://blog.superfeedr.com, you will need to subscribe to http://blog.superfeedr.com#.h-entry.

JSON fragments

Much like subscribing to HTML fragments, Superfeedr also allows you to subscribe to parts of a JSON document, using the fragments API. To do this, Superfeedr uses the JSONPath syntax. To build the topic url, you append the URL-escaped JSONPath to the document URL, using the fragment part of the URL.

Example

We have the following document at http://for.sale/inventory.json:

{
  "store": {
    "book": [
      { "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      { "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      },
      { "category": "fiction",
        "author": "Herman Melville",
        "title": "Moby Dick",
        "isbn": "0-553-21311-3",
        "price": 8.99
      },
      { "category": "fiction",
        "author": "J. R. R. Tolkien",
        "title": "The Lord of the Rings",
        "isbn": "0-395-19395-8",
        "price": 22.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  }
}

If you wanted to keep an eye on the price of the bicycle, you would subscribe to: http://for.sale/inventory.json#%24.store.bicycle.price. Want more examples? Click here.

Other Resources

When you are subscribing to any other type of HTML resource, Superfeedr will compute a signature from the bytes included in the document. If the signature has changed when we next fetch the resource, you will receive the whole document again. Note that timestamps, changes in tracking codes and so on can create false positives here.

Redirects

When fetching resources, Superfeedr will follow up to five redirects. These can be permanent or temporary, and you will be notified of the content at the end of the redirect chain. For this reason, we strongly recommend subscribing to the canonical URL of each resource.

Webhooks

Our API is based on the PubSubHubbub protocol with a couple simplifications. However, you can use your subscriber code with any other hub. You can also use any other library that supports and implements PubSubHubbub.

Our PubSubHubbub endpoint is at https://push.superfeedr.com/. The key difference is that our endpoint uses HTTP Basic Auth to authenticate your PubSubHubbub calls, making all the verification steps of each request optional.

If you want to manually specify an HTTP method different to the one used in the request, we also support the use of the X-HTTP-Method-Override HTTP header.

HTTP Authentication

Authentication using the Webhooks API is performed through HTTP Basic Authentication. Most HTTP libraries will allow for an easy configuration with this. To get started with an authentication, use your Superfeedr login, then pick from one of these options for your password:

• A password token that you can generate.
• Your main Superfeedr password (though we recommend using a token for security purposes).

If your HTTP library does not support HTTP headers, you should submit a base64 encoded string of login:token as a query string parameter named authorization.

Tokens

You can create an unlimited number of tokens, with different combinations of rights associated with them:

• Subscribe
• Unsubscribe
• List
• Retrieve (used for Streaming as well)

The tokens cannot be used to log into the main Superfeedr site.

Tokens can be made public – just keep in mind that any call made using them will be associated with your account. This means that if someone else makes a call with your token, it will be your account that is billed.

If you want your tokens to be private along with the rest of your account details, please remember to use https when sending authentication against our endpoint.

Adding Feeds with PubSubHubbub

Subscriptions at Superfeedr are a unique combination of a resource URL and a callback URL. If you resubscribe with both URLs, we will only keep one. If you use a different callback URL to the resource URL, we will keep both.

Example

curl https://push.superfeedr.com/
  -X POST
  -u demo:demo
  -d'hub.mode=subscribe'
  -d'hub.topic=http://push-pub.appspot.com/feed'
  -d'hub.callback=http://mycallback.tld/ok'

Response by HTTP status code

• 204: the subscription was performed and you should expect notifications
• 202: the subscription has yet to be verified (only if you supplied a hub.verify=async parameter).
• 200: you used the retrieve parameter. The content of the feed is in the body of the response.
• 422: please check the body, as it will include the reason for the subscription failure.
• Other HTTP response codes are outlined in the HTTP spec.

Superfeedr’s Null Device

When subscribing, the callback url can also be https://push.superfeedr.com/dev/null which is our null device. This is particularly useful if you’re not interested in realtime updates to feeds but want to retrieve their statuses from our servers. This way, you’re able to subscribe to tell Superfeedr to not notify you but still let you poll the content of these feeds.

Please note that even though we won’t notify you for the changes in the feed, these notifications will still be accounted for when it comes to billing.

Removing Feeds with PubSubHubbub

Example

curl https://push.superfeedr.com/
  -X POST
  -u demo:demo
  -d'hub.mode=unsubscribe'
  -d'hub.topic=http://push-pub.appspot.com/feed'
  -d'hub.callback=http://mycallback.tld/ok'

Response by HTTP status code

• 204: the unsubscribe was successful
• 202: the unsubscribe has yet to be verified (only if you supplied a hub.verify=async parameter).
• 422: Check the body, as it will include the reason for the unsubscribe failure.
• Other HTTP response codes are outlined in the HTTP spec.

Listing Subscriptions with PubSubHubbub

By default, subscriptions are listed in order of creation. The oldest subscriptions are listed first, while the most recent is listed last.

Response by HTTP status code

• 200: the list of matching subscriptions is in the body in a JSON format (or jsonp if applicable).
• Other HTTP response codes are outlined in the HTTP spec.

Example

curl https://push.superfeedr.com/
  -G
  -u demo:demo
  -d'hub.mode=list'
  -d'page=2'

[{
  "subscription": {
    "format": "atom",
    "endpoint": "http://mycallback.tld/ok",
    "secret": null,
    "feed": {
      "title": "Publisher example",
      "url": "http://push-pub.appspot.com/feed"
    }
  }
}]

Search Queries

Search queries are nested string parameters. We use the following keys:

Retrieving Entries with PubSubHubbub

Example (retrieving by topic)

curl https://push.superfeedr.com/
  -H 'Accept: application/json'
  -G
  -u demo:demo
  -d'hub.mode=retrieve'
  -d'hub.topic=http://push-pub.appspot.com/feed'

Response:

Content-Type: application/json; charset=utf-8
Connection: keep-alive
Status: 200 OK
ETag: "c58f3a54cfa565539672a6f2f3276ddc"
X-Runtime: 275
Content-Length: 3550

{
  "status": {
    "code": 200,
    "feed": "http://push-pub.appspot.com/feed",
    "http": "Fetched (ring) 200 121 and parsed 0/20 entries",
    "lastParse": 1377845845,
    "period": 43200,
    "lastMaintenanceAt": 1377842368,
    "nextFetch": 1377889045,
    "lastFetch": 1377845845
  },
  "title": "Publisher example",
  "items": [
  {
    "title": "mhmmm",
    "published": 1377845723000,
    "id": "http://push-pub.appspot.com/feed/1104006",
    "links": {
      "http://push-pub.appspot.com/entry/1104006": {
        "href": "http://push-pub.appspot.com/entry/1104006",
        "title": "mhmmm",
        "rel": "alternate",
        "mime_type": "text/html",
        "id": "mhmmm"
      }
    },
    "content": "mmhmhjmm",
    "updated": 1377845723000
  },
  {
    "title": "test 2 - bis",
    "published": 1377758355000,
    "id": "http://push-pub.appspot.com/feed/1109002",
    "links": {
      "http://push-pub.appspot.com/entry/1109002": {
        "href": "http://push-pub.appspot.com/entry/1109002",
        "title": "test 2 - bis",
        "rel": "alternate",
        "mime_type": "text/html",
        "id": "test-2-bis"
      }
    },
    "content": "recieved - bis",
    "updated": 1377758355000
  },
  {
    "title": "test",
    "published": 1377756592000,
    "id": "http://push-pub.appspot.com/feed/1111001",
    "links": {
      "http://push-pub.appspot.com/entry/1111001": {
        "href": "http://push-pub.appspot.com/entry/1111001",
        "title": "test",
        "rel": "alternate",
        "mime_type": "text/html",
        "id": "test"
      }
    },
    "content": "not recieving from google hub",
    "updated": 1377756592000
  },
  ...
  {
    "title": "test 2",
    "published": 1377758355000,
    "id": "http://push-pub.appspot.com/feed/1108004",
    "links": {
      "http://push-pub.appspot.com/entry/1108004": {
        "href": "http://push-pub.appspot.com/entry/1108004",
        "title": "test 2",
        "rel": "alternate",
        "mime_type": "text/html",
        "id": "test-2"
      }
    },
    "content": "recieved",
    "updated": 1377758355000
  }
  ]
}

Example (retrieving by matching callback)

curl https://push.superfeedr.com/
  -H 'Accept: application/json'
  -G
  -u demo:demo
  -d'hub.mode=retrieve'
  -d'hub.callback=http://my.domain.com/webhook/1'

Example (retrieving by searching callback)

curl https://push.superfeedr.com/
  -H 'Accept: application/json'
  -G
  -u demo:demo
  -d'hub.mode=retrieve'
  -d'hub.callback[endpoint][hostname]=my.domain.com'

Response by HTTP status code

• 200: the body includes the feed’s status and past items.
• 404: you are not subscribed to the feed.
• 422: please check the body as it will include the reason for the failure.
• Other HTTP response codes are outlined in the HTTP spec.

Streaming RSS

When retrieving past RSS content, it is possible to keep the HTTP connection to Superfeedr alive. This ensures new entries are sent directly to your client. To do this, you need to use the stream.superfeedr.com endpoint:

You will then perform a retrieve call, as outlined above, with the following extra parameter:

Example

# Stream
curl -G 'https://stream.superfeedr.com?wait=stream&hub.mode=retrieve&hub.callback=http://my.webhook.com/path&format=json' \
  -u'demo:demo'

Server Sent Events

Superfeedr also supports Server Sent Events (or EventSource). This W3C specification defines a browser-side Javascript API to receive content from a server in the form of events.

var url = "https://stream.superfeedr.com/";
url += "&hub.mode=retrieve";
url += "&hub.topic=<topic url>";
url += "&authorization=<token>";

var source = new EventSource(url);
source.addEventListener("notification", function(e) {
  var notification = JSON.parse(e.data);
});

PubSubHubbub Notifications

Notifications are POST requests that are sent to the callback you specified for the subscription. ​ The body of the response includes the notification, in the format you specified upon subscription. Please check the schema if you need more detail on this.

We consider notifications successful if we can reach your callback and it returns a 200 code. If the notification fails, we will retry three times after five, ten and 15 seconds (these times may vary slightly). If we are not able to notify you after these additional attempts, we will drop the notification and you can use the our retrieve feature, discussed above, to get the missing data.

Additionally, notifications will include the following headers for you to inspect:

Best Practice: use HTTPS

You should always use the https endpoints when sending requests to Superfeedr. We also recommend using https for your own endpoints. This guarantees privacy and the integrity of the complete notification (including the headers).

Best Practice: use hub.secret

When subscribing to a feed, you should use hub.secret, unless you are using https for your callback URLs. This secret will be used to compute a signature for each notification. You should always make sure these signatures match. You can read more about that here .

Best Practice: use different callback urls

Your callback urls should be hard to guess. More importantly, you should use different callbacks for each of your subscriptions. This way, you can quickly identify which feed is involved in each notification, without having to parse the content. It also makes it easier to identify problems using the access logs of your HTTP servers.

Replaying Notifications

This call is mostly used as a debugging tool. It allows you to replay past notifications. The token used to perform this call must have the retrieve right set to true.

Example

curl https://push.superfeedr.com/
  -G
  -u demo:demo
  -d'hub.mode=replay'
  -d'hub.topic=http://push-pub.appspot.com/feed'
  -d'hub.callback=http://mycallback.tld/ok'

Response by HTTP status code

• 204: the notification was performed successfully.
• 404: you are not subscribed to the feed.
• 422: please check the body as it will include the reason for the failure.
• Other HTTP response codes are outlined in the HTTP spec.

PubSubHubbub API Wrappers

Multiple PubSubHubbub wrappers exist in a variety of languages. All of these should work fine with the Superfeedr endpoint.

You can also check our Github repository for some platform-specific libraries, like our Ruby Rack Gem.