Introduction

Have you ever wished you could get a notification whenever a record type or subset of records changed in Salesforce? You may wish to be able to be notified of those changes for replicating data or to trigger behavior or an external business system.

Enter the Streaming API from Salesforce. This feature consists of several subfeatures which enable you to just that. Opting in to certain changes is possible with Push Topics. Getting a complete list of changes of everything is possible with Change Data Capture. We’re going to talk about both in this post and how you can integrate your .NET applications with Salesforce to capture those changes in near real-time.

If you want to see a full breakdown of the Stream API components and capabilities, you can do so here. Also, please be sure your account meets the criteria for the Streaming API. If you have a developer sandbox from https://developer.salesforce.com, you should already be set.

Finally, I recommend you download the Postman collection I created for Salesforce here, and also the sample .NET Core application I created for connecting to the Streaming API.

CometD, Bayeux Protocol, and Long Polling

The Streaming API is possible thanks to a clustered web messaging technology known as CometD. This event messaging system is built on top of the Bayeux protocol which transmits messages over HTTP. When you connect your .NET Core application to the Salesforce Streaming API, you are not constantly checking for new messages every so often. You are actually establishing an open (long-polling) connection, and you are waiting/listening for those changes to be pushed out to you.

Below is a diagram that shows the various connections that are made when establishing a connection to the Streaming API. The client first performs a handshake to establish a long polling connection. If successful, it sends through a subscription to the channel you want to subscribe to (more on this below). A channel represents which type of event notifications you wish to receive. Once the channel subscription is open, the long-polling sequence takes over and you maintain this connection listening for changes.

One advantage the Streaming API has over the Outbound Messages feature from Salesforce, is that you can run this from behind a firewall on your network without exposing an HTTP endpoint for Salesforce to push records to. You can read more about CometD, Bayeux Protocol, and Long Polling on the Salesforce documentation site.

Streaming API Limits

Just like with our previous posts on the REST API and on the Bulk API, Salesforce enforces limits around how many events can be captured per 24 hour period with the Streaming API. The limits are in place because Salesforce is a multi-tenant application, and resources are shared among customers. Constraints are considered beneficial because they promote good software design, and when you are developing your applications to connect to the Streaming API, you should be cognizant of the limits in place (which may vary from environment to environment, or customer to customer).

What is Change Data Capture?

Change Data Capture, which is set to formally be deployed in the Spring 19 (tentatively set at Feb. 8, 2019) release, is currently in developer preview.

Below is an example payload you would receive when subscribe to a Change Data Capture channel when one of those entity types changed:

A few points are worth calling out in the payload json which gets sent to us are worth talking briefly about. The first is the entity type (1) which was changed. Change Data Capture allows you to subscribe to certain channels, such as /data/AccountChangeEvent for just Account changes, or in the case of the payload above all changes via /data/ChangeEvents (5). More on subscription channels for Change Data Capture is available here.

The next piece of information we get is the changeType (2) which outlines what type of operation happened. We’re also given just the data which changed (3) so we can limit the size of our payload and know what exactly needs to be updated.

Finally, we’re given a replayId (4) which can be used as a sort of audit trail for this change. Salesforce supports the notion of durable messages which means that these events are saved on the platform, and can be ‘replayed’, for a period of time (1 – 3 days). When you establish your ‘subscription’ in the connection diagram above, you can also provide a replay value. A value of -1 (default) means you want just new messages on this channel. A value of -2 means you want everything available (be aware of your api limits and processing time required for this). A value of X (a prior replayId) means you want everything after a certain event in time you had captured (assuming it falls in the availability window).

I won’t be discussing replays in this post, partly because I’m still working on updating my sample code to support them, but if you want to learn more about them you can see a nice write-up here.

Setting up Change Data Capture Events

Inside your development org, head over to Settings > Integrations > Change Data Capture. Here you will see a screen that looks similar to the screenshot below. Listed are two columns: Available entities which you can subscribe, and Selected entities to which you have already subscribed to. Select a few examples, such as Lead, Account, and Contact. You can also subscribe to changes to custom objects you have created as well.

For each standard entity you selected, you will be able to retrieve those events via /data/<Type>ChangeEvent channel. Custom objects are available via a /data/<CustomObject>_ChangeEvent channel. If you want to subscribe to all changes for all of your selected entities, the /data/ChangeEvents channel will allow you to do that.

Subscribing to Change Data Capture Events

I’ve created a sample project which connects to the Salesforce Streaming API and set it up on Github which you can use as a starting point to begin receiving notifications. You’ll need to be sure to update the appsettings.json file with your relevant values. As an important aside, please be sure to use user secrets for any development and environment variables or azure key vault for any production configuration to ensure you are safely maintaining credentials for your orgs.

Once you’ve updated your settings, you can run the application. I’d encourage you to step through the code as well, which first establishes an OAuth token with your credentials, then performs a channel subscription to the channel you provided, which I’ve defaulted to /data/ChangeEvents to see all changes. If you have trouble connecting, be sure you can connect to the REST API using Postman and then try again.

As you can see in the screenshot above, we’ve managed to successfully connect to our org (1), perform a handshake to establish a long-polling connection (2), and designate our channel subscription (3, 4). Once the application began waiting for changes, we updated a Lead record and changed the company field to “American Banking Corp2” which you see outlined in our change data capture payload (5).

That about wraps it up for Change Data Capture. There is certainly a lot more to explore, but if you are looking to replicate every change to certain record types in your org for replication or other purposes, you can obtain that information using this feature of the Streaming API. There is also a great Trailhead module on Change Data Capture if you want to apply this badge to your Trailhead account and learn more.

What are PushTopics?

Push Topics are a way to receive more granular notifications based on a SOQL query you define. If you’re not familiar with SOQL, it is very similar to SQL but with a few minor differences around how records are joined. You can see a few examples here. Salesforce analyzes record changes against your SOQL query, and based on criteria outlined below, determines if the record should be pushed out to you.

Another parameter to consider when developing your Push Topic SOQL query, is the NotifyForFields value. Your options for this are: All, Referenced (default), Select, and Where. All means you want all field changes pushed when the record matches the WHERE clause of your query. Referenced means you would like only changes to fields in your SELECT and WHERE, for all records which match the WHERE clause. Select indicates you would like only changes to fields in your SELECT statement, for all records that match your WHERE clause. Where is just like the Select behavior, but instead of basing changes on fields in the Select, it monitors changes for fields denoted in the Where clause only.

Setting Up PushTopics

Thankfully, it’s possible to create Push Topic records with the REST API. You can download the latest Postman collection which includes these calls. To create a push topic, we create a new sObject with a name (1), which we will later use to subscribe to our channel at
/topic/Chicago_Leads. We also provide a SOQL query (2) which indicates Name, FirstName, LastName, and Industry in the fields (Id is always required) and a where clause looking for Leads just in Chicago. We also designate a NotifyForFields parameter of Select which means we’ll get push notifications when the FirstName or LastName change (since they are in the Select fields) but only for leads in Chicago.

Once the push topic has been created we get an Id back (4) which we can use for updating this topic (if needed). One point to also mention here, is we also included the Name field in our query. Name is a compound field, and Salesforce has some special rules around notifications for compound fields like name and address when creating push topics.

Subscribing to PushTopics

Updating our appsettings.json file in our demo application to set the channel to “/topic/Chicago_Leads”, and connecting to the channel (1), then in our browser editing a lead to change the Industry, where that lead had the City = ‘Chicago’, will result in a streaming API event being pushed to us (2).

Another way we can test our Push Topics, is to use the Salesforce Workbench utility. After logging in, we navigate to Queries > Streaming Push Topics. Here we can create a new test Push Topic (there is no option to specify the NotifyForFields parameter, so assume Referenced), and we can view the stream of events/messages that are happening.

Salesforce has a nice Trailhead module on Push Topics and the Streaming API if you’d like to learn more about this subject.

Summary

To recap, the Streaming API is a great way to connect an externally hosted application or dataset to the Salesforce system to keep that system up to date with changes that are happening in Salesforce. There are a number of options for pushing just a certain predefined set of records with Push Topics, or getting a fire hose of all the changes with Change Data Capture, the Streaming API enables us as developers to keep systems in sync with each other.