Page updated Jan 16, 2024

Advanced workflows

Delta Sync

DeltaSync allows you to perform automatic synchronization with an AWS AppSync GraphQL server. The client will perform reconnection, exponential backoff, and retries when network errors take place for simplified data replication to devices. It does this by taking the results of a GraphQL query and caching it in the local Apollo cache. The DeltaSync API manages writes to the Apollo cache for you, and all rendering in your app (such as from React components, Angular bindings) should be done through a read-only fetch.

In the most basic form, you can use a single query with the API to replicate the state from the backend to the client. This is referred to as a "Base Query" and could be a list operation for a GraphQL type which might correspond to a DynamoDB table. For large tables where the content changes frequently and devices switch between offline and online frequently as well, pulling all changes for every network reconnect can result in poor performance on the client. In these cases you can provide the client API a second query called the "Delta Query" which will be merged into the cache. When you do this the Base Query is run an initial time to hydrate the cache with data, and on each network reconnect the Delta Query is run to just get the changed data. The Base Query is also run on a regular bases as a "catch-up" mechanism. By default this is every 24 hours however you can make it more or less frequent.

By allowing clients to separate the base hydration of the cache using one query and incremental updates in another query, you can move the computation from your client application to the backend. This is substantially more efficient on the clients when regularly switching between online and offline states. This could be implemented in your AWS AppSync backend in different ways such as using a DynamoDB Query on an index along with a conditional expression. You can also leverage Pipeline Resolvers to partition your records to have the delta responses come from a second table acting as a journal. A full sample with CloudFormation is available in the AppSync documentation. The rest of this documentation will focus on the client usage.

You can also use Delta Sync functionality with GraphQL subscriptions, taking advantage of both only sending changes to the clients when they switch network connectivity but also when they are online. In this case you can pass a third query called the "Subscription Query" which is a standard GraphQL subscription statement. When the device is connected, these are processed as normal and the client API simply helps make setting up realtime data easy. However, when the device transitions from offline to online, to account for high velocity writes the client will execute the resubscription along with synchronization and message processing in the following order:

  1. Subscribe to any queries defined and store results in an incoming queue
  2. Run the appropriate query (If baseRefreshIntervalInSeconds has elapsed, run the Base Query otherwise only run the Delta Query)
  3. Update the cache with results from the appropriate query
  4. Drain the subscription queue and continue processing as normal

Finally, you might have other queries which you wish to represent in your application other than the base cache hydration. For instance a getItem(id:ID) or other specific query. If your alternative query corresponds to items which are already in the normalized cache, you can point them at these cache entries with the cacheUpdates function which returns an array of queries and their variables. The DeltaSync client will then iterate through the items and populate a query entry for each item on your behalf. If you wish to use additional queries which don't correspond to items in your base query cache, you can always create another instance of the client.sync() process.

Usage

1// Start DeltaSync
2const subscription = client.sync(options)
3/*
4Under the covers, this is actually an Observable<T> that the AppSync client automatically subscribes to for you, so the returned object is a "subscription". This means that you can automatically stop the synchronization process like so:
5*/
6// Stop DeltaSync
7subscription.unsubscribe();

The options object

baseQuery

  • query: A DocumentNode for the base data (e.g. as returned by gql)
  • variables [optional]: An object with the query variables, if any.
  • baseRefreshIntervalInSeconds [optional]: Number of seconds after which the base query will be run again. Default value: 86400 (24 hrs)
  • update [optional]: A function to update the cache, see: Apollo's update function

subscriptionQuery

  • query: A DocumentNode for the subscription (e.g. as returned by gql)
  • variables [optional]: An object with the query variables, if any.
  • update [optional]: A function to update the cache, see: Apollo's update function

deltaQuery

  • query: A DocumentNode for the deltas (e.g. as returned by gql)
  • variables [optional]: An object with the query variables, if any.
  • update [optional]: A function to update the cache, see: Apollo's update function

The buildSync helper

The quickest way to get started with the DeltaSync feature is by using the buildSync helper function. This helper function will build an options object with the appropriate update functions that will update the cache for you in a similar fashion to the offline helpers.

The first argument you need to pass is the GraphQL __typename for your base query. The second argument is the options object from the previous section (without the update keys, since those will be generated for you by this helper function).

You can optionally pass a cacheUpdates parameter to the second argument with the following structure:

  • deltaRecord: A function which receives a deltaRecord (e.g. an individual item in the cache populated by the base/delta/subscription query) and returns an array of GraphQL queries and it's variables to be written to the cache.

Example:

1client.sync(
2 buildSync("Post", {
3 baseQuery: {
4 query: DeltaSync.BaseQuery
5 },
6 subscriptionQuery: {
7 query: DeltaSync.Subscription
8 },
9 deltaQuery: {
10 query: DeltaSync.DeltaSync
11 },
12 cacheUpdates: ( deltaRecord ) => {
13 const id = deltaRecord.id;
14 return [{ query: DeltaSync.GetItem, variables: { id: id } }];
15 }
16 })
17 )

Requirements for helper function

  • Your baseQuery returns a list, not a nested type
  • Your deltaQuery expects a parameter called lastSync of type AWSTimestamp and returns a list with the same fields as your baseQuery (an optionally, an aws_ds field with a value of 'DELETE' for deletions, any other value for insert/update)
  • The mutations that trigger the subscription in your subscriptionQuery should return a single record with the same fields as the items from your baseQuery, (an optionally, an aws_ds field with a value of 'DELETE' for deletions, any other value for insert/update)

Example

The schema for this sample is below. A full sample with CloudFormation is available in the AppSync documentation.

1input CreatePostInput {
2 author: String!
3 title: String!
4 content: String!
5 url: String
6 ups: Int
7 downs: Int
8}
9
10enum DeltaAction {
11 DELETE
12}
13
14type Mutation {
15 createPost(input: CreatePostInput!): Post
16 updatePost(input: UpdatePostInput!): Post
17 deletePost(id: ID!): Post
18}
19
20type Post {
21 id: ID!
22 author: String!
23 title: String!
24 content: String!
25 url: AWSURL
26 ups: Int
27 downs: Int
28 createdDate: String
29 aws_ds: DeltaAction
30}
31
32type Query {
33 getPost(id: ID!): Post
34 listPosts: [Post]
35 listPostsDelta(lastSync: AWSTimestamp): [Post]
36}
37
38type Subscription {
39 onDeltaPost: Post
40 @aws_subscribe(mutations: ["createPost","updatePost","deletePost"])
41}
42
43input UpdatePostInput {
44 id: ID!
45 author: String
46 title: String
47 content: String
48 url: String
49 ups: Int
50 downs: Int
51}
52
53schema {
54 query: Query
55 mutation: Mutation
56 subscription: Subscription
57}

Sample queries

1query Base {
2 listPosts {
3 id
4 title
5 author
6 content
7 }
8}
9
10query Delta($lastSync: AWSTimestamp!) {
11 listPostsDelta(
12 lastSync: $lastSync
13 ) {
14 id
15 title
16 author
17 content
18 aws_ds
19 }
20}
21
22subscription Subscription {
23 onDeltaPost {
24 id
25 title
26 author
27 content
28 aws_ds
29 }
30}

Define the queries from above in a ./graphql/DeltaSync.js file to import in your app:

1import gql from "graphql-tag";
2
3export const BaseQuery = gql`query Base{
4 listPosts {
5 id
6 title
7 author
8 content
9 }
10}`;
11
12export const GetItem = gql`query GetItem($id: ID!){
13 getPost(id: $id) {
14 id
15 title
16 author
17 content
18 }
19}`;
20
21export const Subscription = gql`subscription Subscription {
22 onDeltaPost {
23 id
24 title
25 author
26 content
27 }
28}`;
29
30export const DeltaSync = gql`query Delta($lastSync: AWSTimestamp!) {
31 listPostsDelta(
32 lastSync: $lastSync
33 ) {
34 id
35 title
36 author
37 content
38 aws_ds
39 }
40}`;
1import { AWSAppSyncClient, buildSync } from "aws-appsync";
2import * as DeltaSync from "./graphql/DeltaSync";
3
4const client = new AWSAppSyncClient({
5 // ...
6});
7
8const subscription = client.sync(
9 buildSync('Post', {
10 baseQuery: { query: DeltaSync.BaseQuery },
11 subscriptionQuery: { query: DeltaSync.Subscription },
12 deltaQuery: { query: DeltaSync.DeltaSync },
13 cacheUpdates : ({id}) => [{query: DeltaSync.getItem, variables: {id}]
14 })
15);

React example

Suppose you have an app created with Create React App with the following structure:

  • App.js
    • Sets up AWSAppSyncClient and client.sync as above
    • Renders <AllPosts /> and <SinglePost item={2}>
  • AllPosts.jsx exports <AllPosts />
  • GetPost.jsx exports <SinglePost item={id}>

App.js

1const client = new AWSAppSyncClient({
2 url: awsconfig.aws_appsync_graphqlEndpoint,
3 region: awsconfig.aws_appsync_region,
4 auth: {
5 type: awsconfig.aws_appsync_authenticationType,
6 apiKey: awsconfig.aws_appsync_apiKey
7 }
8});
9
10client.hydrated().then(() =>
11 client.sync(
12 buildSync("Post", {
13 baseQuery: {
14 query: DeltaSync.BaseQuery
15 },
16 subscriptionQuery: {
17 query: DeltaSync.Subscription
18 },
19 deltaQuery: {
20 query: DeltaSync.DeltaSync
21 },
22 cacheUpdates: ({ id }) => [
23 { query: DeltaSync.GetItem, variables: { id } }
24 ]
25 })
26 )
27);
28
29const App = () => (
30 <ApolloProvider client={client}>
31 <Rehydrated>
32 <div>
33 <OnePost id="96d5e889-38ba-4846-84d0-a11d6447d34b" />
34 <hr />
35 <AllPosts />
36 </div>
37 </Rehydrated>
38 </ApolloProvider>
39);

In AllPosts.jsx you would have code like so:

1const AllPosts = ({ postsList }) => (
2 <div>
3 <pre style={% raw %}{{ textAlign: "left" }}{% endraw %}>
4 {JSON.stringify(postsList, null, 2)}
5 </pre>
6 </div>
7);
8
9export default graphql(DeltaSync.BaseQuery, {
10 options: {
11 fetchPolicy: "cache-only"
12 },
13 props: ({ data }) => ({
14 postsList: data.listPosts || []
15 })
16})(AllPosts);

In GetPost.jsx you would have:

1const OnePost = ({ post }) => (
2 <div>
3 <pre style={% raw %}{{ textAlign: "left" }}{% endraw %}>{JSON.stringify(post, null, 2)}</pre>
4 </div>
5);
6
7export default graphql(DeltaSync.GetItem, {
8 options: ({ id }) => ({
9 variables: { id },
10 fetchPolicy: "cache-only"
11 }),
12 props: ({ data: { getPost } }) => ({
13 post: getPost
14 })
15})(OnePost);

Note: The fetchPolicy is cache-only as all of the network requests are handled automatically by the client.sync() operation. You should use this if using different queries in other components as the client.sync() API manages the cache lifecycle. If you use another fetch-policy such as cache-and-network then extra network requests may take place negating the Delta Sync benefits.

Writing update functions

If you do not want to use the buildSync helper then you are responsible for managing cache updates in your application code. Note that this can be a complex process as you will need to manage create, update, and deletes appropriately. An example of this would be updating the cache with a delta record as below, noting that you must update the returned type to match the type from your base query.

1client.sync({
2 baseQuery: { query: DeltaSyncQueries.BaseQuery },
3 deltaQuery: {
4 query: DeltaSyncQueries.DeltaSync,
5 update: (cache, { data: { listPostsDelta } }) => {
6 const query = DeltaSyncQueries.GetItem;
7
8 listPostsDelta.forEach(deltaRecord => {
9 const variables = { id: deltaRecord.id };
10
11 cache.writeQuery({
12 query,
13 variables,
14 data: { getPost: { ...deltaRecord, __typename: 'Post' } }
15 });
16 });
17 }
18 }
19 });