& Continuous Learning

Prerequisites

This article builds on the prior article: Node Reference - Get Product by Id.

Supporting Updates

Sooner or later, there will be a need to modify a product. Except in the case of immutable data stores, rarely are there entities that are truely "insert only". In order to support this, we need an endpoint that allows the modification of a product.

One way to accomplish this is to add support for issuing an HTTP PUT request to the entity URL (the same URL as our GET request). This HTTP request would contain the new product as the body and would overwrite the product in the database with what was received.

The HTTP PUT strategy accomplishes this goal, but it introduces a subtle limitation: A client will generally (1) issue a GET request for a product (2) Parse the JSON response into an object (3) Modify the object, then (4) Serialize the object as JSON back to the server in the body of a PUT request. If the client does not hold every field it receives from the server, then the PUT request will not contain this field and the server will inadvertently remove that data from the database. This isn't a huge problem when the model is small and the clients are mapping every field into their native object types. Unless the clients are correctly configured to hold unknown fields inside their model, simply adding a new attribute to the server becomes a breaking change!

What we need instead is a way for the client to tell the server which select fields it is interested in modifying. One implementation approach is to have the server do some sort of "merge" operation between the database state and the incoming object and only overwrite fields present in the client request. This could work, but is less obvious when dealing with modifying array elements and nested objects. It also involves extra complexity on the server.

Fortunately, there is an RFC{:target="_blank"} that addresses this exact problem. JSON Patch{:target="_blank"} is an RFC that describes a JSON structure that allows a client to explicitly declare what fields should be modified. Each request (a.k.a. "Patch Document") is an array of objects. Each object represents one modification to an entity. This document is sent as the body to the entity url (the same URL that is used to fetch the entity) with an HTTP method of "PATCH{:target="_blank"}". For example, if we wanted to modify just the name of a product, we could send this structure:

[
    {
        "op": "replace", "path": "/name", "value": "Better product name"
    }
]

The "op" field represents the operation we are taking. The "path" field is a JSON Pointer{:target="_blank"} to the location of the edit. The "value" field depends on the operation. In our case, it is the new value we are updating to.

Most of the JSON patch operations ("add", "remove", "replace", "move" & "copy") modify the entity. There is one additional operation called "test" that is worth mentioning because it is extraordinarily powerful. The test operation requires that the "value at the target location is equal to a specified value." This operator allows the client to put conditions on the server to prevent a whole host of concurrency problems. If we were worried about overwriting another users product name in the above example and we know the current value of the name is "Great Widget" we could send this patch document:

[
    {"op": "test", "path": "/name", "value": "Best Widget"},
    {"op": "replace", "path": "/name", "value": "Great Widget"}
]

When the server loads the product to modify it, it will reject the PATCH (probably with a 409 Conflict{:target="_blank"} HTTP status code) if another user has changed the name field while this user was editing it.

We could also more generically have a "version" or "modifiedOn" field on the entity that we could check to support Optimistic Concurrency Control{:target="_blank"}. (This technique is also called Optimistic Offline Lock{:target="_blank"} and can be found built into Java's JPA{:target="_blank"}).

To support JSON patch for products, we need a route that does the following sequence of operations:

1. Load the product from the database
2. If not found, return a 404
3. Apply the patch document to the product
4. Call the validator and validate the product, if validation fails, then return a 400
5. Save the updated product back into the database

Because we are using an HTTP standard, there are libraries available that do all of the heavy lifting when it comes to implementing the PATCH operations. Install fast-json-patch and create products/updateProduct.spec.js with the content below:

npm install fast-json-patch --save

Create products/updateProduct.spec.js:

const proxyquire = require('proxyquire');

describe('products', function () {

    describe('updateProduct', function () {
        beforeEach(function () {
            this.context = {
                params: {
                    id: 'abc'
                },
                request: {
                    body: [
                        {op: 'replace', path: '/name', value: 'new name'}
                    ]
                }
            };
            this.getResponse = {
                Item: {
                    lastModified: '2018-01-02T03:04:05.000Z'
                }
            };
            const documentClient = this.documentClient = {
                get: () => ({
                    promise: () => Promise.resolve(this.getResponse)
                }),
                put: () => ({
                    promise: () => Promise.resolve()
                })
            };
            spyOn(this.documentClient, 'get').and.callThrough();
            spyOn(this.documentClient, 'put').and.callThrough();

            this.validateProduct = (product) => undefined;
            spyOn(this, 'validateProduct').and.callThrough();

            this.updateProduct = proxyquire('./updateProduct', {
                'aws-sdk': {
                    DynamoDB: {
                        DocumentClient: function() {
                            return documentClient;
                        }
                    }
                },
                './validateProduct': this.validateProduct
            });
        });

        afterEach(function() {
            jasmine.clock().uninstall();
        });

        it('should use the correct parameters to get the current state of the product', async function() {
            await this.updateProduct(this.context);
            const expectedParams = {
                TableName: 'Products',
                Segment: undefined,
                Key: {
                    id: 'abc'
                }
            };
            expect(this.documentClient.get.calls.argsFor(0)[0]).toEqual(expectedParams);
        });

        it('should validate the patched product', async function () {
            this.context.request.body = [
                {op: 'replace', path: '/name', value: 'new name'}
            ];
            await this.updateProduct(this.context);
            expect(this.validateProduct.calls.argsFor(0)[0].name).toEqual('new name')
        });

        it('should pass the tablename to save the document', async function() {
            await this.updateProduct(this.context);
            expect(this.documentClient.put.calls.argsFor(0)[0].TableName).toEqual('Products');
        });

        it('should save the patched product', async function () {
            this.context.request.body = [
                {op: 'replace', path: '/name', value: 'new name'}
            ];
            await this.updateProduct(this.context);
            expect(this.documentClient.put.calls.argsFor(0)[0].Item.name).toEqual('new name')
        });

        it('should set the lastModified timestamp', async function () {
            jasmine.clock().mockDate(new Date(Date.UTC(2018, 03, 05, 06, 07, 08, 100)));
            await this.updateProduct(this.context);
            expect(this.documentClient.put.calls.argsFor(0)[0].Item.lastModified).toEqual('2018-04-05T06:07:08.100Z');
        });

        it('should be a conditional update', async function () {
            await this.updateProduct(this.context);
            expect(this.documentClient.put.calls.argsFor(0)[0].ConditionExpression).toEqual('lastModified = :lastModified');
        });

        it('should provide lastModifed as the condition', async function () {
            await this.updateProduct(this.context);
            const expectedValues = {
                ':lastModified': '2018-01-02T03:04:05.000Z'
            }
            expect(this.documentClient.put.calls.argsFor(0)[0].ExpressionAttributeValues).toEqual(expectedValues);
        });

        it('should return a 400 status code if the patch document is invalid', async function () {
            this.context.request.body[0].op = 'bad';
            await this.updateProduct(this.context);
            expect(this.context.status).toEqual(400);
        });

        describe('validation fails', function () {
            beforeEach(function() {
                this.validationError = {
                    '/name': 'some error'
                };
                this.validateProduct.and.returnValue(this.validationError);
            });

            it('should return a 400 status', async function () {
                await this.updateProduct(this.context);
                expect(this.context.status).toEqual(400);
            });

            it('should return the error as the body', async function () {
                await this.updateProduct(this.context);
                expect(this.context.body).toEqual(this.validationError);
            });

            it('should not save the product', async function () {
                await this.updateProduct(this.context);
                expect(this.documentClient.put).not.toHaveBeenCalled();
            });
        });

        describe('patch test fails', function () {
            beforeEach(function() {
                this.getResponse.Item.name = 'Apple';
                this.context.request.body = [
                    {op: 'replace', path: '/name', value: 'Grape'},
                    {op: 'test', path: '/name', value: 'Orange'}
                ];
            });

            it('should return a 409 status', async function () {
                await this.updateProduct(this.context);
                expect(this.context.status).toEqual(409);
            });

            it('should not save the product', async function () {
                await this.updateProduct(this.context);
                expect(this.documentClient.put).not.toHaveBeenCalled();
            });
        });

        it('should return a 409 status if dynamo throws a constraint exception', async function () {
            const checkFailedError = {
                name: 'ConditionalCheckFailedException'
            };
            this.documentClient.put.and.returnValue({
                promise: () => Promise.reject(checkFailedError)
            });
            await this.updateProduct(this.context);
            expect(this.context.status).toEqual(409);
        });
    });
});

And an implementation of the above in a new products/updateProduct.js file:

const AWS = require('aws-sdk');
const documentClient = new AWS.DynamoDB.DocumentClient();
const validateProduct = require('./validateProduct');
const jsonPatch = require('fast-json-patch');
const productsTableName = process.env.PRODUCTS_TABLE_NAME || 'Products';

async function loadProduct(id, Segment) {
    const result = await documentClient.get({
        TableName: productsTableName,
        Segment,
        Key: {id}
    }).promise();

    return result.Item;
}

function validatePatchDocument(patchDocument) {
    const patchErrors = jsonPatch.validate(patchDocument);
    if (patchErrors) {
        return {
            status: 400
        }
    }
}

function applyPatchDocument(product, patchDocument) {
    try {
        jsonPatch.applyPatch(product, patchDocument);
    } catch (e) {
        if (e.name === 'TEST_OPERATION_FAILED') {
            return {
                body: e.operation,
                status: 409
            };
        }
        throw e;
    }
}

function validatePatchedDocument(product) {
    const validationErrors = validateProduct(product);
    if (validationErrors) {
        return {
            body: validationErrors,
            status: 400
        };
    }
}

async function saveProduct(product, lastModified, Segment) {
    product.lastModified = (new Date(Date.now())).toISOString();
    try {
        await documentClient.put({
            TableName: productsTableName,
            Segment,
            Item: product,
            ConditionExpression: 'lastModified = :lastModified',
            ExpressionAttributeValues: {
                ':lastModified': lastModified
            }
        }).promise();
    } catch (e) {
        if (e.name === 'ConditionalCheckFailedException') {
            return {
                status: 409
            };
        }
        throw e;
    }

    return {
        status: 200,
        body: product
    };
}

module.exports = async function(ctx) {
    const id = ctx.params.id;
    const patchDocument = ctx.request.body;
    const product = await loadProduct(id, ctx.segment);
    const lastModified = product.lastModified;

    const response = validatePatchDocument(patchDocument) ||
        applyPatchDocument(product, patchDocument) ||
        validatePatchedDocument(product) ||
        await saveProduct(product, lastModified, ctx.segment);

    ctx.body = response.body;
    ctx.status = response.status;
};

Remember to add the route to your server.js file:

router.patch('/products/:id', require('./products/updateProduct'));

The above patching strategy allows the modification of any field to any value. If another request modifies the product while the patch request is executing, then DynamoDB will throw a "ConditionalCheckFailedException{:target="_blank"}". We are throwing this back to the client as an HTTP 409{:target="_blank"} but we could instead automatically retry the operation.

Security Note: If you have the need to restrict what fields can be modified, a simple check of the patch document to ensure those restricted paths do not exist, can enforce those types of rules.

To test our new product update functionality, first generate a valid JWT:

export USER_POOL_ID=$(aws cloudformation describe-stacks \
    --stack-name Cognito \
    --query 'Stacks[0].Outputs[?OutputKey==`UserPoolId`].OutputValue' \
    --output text)

USER_POOL_CLIENT_ID=$(aws cognito-idp list-user-pool-clients \
    --user-pool-id "$USER_POOL_ID" \
    --max-results 1 \
    --query 'UserPoolClients[0].ClientId' --output text)

CLIENT_SECRET=$(aws cognito-idp describe-user-pool-client --user-pool-id "$USER_POOL_ID" --client-id "$USER_POOL_CLIENT_ID" --query 'UserPoolClient.ClientSecret' --output text)

BEARER_TOKEN=$(curl -s -X POST \
  https://${USER_POOL_CLIENT_ID}:${CLIENT_SECRET}@${AUTH_NAME}.auth.us-east-1.amazoncognito.com/oauth2/token \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d grant_type=client_credentials | \
  python -c "import sys, json; print json.load(sys.stdin)['access_token']")

Now we can grab the id of an existing product and issue a PATCH to that endpoint:

curl --request PATCH  --verbose  \
 http://localhost:3000/products/HyC4DDpbX \
 -H "Authorization: Bearer $BEARER_TOKEN" \
 -H "Content-Type: application/json" \
 --data '[ {"op": "replace", "path": "/name", "value": "Great Widget"} ]'

You can see our template changes here{:target="_blank"}.

Table of Contents

• Introduction
• Unit Testing
• Koa
• Docker
• Cloudformation
• CodePipeline
• Fargate
• Application Load Balancer
• HTTPS/DNS
• Cognito
• Authentication
• DynamoDB
• Put Product
• Validation
• Smoke Testing
• Monitoring
• List Products
• Get Product
• Patch Product (this post)
• History Tracking
• Delete
• Change Events
• Conclusion

If you have questions or feedback on this series, contact the authors at nodereference@sourceallies.com.