Implementing Pagination in DynamoDB — A Comprehensive Guide

Introduction:

Pagination is a crucial feature when dealing with large data, allowing you to efficiently manage and display data in smaller and more manageable chunks. Unlike popular SQL and NoSQL databases like MySQL, PostgreSQL, and MongoDB etc., pagination in DynamoDB is a bit different. In the forementioned database management systems, you can create a pagination system by counting all records and setting the correct values for limit and offset. Consider the following example of a MySQL query (using Sequelize ORM), in which all projects that are In Progress status are fetched with pagination.

const { pageNumber } = req.query;const limit = 10; // Or get it from request query as req.query.limit if provided by frontendconst offset = (pageNumber - 1) * limit;const { count, rows } = await Projects.findAndCountAll({ where: { status: "In Progress" }, order: [["createdAt", "DESC"]], limit: limit, offset: offset, });res.status(200).json({ count: count, totalPages: Math.ceil(count / limit), pageNumber: pageNumber, projects: rows, });

From the above example, you can guess that there are few things to keep track of on the frontend, that are, totalPages and pageNumber Using totalPages , we can show the number of pages available and pageNumber for showing the current page that a user is viewing. If you want to control limit from frontend as well, you can include it in the query variable and consider it on the backend as well, no problem at all. The important thing is that count is needed for calculating total pages or telling the frontend how many records we have in total.

The problem in DynamoDB is, we don’t have any count specific query. For counting, we would scan the whole table and then find the length of the returned array which we want to avoid (what is the point of pagination then, if we are scanning the whole table). Imagine you have millions of records, and you are scanning every bit of it, Scary!! How to implement pagination in DynamoDB then? Let’s learn.

Pagination In DynamoDB:

Every scan or query operation in DynamoDB returns a property, which is LastEvaluatedKey that indicates the last item that was read in the scan or query operation. It serves as a marker or token for the next page of results. When LastEvaluatedKey is undefined, it typically indicates that there are no more pages of results to fetch beyond the current page. There is another parameter in DynamoDB query operations,ExclusiveStartKey that takes Primary key as value (Partition key and sort key both if both are present) and is used in paginated queries to specify where to start fetching results from the provided index. Sad news, there is no count query in DynamoDB as discussed earlier so we can’t get the total number of records available and the total number of pages as well. So, it is better to have an infinite scroll or Load more UI on frontend while showing paginated DynamoDB items.

Using LastEvaluatedKey and ExclusiveStartKey , we can implement our pagination as following:

import { DynamoDBDocument } from "@aws-sdk/lib-dynamodb";import { DynamoDB } from "@aws-sdk/client-dynamodb";// In case of serverless frameworklet { lastEvaluatedKey, limit } = event.queryStringParameters;// In express frameworklet { lastEvaluatedKey, limit } = req.query;limit = Number(limit);if (lastEvaluatedKey) { // Since the lastEvaluatedKey may contain special characters that may  // cause problems in URL, the best approach is to encode the URL on // frontend before sending it to the backend. Since the value is URL // encoded, we need to decode it on backend. A typical lastEvaluatedKey // may have value (without URL encoding) as following: // { projectId: "40290" } lastEvaluatedKey = JSON.parse(decodeURIComponent(lastEvaluatedKey));}const dynamodb = DynamoDBDocument.from( new DynamoDB({ region: DYNAMODB_REGION, credentials: { accessKeyId: CUSTOM_AWS_ACCESS_KEY_ID, secretAccessKey: CUSTOM_AWS_SECRET_ACCESS_KEY, }, }));const { Items, Count, LastEvaluatedKey } = await dynamodb.query({ TableName: "Projects", ExclusiveStartKey: lastEvaluatedKey ? lastEvaluatedKey : undefined, Limit: limit, // Limit number of items scanned });// LastEvaluatedKey must be sent to the frontend along with Items so that // it can keep track of the pagination

• Count is representing the number of items in the response.
• Use the Limit parameter to control the maximum number of items returned per page, optimizing throughput and reducing resource consumption.
• If the value of LastEvaluatedKey is undefined, the initial set of items will be returned according to the specified limit. However, if a valid value is provided, the query will return a set of items beginning from the value of LastEvaluatedKey, which contains the primary key of the last evaluated item.