Below are the commands to clone the source code (frontend and backend) for the application used in this tutorial
git clone https://github.com/redis-developer/ebook-speed-mern-frontend.git
git clone https://github.com/redis-developer/ebook-speed-mern-backend.git
Cache prefetching is a technique used in database management systems (DBMS) to improve query performance by anticipating and fetching data from the storage subsystem before it is explicitly requested by a query.
There are three main strategies for cache prefetching:
This tutorial will cover the third strategy, prefetching based on data access patterns.
Imagine you're building a movie streaming platform. You need to be able to provide your users with a dashboard that allows them to quickly find the movies they want to watch. You have an extensive database filled with movies, and you have them categorized by things like country of origin, genre, language, etc. This data changes infrequently, and is regularly referenced all over your app and by other data. This kind of data that is long-lived and changes infrequently is called "master data."
One ongoing developer challenge is to swiftly create, read, update, and delete master data. You might store your master data in a system of record like a SQL database or document database, and then use Redis as a cache to speed up lookups for that data. Then, when an application requests master data, instead of coming from the system of record, the master data is served from Redis. This is called the "master data-lookup" pattern.
From a developer's point of view, "master data lookup" refers to the process by which master data is accessed in business transactions, in application setup, and any other way that software retrieves the information. Examples of master data lookup include fetching data for user interface (UI) elements (such as drop-down dialogs, select values, multi-language labels), fetching constants, user access control, theme, and other product configuration.
Below you will find a diagram of the data flow for prefetching master data using Redis with MongoDB as the system of record.
The steps involved in fetching data are as follows:
If you use Redis Cloud, cache prefetching is easier due to its support for JSON and search. You also get additional features such as real-time performance, high scalability, resiliency, and fault tolerance. You can also call upon high-availability features such as Active-Active geo-redundancy.
The demo application used in the rest of this tutorial showcases a movie application with basic create, read, update, and delete (CRUD) operations.
The movie application dashboard contains a search section at the top and a list of movie cards in the middle. The floating plus icon displays a pop-up when the user selects it, permitting the user to enter new movie details. The search section has a text search bar and a toggle link between text search and basic (that is, form-based) search. Each movie card has edit and delete icons, which are displayed when a mouse hovers over the card.
Below are the commands to clone the source code (frontend and backend) for the application used in this tutorial
git clone https://github.com/redis-developer/ebook-speed-mern-frontend.git
git clone https://github.com/redis-developer/ebook-speed-mern-backend.git
Certain fields used in the demo application serve as master data, including movie language, country, genre, and ratings. They are master data because they are required for almost every application transaction. For example, the pop-up dialog (seen below) that appears when a user who wants to add a new movie clicks the movie application plus the icon. The pop-up includes drop-down menus for both country and language. In this case, Redis stores and provides the values.
The code snippet below is used to prefetch MongoDB JSON documents and store them in Redis (as JSON) using the Redis OM for Node.js library.
async function insertMasterCategoriesToRedis() {
...
const _dataArr = await getMasterCategories(); //from MongoDb
const repository = MasterCategoryRepo.getRepository();
if (repository && _dataArr && _dataArr.length) {
for (const record of _dataArr) {
const entity = repository.createEntity(record);
entity.categoryTag = [entity.category]; //for tag search
//adds JSON to Redis
await repository.save(entity);
}
}
...
}
async function getMasterCategories() {
//fetching data from MongoDb
...
db.collection("masterCategories").find({
statusCode: {
$gt: 0,
},
category: {
$in: ["COUNTRY", "LANGUAGE"],
},
});
...
}You can also check RedisInsight to verify that JSON data is inserted, as seen below:
RedisInsight is the free redis GUI for viewing data in redis. Click here to download.
Prior to prefetching with Redis, the application searched the static database (MongoDB) to retrieve the movie's country and language values. As more people started using the application, the database became overloaded with queries. The application was slow and unresponsive. To solve this problem, the application was modified to use Redis to store the master data. The code snippet below shows how the application queries Redis for the master data, specifically the country and language values for the dropdown menus:
*** With Redis ***
*** Redis OM Node query ***
function getMasterCategories() {
...
masterCategoriesRepository
.search()
.where("statusCode")
.gt(0)
.and("categoryTag")
.containOneOf("COUNTRY", "LANGUAGE");
...
}In this tutorial you learned how to use Redis for cache prefetching with a "master data lookup" example. While this is one way Redis is used in an application, it's possible to incrementally adopt Redis wherever needed with other caching strategies/patterns. For more resources on the topic of caching, check out the links below:
