Merge pull request #257 from valor-software/monitoring-the-progress

feat(article): added Monitoring progress requests

Author: sergeimelnik1980

assets/articles/0064-monitoring-the-progress-of-an-http-request-in-nestjs-via-websockets/Slava_Chub.jpg

@@ -0,0 +1,346 @@
+A couple of days ago, I faced an issue with Monitoring the Progress of an HTTP Request. According to old developers' tradition, firstly, I asked Google for it. I anticipated getting a bunch of different answers and choosing an appropriate one. This time my instincts failed me. Even though I got a bunch of similar solutions, I didn't find the appropriate example. It is worth clarifying that I'm working on a NestJS-based project. Let me explain why I decided to create my solution from scratch and why most of the existing solutions on the topic need to be revised in my case.
+
+First, I want to share the https://dev.to/tqbit/how-to-monitor-the-progress-of-a-javascript-fetch-request-and-cancel-it-on-demand-107f[article that describes a bunch of the results above as well as possible, window=_blank]. Let me provide essential thoughts on the article.
+
+1. The article describes the request that provides content downloading. In this case, we are talking about the precious content size. +
+2. The #Content-Length# HTTP header is essential to the correct HTTP response. +
+3. After the server application sets #Content-Length#, chunked data writing process should be run. +
+4. First, the client application gets #Content-Length#. +
+5. After that, it gets every data chunk and calculates the progress as the following.
+
+[,js]
+----
+progress = 100 * (chunkSize / contentLength)
+----
+
+The approach above is beneficial if we are talking about content downloading. Despite it doesn't work in my case due to the following reasons.
+
+1. My task is about something other than content downloading. Moreover, we need to have a functionality that allows us to calculate the progress according to calculations, not only according to data transfer. +
+2. Despite the application not knowing the content size, it has a total number of iterations. +
+3. Chunk-based approach doesn't work in this case. The final result preparation will take a long time, and the output data should be written to the response simultaneously. That's why we need to inform the client before sending a response.
+
+In other words, the requirements for the new approach are the following.
+
+1. The response writing goes simultaneously without any data chunking. +
+2. The progress should be provided before that. +
+
+I don't want to waste your time and give a couple of conceptual points of my approach regarding accounting the requirements above.
+
+* Provide the progress via WebSockets because of persistent connection and high performance.
+* Connect WebSockets with a current session to pass all needed data from the HTTP request processing process.
+
+All thoughts and code below will be strongly connected to these points. But before, let me share the https://github.com/buchslava/nest-request-progress[final solution, window=_blank].
+
+== Data Providing Example
+
+I provided a simplified version of the data processing because I want to focus on this task. We have 150 iterations in the example below. The result is an array of 150 random numbers, each calculates in 100 - 1000 milliseconds. I found this example as a minimally viable model of the objective process.
+
+[, js]
+----
+import { Injectable } from '@nestjs/common';
+
+const getRandomArbitrary = (min: number, max: number): number =>
+  Math.random() * (max - min) + min;
+const delay = (time: number) =>
+  new Promise((resolve) => setTimeout(resolve, time));
+
+@Injectable()
+export class AppService {
+
+  getIterationCount(): number {
+    return 150;
+  }
+
+  async getData(token: string): Promise<string[]> {
+    return new Promise(async (resolve, reject) => {
+      try {
+        const result = [];
+
+        for (let i = 0; i < this.getIterationCount(); i++) {
+          result.push(getRandomArbitrary(1, 9999));
+          await delay(getRandomArbitrary(100, 1000));
+        }
+
+        resolve(result);
+      } catch (e) {
+        reject(e);
+      }
+    });
+  }
+}
+----
+
+== Progress Manager
+The future steps are regarding the #ProgressManager# implementation.
+
+The #ProgressManager# should be a separate NestJS service able to do the following.
+
+1. Start the "Progress" session (Not the HTTP session) with the unique token taken from the client application. +
+2. Stop the "Progress" session +
+3. Increase the value of the progress. +
+
+Please look at the following commented code.
+[, js]
+----
+import { Injectable } from '@nestjs/common';
+import { Server } from 'socket.io';
+
+export interface ProgressSession {
+  token: string;
+  total: number;
+  counter: number;
+  timerId: any;
+}
+
+@Injectable()
+export class ProgressManager {
+  // The Socket Server injection will be described later
+  public server: Server;
+  // This map contains all Progress session data
+  private storage: Map<string, ProgressSession> = new Map();
+
+  // Start the session with the token and the total number of iterations
+  startSession(token: string, total: number, delay = 2000) {
+    // Get current session from the storage
+    const currentSession = this.storage.get(token);
+    // Do nothing if it's already exist
+    if (currentSession) {
+      return;
+    }
+    // Send the progress every "delay" milliseconds
+    const timerId = setInterval(async () => {
+      const currentSession: ProgressSession = this.storage.get(token);
+      // Protect the functionality: if the current session is missing then do nothing
+      if (!currentSession) {
+        return;
+      }
+      // Calculate the progress
+      let progress = Math.ceil(
+        (currentSession.counter / currentSession.total) * 100
+      );
+      // Protect the progress value, it should be less or equal 100
+      if (progress > 100) {
+        progress = 100;
+      }
+      // Send the progress. Pay attention that the event name should contain the "token"
+      // Client will use this token also
+      this.server.emit(`progress-${token}`, progress);
+    }, delay);
+    // Initial Progress Session settings. Token is a key.
+    this.storage.set(token, {
+      token,
+      total,
+      counter: 0,
+      timerId,
+    });
+  }
+
+  // This method increases the progress
+  step(token: string, value = 1) {
+    // Get the current session
+    const currentSession: ProgressSession = this.storage.get(token);
+    // Do nothing if it doesn't exist
+    if (!currentSession) {
+      return;
+    }
+    // Increase the counter
+    const counter = currentSession.counter + value;
+    // Update the storage
+    this.storage.set(token, {
+      ...currentSession,
+      counter,
+    });
+  }
+
+  // Stop the session by the token
+  stopSession(token: string) {
+    // Get the current session
+    const currentSession: ProgressSession = this.storage.get(token);
+    // Do nothing if it doesn't exist
+    if (currentSession) {
+      // Stop the current timer
+      clearInterval(currentSession.timerId);
+      // Remove information regarding the current session from the storage
+      this.storage.delete(token);
+    }
+  }
+}
+----
+You can find the code above https://github.com/buchslava/nest-request-progress/blob/main/packages/server/src/app/progress-manager.ts[here, window=_blank].
+
+== WebSockets Server
+
+Another important is the integration of NestJS with WebSockets and connecting the Progress Manager with it. The following code is responsible for that.
+
+[, js]
+----
+import {
+  WebSocketGateway,
+  WebSocketServer,
+  OnGatewayInit,
+} from '@nestjs/websockets';
+import { Server } from 'socket.io';
+import { ProgressManager } from './progress-manager';
+
+@WebSocketGateway({ cors: true })
+export class AppGateway implements OnGatewayInit {
+  constructor(private progressManager: ProgressManager) {}
+
+  @WebSocketServer() server: Server;
+
+  afterInit() {
+    // After the WebSockets Gateway has to init, then pass it to the ProgressManager
+    this.progressManager.server = this.server;
+  }
+}
+----
+https://github.com/buchslava/nest-request-progress/blob/main/packages/server/src/app/app.gateway.ts[The source >>, window=_blank] +
+And, of course, according to NestJS requirements, we need to tell the related module about that.
+[, js]
+----
+import { Module } from '@nestjs/common';
+
+import { AppController } from './app.controller';
+import { AppService } from './app.service';
+import { AppGateway } from './app.gateway';
+import { ProgressManager } from './progress-manager';
+
+@Module({
+  imports: [],
+  controllers: [AppController],
+  providers: [AppService, AppGateway, ProgressManager],
+})
+export class AppModule {}
+----
+https://github.com/buchslava/nest-request-progress/blob/main/packages/server/src/app/app.module.ts[The source >>, window=_blank]
+
+== Data Processing
+
+It's time to focus on the endpoint's controller. It looks pretty simple.
+
+[, js]
+----
+import { Controller, Get, Query } from '@nestjs/common';
+import { AppService } from './app.service';
+
+@Controller()
+export class AppController {
+  constructor(private readonly appService: AppService) {}
+
+  @Get()
+  getData(@Query() query: { token: string }) {
+    return this.appService.getData(query.token);
+  }
+}
+----
+https://github.com/buchslava/nest-request-progress/blob/main/packages/server/src/app/app.controller.ts[The source >>, window=_blank]
+
+And the last thing about the server is regarding the Data Providing Example modification. The following example is close to the first example in this article. The main aim is to add "Progress functionality" here. Please, read the comment in the code. It's important.
+
+[, js]
+----
+import { Injectable } from '@nestjs/common';
+import { ProgressManager } from './progress-manager';
+
+const getRandomArbitrary = (min: number, max: number): number =>
+  Math.random() * (max - min) + min;
+const delay = (time: number) =>
+  new Promise((resolve) => setTimeout(resolve, time));
+
+@Injectable()
+export class AppService {
+  // Use progressManager
+  constructor(private readonly progressManager: ProgressManager) {}
+
+  // 150 iterations should be processed
+  getIterationCount(): number {
+    return 150;
+  }
+
+  async getData(token: string): Promise<string[]> {
+    return new Promise(async (resolve, reject) => {
+      // We need to start the Progress Session before data preparation
+      this.progressManager.startSession(token, this.getIterationCount());
+      try {
+        // Initialize the array of results
+        const result = [];
+
+        for (let i = 0; i < this.getIterationCount(); i++) {
+          // Calculate the result        
+          result.push(getRandomArbitrary(1, 9999));
+          // Increase the Progress counter
+          this.progressManager.step(token);
+          // Random delay
+          await delay(getRandomArbitrary(100, 1000));
+        }
+
+        // Return the result
+        resolve(result);
+      } catch (e) {
+        reject(e);
+      } finally {
+        // We need to stop the ProgressManager in any case.
+        // Otherwise, we have a redundant timeout.
+        this.progressManager.stopSession(token);
+      }
+    });
+  }
+}
+----
+https://github.com/buchslava/nest-request-progress/blob/main/packages/server/src/app/app.service.ts[The source >>, window=_blank]
+
+The backend part of my example is ready. You can find the full backend solution https://github.com/buchslava/nest-request-progress/tree/main/packages/server[here, window=_blank].
+
+== The Client
+
+The client part of my example is placed https://github.com/buchslava/nest-request-progress/tree/main/packages/client[here, window=_blank]. Both parts are placed in one monorepo. Thanks https://nx.dev/[Nx, window=_blank] for that. Lets look at it. Please, read the comments in the code below.
+
+[, js]
+----
+import * as io from 'socket.io-client';
+import { v4 } from 'uuid';
+import axios from 'axios';
+
+// Generate a unique ID (token)
+const token = v4();
+
+console.info(new Date().toISOString(), `start the request`);
+
+// Call the endpoint described above
+axios
+  .get(`http://localhost:3333/api?token=${token}`)
+  .then((resp) => {
+    // Print the total length of requested data (an array of random numbers)
+    console.info(new Date().toISOString(), `got ${resp.data.length} records`);
+    process.exit(0);
+  })
+  .catch((e) => {
+    console.info(e);
+    process.exit(0);
+  });
+// We need to connect to the related Socket Server
+const ioClient = io.connect('ws://localhost:3333');
+// And wait for `progress-${token}` event
+ioClient.on(`progress-${token}`, (progress) =>
+  console.info(new Date().toISOString(), `processed ${progress}%`)
+);
+----
+
+== The Final Steps
+It's time to try the solution.
+[, bash]
+----
+git clone git@github.com:buchslava/nest-request-progress.git
+cd nest-request-progress
+npm i
+npx nx run server:serve
+----
+Open another terminal and run:
+[, bash]
+----
+npx nx run client:serve
+----
+
+== Voilà
+[.img]
+image::img1.png[]

assets/articles/0064-monitoring-the-progress-of-an-http-request-in-nestjs-via-websockets/img1.png

@@ -0,0 +1,12 @@
+{
+"title": "Monitoring the Progress of an HTTP Request in NestJS via WebSockets",
+"order": 64,
+"domains": ["dev_quality_assurance"],
+"authorImg": "assets/articles/monitoring-the-progress-of-an-http-request-in-nestjs-via-websockets/Slava_Chub.jpg",
+"language": "en",
+"bgImg": "assets/articles/monitoring-the-progress-of-an-http-request-in-nestjs-via-websockets/main-monitoring-the-progress.png",
+"author": "Vyacheslav Chub",
+"position": "Full Stack Software Engineer",
+"date": "Mon Feb 28 2021 10:45:55 GMT+0000 (Coordinated Universal Time)",
+"seoDescription": "The article describes the request that provides content downloading"
+}