Architecting Out-of-the-Box Node.js Middleware Dev...

ryutaro_umezuki1 · ‎2023 Aug 15

Motivation

I have frequently developed small-scale applications by integrating SAP S/4HANA with other platforms such as Google or OpenAI. I have also reviewed code written by other developers who had similar requirements. In such cases, the use of middleware has been common, but I have identified areas for improvement in both my own code and that of others. As a result, I have studied middleware architecture to establish a streamlined approach for developing small applications that emphasizes reusability, ensures testability, and enables rapid development. I would like to share the underlying concept of this approach.

GitHub repository

Middleware Development

Middleware development allows us to offer common processes as APIs, eliminating the need for separate developments for each client type such as PC, iOS or Android.In addition, by combining the middleware that embeds complex business logic with a LCNC tool, SAP Build Apps, it can facilitate the implementations of Thin Client efficiently.

API communication without vs. with Node.js middleware

However, when it comes to middleware development, there are certain factors to keep in mind. Firstly, the process of choosing and building a middleware architecture takes effort. Decisions involving libraries, error handling, routing, module testing, caching, and security require careful consideration. Additionally, if the API requests and responses are not well-defined for the clients, it can result in time-consuming to verify them. To address these issues, I've put together a TypeScript × Node.js project that offers the following features:

Layered Architecture: Server Config, Route, Controller, Service, Repository

Unified Error Handling

Independent Routing for different platforms

Utilization of Dependency Injection (DI) and DI Container for loose coupling

Data Transfer Objects (DTOs) to declaratively express the HTTP request and response types

Recreating a similar setup in your project might seem laborious, but no need to worry about that. I've provided a 'create-template' yarn command with few parameters that generates files for the Route, Controller, Service, and Repository layers and DI, enabling you to run a SAP S/4HANA GET method. Let's watch a video to see it.

Create templates for Route, Control, Service, Repository layers and DI by yarn command

The Middleware Architecture Overview

The middleware architecture overview

Positive Aspects

Agility: Unified error handling, layered architecture, and template creation combine to enhance agility. Error handling is standardized, reducing redundancy code and enabling rapid addition of new features. Layered architecture facilitates clear responsibility separation, ensuring efficient development. The template creation tool streamlines API template creation through yarn commands, allowing developers to focus on realizing ideas. Due to the pursuit of agility and the consideration of the small project scale, caching has been intentionally omitted.

Flexibility: Independent folder structures for each platform enable easy addition, modification, or removal of platform-specific code, reducing redundancy.

Loose Coupling: Layered architecture, DI, and DI Container foster loose coupling, separating responsibilities. This enables independent development and isolated unit testing for each layer and file, limiting the scope of impact when changes occur.

Negative Aspects

Performance: Having independent API routes for each platform results in multiple requests when serving one business requirement, leading to overhead.

Redundancy: Independent API routes for each platform increase the code quantity.

Note 1. Software Architecture Metrics

In my opinion, achieving perfection in all software architecture metrics is unrealistic due to the trade-offs inherent among various metrics. Giving priority to one metric can negatively affect another. For example, improving security might lead to higher processing demands, potentially slow down performance. Emphasizing robustness, in general, limits flexibility. The selection and prioritization of metrics that align with both present and future project requirements is crucial.

Annotation: Most of actual code has been omitted on this blog post.

Core Object Types

Relationship diagram between object types and layers

DTO (Data Transfer Object): This concept differs from the commonly used DAO/DTO pattern in programming languages like Java. In this context, DTOs are strict types converted from general types of request data sent by clients. In this architecture, you can validate the request data within these DTOs by using 'class-validator' package. This helps identify and prevent invalid values even before they reach the main logic within the Service layer. Additionally, DTOs are utilized in crafting responses. They enable clients to understand the expected formats for both requests and responses when interacting with the middleware API. Moreover, the use of DTOs avoids direct exposure of models to external clients. This way, intricate and unnecessary details remain hidden, offering a more streamlined interface.

import { IsNotEmpty, IsString } from "class-validator";

import { ParsedQs } from "qs";



export class GetInspectionLotQueryDTO {

  @IsNotEmpty()

  @IsString()

  lotID: string;



  constructor(query: ParsedQs) {

    this.lotID = query.lotID as string;

  }

}

export class GetInspectionLotResDTO {

  id: string;

  plant: string;



  constructor(data: {

    id: string;

    plant: string;

  }) {

    this.id = data.id;

    this.plant = data.plant;

  }

}

Model: The classes are converted versions of DAOs and DTOs, offering methods for the application's business logic. In SAP S/4HANA system, APIs might use abstract or repetitive key names to handle different business requests. Therefore, it's crucial to map proper names in the model by 'class-transformer' according to project needs like 'InspectionLot' to 'id'.

import { Exclude, Expose, Transform, Type } from "class-transformer";



@Exclude()

export class InspectionLot {

  @Expose({ name: "InspectionLot" })

  id: string;



  @Expose({ name: "Plant" })

  plant: string;

}

DAO(Data Access Object): It defines types for external API HTTP requests and responses. They are interfaces that do not contain function implementations and used in repository layer.

export interface InspectionLotResDAO {

  InspectionLot: string;

  InspectionLotObjectText: string;

  InspectionLotActualQuantity: string;

  InspectionLotQuantityUnit: string;

  InspectionLotStartDate: string;

  InspectionLotEndDate: string;

  to_InspectionLotWithStatus: { InspLotStatusInspCompleted: string };

}

Layered Architecture

Repository Layer

Be responsible for communication with SAP S/4HANA and other platform APIs.

Manages communication protocols and endpoints with external systems.

Performs operations such as data retrieval, sending, updating, and deletion.

Creates Data Access Objects (DAO) from received data and converts them into models.

import { injectable } from "inversify";



import { InspectionLotResDAO } from "~/s4/dao/inspectionLotResDAO";

import { UsageDecisionReqDAO } from "~/s4/dao/usageDecisionReqDAO";

import { InspectionLot } from "~/s4/model/inspectionLot";

import { UsageDecision } from "~/s4/model/usageDecision";

import { createAPIClientDefault } from "~/s4/repository/index";

import { InspectionRepository } from "~/s4/service/inspectionRepository";

import { AuthData } from "~/type/authData";

import { fromDAO2Model } from "~/utils/fromDAO2Model";



const client = createAPIClientDefault(

  "/sap/opu/odata/sap/API_INSPECTIONLOT_SRV"

);



@injectable()

export class InspectionRepositoryImpl implements InspectionRepository {

  public async findInspectionLotByLotID(lotID: string): Promise<InspectionLot> {

    const res = await client.get(

      `/A_InspectionLot('${lotID}')?$expand=to_InspectionLotWithStatus`

    );

    const dao: InspectionLotResDAO = res.data.d;

    return fromDAO2Model(InspectionLot, dao);

  }



  public async saveUsageDecision(

    authData: AuthData,

    usageDecision: UsageDecision

  😞 Promise<void> {

    const body: UsageDecisionReqDAO = {

      d: {

        InspectionLot: usageDecision.lotID,

        InspLotUsageDecisionLevel: usageDecision.usageDecisionLevel,

        InspectionLotQualityScore: usageDecision.qualityScore,

        InspLotUsageDecisionCatalog: usageDecision.catalogID,

        SelectedCodeSetPlant: usageDecision.plant,

        InspLotUsgeDcsnSelectedSet: usageDecision.usageDecisionSelectedSet,

        InspLotUsageDecisionCodeGroup: usageDecision.catalogName,

        InspectionLotUsageDecisionCode: usageDecision.usageDecisionCode,

        ChangedDateTime: usageDecision.changedDate,

      },

    };



    await client.post(`/A_InspLotUsageDecision`, body, {

      headers: {

        "Content-Type": "application/json",

        "x-csrf-token": authData.xCSRFToken,

        Cookie: authData.cookie,

      },

    });

  }

}

Service Layer

Contains business logic and provides core processing for the application.

Executes model methods.

Interacts with the database through the Repository layer.

Handles tasks such as complex calculations, data transformations, and validations.

import { inject, injectable } from "inversify";



import { TYPES } from "~/di/types";

import { PostUsageDecisionBodyDTO } from "~/s4/dto/postUsageDecisionBodyDTO";

import { InspectionLot } from "~/s4/model/inspectionLot";

import { UsageDecision } from "~/s4/model/usageDecision";

import { CatalogRepository } from "~/s4/service/catalogRepository";

import { InspectionRepository } from "~/s4/service/inspectionRepository";

import { InspectionService } from "~/s4/service/inspectionService";



@injectable()

export class InspectionServiceImpl implements InspectionService {

  private inspectionRepo: InspectionRepository;

  private catalogRepo: CatalogRepository;



  constructor(

    @inject(TYPES.InspectionRepository)

    inspectionRepo: InspectionRepository,

    @inject(TYPES.CatalogRepository)

    catalogRepo: CatalogRepository

  ) {

    this.inspectionRepo = inspectionRepo;

    this.catalogRepo = catalogRepo;

  }



  public async getInspectionLot(lotID: string): Promise<InspectionLot> {

    return await this.inspectionRepo.findInspectionLotByLotID(lotID);

  }



  public async postUsageDecision(

    bodyDTO: PostUsageDecisionBodyDTO

  😞 Promise<void> {

    const authData = await this.inspectionRepo.findAuthData();

    const usageDecision = new UsageDecision({ ...bodyDTO });

    const inspectionResults =

      await this.inspectionRepo.findInspectionResultsByLotID(bodyDTO.lotID);

    usageDecision.qualityScore =

      usageDecision.calcQualityScore(inspectionResults);

    usageDecision.usageDecisionCode = usageDecision.addUsageDecision();

    await this.inspectionRepo.saveUsageDecision(authData, usageDecision);

  }

}

Controller Layer

Receives requests from users and calls appropriate service methods.

Manages tasks like parsing and validating requests, as well as generating responses.

Controls the HTTP request and response processes.

import { inject, injectable } from "inversify";



import { TYPES } from "~/di/types";

import { InspectionController } from "~/s4/controller/inspectionController";

import { GetInspectionLotQueryDTO } from "~/s4/dto/getInspectionLotQueryDTO";

import { GetInspectionLotResDTO } from "~/s4/dto/getInspectionLotResDTO";

import { UsageDecision } from "~/s4/model/usageDecision";

import { InspectionService } from "~/s4/service/inspectionService";

import { ControllerMethod } from "~/type/controllerMethod";

import { HttpStatusCode } from "~/type/httpStatusCode";

import { validateDTO } from "~/utils/validateDTO";



@injectable()

export class InspectionControllerImpl implements InspectionController {

  private service: InspectionService;



  constructor(

    @inject(TYPES.InspectionService) inspectionService: InspectionService

  ) {

    this.service = inspectionService;

  }

  public getInspectionLot: ControllerMethod = async (req, res) => {

    const queryDTO = new GetInspectionLotQueryDTO(req.query);

    await validateDTO(queryDTO);

    const lot = await this.service.getInspectionLot(queryDTO.lotID);

    const resDTO = new GetInspectionLotResDTO(lot);

    res.send(resDTO);

  };



  public postUsageDecision: ControllerMethod = async (req, res) => {

    const bodyDTO = new UsageDecision(req.body);

    await validateDTO(bodyDTO);

    await this.service.postUsageDecision(bodyDTO);

    res.status(HttpStatusCode.CREATED).send({ message: "Post success" });

  };

}

The method for throwing validation errors is standardized in 'validateDTO'.

import { validate } from "class-validator";



import { HTTP400Error } from "~/utils/errors";



export const validateDTO = async (object: object): Promise<void> =>

  validate(object).then((res) => {

    if (res.length > 0) {

      throw new HTTP400Error(

        res[0].constraints[Object.keys(res[0].constraints)[0]]

      );

    }

  });

Route Layer

Executes controller methods based on the HTTP requests from the client, setting the URL path and HTTP method.

Uses a wrapper function to pass errors from asynchronous processes to the server-config layer.

import { Router } from "express";



import { myContainer } from "~/di/inversify.config";

import { TYPES } from "~/di/types";

import { InspectionController } from "~/s4/controller/inspectionController";

import { asyncWrapper } from "~/utils/asyncWrapper";



const controller = myContainer.get<InspectionController>(

  TYPES.InspectionController

);



const inspectionRoute = Router();

inspectionRoute.get(

  "/lot",

  asyncWrapper(async (req, res) => await controller.getInspectionLot(req, res))

);



inspectionRoute.post(

  "/usage-decision",

  asyncWrapper(async (req, res) => await controller.postUsageDecision(req, res))

);



export { inspectionRoute };

By wrapping the try/catch processing with this 'asyncWrapper', you can eliminate the redundancy in the code by not having to write that processing in the controller, service, and repository layers.

import { NextFunction, Request, Response } from "express";



type RouteHandler = (

  req: Request,

  res: Response,

  next: NextFunction

) => Promise<void>;



type AsyncWrapper = (handler: RouteHandler) => RouteHandler;



export const asyncWrapper: AsyncWrapper =

  (handler) => async (req, res, next) => {

    try {

      await handler(req, res, next);

    } catch (error) {

      next(error);

    }

  };

Server Config Layer

Manages overall server settings, including error handling, logging, parsing, routing, and security.

Utilizing the error type received from the asyncWrapper in Route layer, provide error data to the client.

import { AxiosError } from "axios";

import { Application, NextFunction, Request, Response } from "express";



import { HttpStatusCode } from "~/type/httpStatusCode";

import { HTTP400Error } from "~/utils/errors";



interface ErrorResponse {

  name: string;

  httpCode: number;

  isOperational: boolean;

  message: string;

  stack: string | undefined;

}



export const configureErrorHandler = (app: Application): void => {

  app.use(

    (error: Error, _: Request, res: Response, next: NextFunction): void => {

      if (res.headersSent) {

        return next(error);

      }



      if (error instanceof AxiosError) {

        const httpCode =

          error.response?.status || HttpStatusCode.INTERNAL_SERVER;

        const axiosError: ErrorResponse = {

          name: error.code || "AxiosError",

          httpCode: httpCode,

          isOperational: false,

          message:

            error.response?.data?.error?.message?.value ||

            "An error occurred while processing the request.",

          stack: error.stack,

        };

        res.status(httpCode).json(axiosError);

      } else if (error instanceof HTTP400Error) {

        const errorJson: ErrorResponse = createErrorResponse(

          error,

          error.httpCode

        );

        res.status(error.httpCode).json(errorJson);

      } else {

        const errorJson: ErrorResponse = createErrorResponse(

          error,

          HttpStatusCode.INTERNAL_SERVER

        );

        res.status(HttpStatusCode.INTERNAL_SERVER).json(errorJson);

      }

    }

  );

};

This is a route file that aggregates the routes defined on each platform. The 'getInspectionLot' and 'postUsageDecision' functions called within the previous 'inspectionRoute' are executable through the APIs with URLs http:<host>/s4/inspection/lot and http:<host>/s4/inspection/usage-decision, respectively.

import { Application } from "express";



import { userRoute } from "~/externalPlatform1/route/userRoute";

import { inspectionRoute } from "~/s4/route/inspectionRoute";



export const configureRoute = (app: Application): void => {

  app.use("/external-platform1/user", userRoute);

  app.use("/s4/inspection", inspectionRoute);

};

This server.ts runs all server-related configurations and launches the server.

import  express, { Application } from "express";

import http from "http";



import "reflect-metadata";



import { configureErrorHandler } from "~/server-config/errorHandler";

import { configureLogging } from "~/server-config/logging";

import { configureRequestParser } from "~/server-config/requestParser";

import { configureRoute } from "~/server-config/route";

import { configureSecurity } from "~/server-config/security";



const app: Application = express();

const port: number = 3000;



configureLogging(app)

configureRequestParser(app)

configureRoute(app);

configureSecurity(app);

configureErrorHandler(app);



const server = http.createServer(app);

server.listen(port, () => {

  console.log(`Example app listening on port ${port}`);

});

Note 2: Inter-Layer Relationships

Controller Layer can invoke multiple Service Layers, and similarly, Service Layer can call multiple Repository Layers.

Controller - Service: For example, in the case of the Order Controller, both an Order Service for processing orders and an Email Notification Service for notifying order recipients via email are necessary.

Service - Repository: For instance, within the Order Service, access to APIs related to inventory information and payment information requires the use of both the Inventory Repository and Payment Repository.

Inter-layer relationships

Note 3: Dependency Injection (DI) and DI Container

Dependency Injection (DI) is a software design pattern that helps manage dependencies between modules and improve testability. It is a way to flexibly manage dependencies between different parts of software.

Loose coupling, which is achievable through DI, between the Service layer and Repository layer implies that the Service layer depends on interfaces (abstractions) rather than the concrete implementation of the Repository. This methodology simplifies the swapping of modules and facilitates testing.

However, what does it mean to depend on abstractions rather than concrete implementations?　Let's illustrate this concept, and DI and DI container for that.

1. When utilizing methods from a Repository class in the Service Layer, instances might be created directly using "new Repository()" in the constructor. However, this leads to the Service Layer depending directly on specific Repository implementation classes (InspectionRepositoryImpl, CatalogRepositoryImpl), making it difficult to switch repositories for testing. As a project grows and more classes are involved, this process can become complex and hinder testing.

export class InspectionServiceImpl implements InspectionService {

  private inspectionRepo: InspectionRepository;

  private catalogRepo: CatalogRepository;



  constructor() {

    this.inspectionRepo = new InspectionRepositoryImpl();

    this.catalogRepo = new CatalogRepositoryImpl();

  }

}

2. A solution to this problem is to inject the Repository instances as arguments when instantiating the Service Layer. This way, the Service Layer depends on the Repository Interfaces (InspectionRepository, CatalogRepository), allowing easy swapping during testing by injecting concrete implementations like TestInspectionRepositoryImpl.

export class InspectionServiceImpl implements InspectionService {

  private inspectionRepo: InspectionRepository;

  private catalogRepo: CatalogRepository;



  constructor(

    inspectionRepo: InspectionRepository,

    catalogRepo: CatalogRepository

  ) {

    this.inspectionRepo = inspectionRepo();

    this.catalogRepo = catalogRepo();

  }

}

import { GetInspectionLotsQueryDTO } from "~/s4/dto/getInspectionLotsQueryDTO";

import { PostInspectionQlResultBodyDTO } from "~/s4/dto/postInspectionQlResultBodyDTO";

import { PostInspectionQnResultBodyDTO } from "~/s4/dto/postInspectionQnResultBodyDTO";

import { PostUsageDecisionBodyDTO } from "~/s4/dto/postUsageDecisionBodyDTO";

import { CharacteristicCode } from "~/s4/model/characteristicCode";

import { InspectionCharacteristic } from "~/s4/model/inspectionCharacteristic";

import { InspectionLot } from "~/s4/model/inspectionLot";



export interface InspectionService {

  getInspectionLot(lotID: string): Promise<InspectionLot>;

  getInspectionLots(

    queryDTO: GetInspectionLotsQueryDTO

  😞 Promise<InspectionLot[]>;

  getInspectionCharacteristics(

    lotID: string

  😞 Promise<InspectionCharacteristic[]>;

  getCharacteristicCodes(catalogName: string): Promise<CharacteristicCode[]>;

  postInspectionQnResult(bodyDTO: PostInspectionQnResultBodyDTO): Promise<void>;

  postInspectionQlResult(bodyDTO: PostInspectionQlResultBodyDTO): Promise<void>;

  postUsageDecision(bodyDTO: PostUsageDecisionBodyDTO): Promise<void>;

}

However, as the number of dependent classes increases, managing multiple arguments can become complicated.

const controller = new InspectionControllerImpl(

  new InspectionServiceImpl(

    new InspectionRepositoryImpl(),

    new CatalogRepositoryImpl()

  )

);



const inspectionRoute = Router();

inspectionRoute.get(

  "/lot",

  asyncWrapper(async (req, res) => await controller.getInspectionLot(req, res))

);

3.To address this, a DI Container was created. The container automatically resolves dependencies and provides instances. InversifyJS, which offers DI and a DI Container for JavaScript applications, can be an option. By using it, you can configure dependencies and simplify module instantiation.

import { inject, injectable } from "inversify";



import { TYPES } from "~/di/types";

import { CatalogRepository } from "~/s4/service/catalogRepository";

import { InspectionRepository } from "~/s4/service/inspectionRepository";

import { InspectionService } from "~/s4/service/inspectionService";



@injectable()

export class InspectionServiceImpl implements InspectionService {

  private inspectionRepo: InspectionRepository;

  private catalogRepo: CatalogRepository;



  constructor(

    @inject(TYPES.InspectionRepository)

    inspectionRepo: InspectionRepository,

    @inject(TYPES.CatalogRepository)

    catalogRepo: CatalogRepository

  ) {

    this.inspectionRepo = inspectionRepo;

    this.catalogRepo = catalogRepo;

  }

}

const TYPES = {

  //Controller

  InspectionController: Symbol.for("InspectionController"),

  UserController: Symbol.for("UserController"),



  //Service

  InspectionService: Symbol.for("InspectionService"),

  UserService: Symbol.for("UserService"),



  //Repository

  InspectionRepository: Symbol.for("InspectionRepository"),

  CatalogRepository: Symbol.for("CatalogRepository"),

  UserRepository: Symbol.for("UserRepository"),

};



export { TYPES };

import { Container } from "inversify";



import { TYPES } from "~/di/types";

import { UserController } from "~/externalPlatform1/controller/userController";

import { UserControllerImpl } from "~/externalPlatform1/controller/userControllerImpl";

import { UserRepositoryImpl } from "~/externalPlatform1/repository/userRepositoryImpl";

import { UserRepository } from "~/externalPlatform1/service/userRepository";

import { UserService } from "~/externalPlatform1/service/userService";

import { UserServiceImpl } from "~/externalPlatform1/service/userServiceImpl";

import { InspectionController } from "~/s4/controller/inspectionController";

import { InspectionControllerImpl } from "~/s4/controller/inspectionControllerImpl";

import { CatalogRepositoryImpl } from "~/s4/repository/catalogRepositoryImpl";

import { InspectionRepositoryImpl } from "~/s4/repository/inspectionRepositoryImpl";

import { CatalogRepository } from "~/s4/service/catalogRepository";

import { InspectionRepository } from "~/s4/service/inspectionRepository";

import { InspectionService } from "~/s4/service/inspectionService";

import { InspectionServiceImpl } from "~/s4/service/inspectionServiceImpl";



const myContainer = new Container();



//Controller

myContainer

  .bind<InspectionController>(TYPES.InspectionController)

  .to(InspectionControllerImpl);

myContainer.bind<UserController>(TYPES.UserController).to(UserControllerImpl);



//Service

myContainer

  .bind<InspectionService>(TYPES.InspectionService)

  .to(InspectionServiceImpl);

myContainer.bind<UserService>(TYPES.UserService).to(UserServiceImpl);



//Repository

myContainer

  .bind<InspectionRepository>(TYPES.InspectionRepository)

  .to(InspectionRepositoryImpl);

myContainer

  .bind<CatalogRepository>(TYPES.CatalogRepository)

  .to(CatalogRepositoryImpl);

myContainer.bind<UserRepository>(TYPES.UserRepository).to(UserRepositoryImpl);



export { myContainer };

It's important to note that while appropriate usage of DI and other abstractions enhances testability, it's not always necessary to abstract all modules. Over-abstracting with DI can lead to complicated and redundant code. In practice, it's advisable to introduce abstraction only for modules (or layers) that genuinely necessitate loose coupling. This is why, in this scenario, DI has been implemented for the Controller, Service, and Repository layer.

Furthermore, this DI enables the realization of the Dependency Inversion Principle between Service layer and Repository layer.

Dependency injection and dependency inversion principle