Hi,
I am starting with the first blog.
Recently, I have come across a requirement where we are supposed to integrate with MinIO. MinIO is a High-Performance Object Storage released under GNU Affero General Public License v3.0.
It is API-compatible with the Amazon S3 cloud storage service. It can handle unstructured data such as photos, videos, log files, backups, and container images with a current maximum supported object size of 50TB.
We had a requirement to upload and download files to/from MinIO bucket. This will cover both functionalities in single blog.
In the Upload scenario, we used to get PDF files from SAP System as attachment to proxy which was extracted in Java mapping and sent to MinIO bucket.
Source structure,
<?xml version="1.0" encoding="utf-8"?>
<n0:MT_ProxyminioSender xmlns:n0=" " xmlns:prx="urn:sap.com:proxy:ECP:/1SAI/TAS8E5273429B1BFFB79A16:756">
<DOCNO>5400046098</DOCNO>
<Filename>5400046098_1.PDF</Filename>
</n0:MT_ProxyminioSender>
Here with
DOCNO, we are creating a folder and filename will be referred from
Filename field.
Here we are using parameterized mapping to get the URL, bucket, secret key and access key.
Java mapping can be downloaded from below link,
link
Also pasting code here,
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import com.sap.aii.mapping.api.*;
import io.minio.BucketExistsArgs;
import io.minio.GetPresignedObjectUrlArgs;
import io.minio.MinioClient;
import io.minio.PutObjectArgs;
import io.minio.http.Method;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import com.sap.engine.interfaces.messaging.api.auditlog.*;
import com.sap.engine.interfaces.messaging.api.exception.MessagingException;
import com.sap.engine.interfaces.messaging.api.*;
public class GenericFileUploadtoMinio extends AbstractTransformation{
@Override
public void transform(TransformationInput transformationInput, TransformationOutput transformationOutput) throws StreamTransformationException {
MessageKey key = null;
AuditAccess audit = null;
final String DASH = "-";
String msgID=transformationInput.getInputHeader().getMessageId();
String uuidTimeLow = msgID.substring(0, 8);
String uuidTimeMid = msgID.substring(8, 12);
String uuidTimeHighAndVersion = msgID.substring(12, 16);
String uuidClockSeqAndReserved = msgID.substring(16, 18);
String uuidClockSeqLow = msgID.substring(18, 20);
String uuidNode = msgID.substring(20, 32);
String msgUUID = uuidTimeLow + DASH + uuidTimeMid + DASH + uuidTimeHighAndVersion + DASH + uuidClockSeqAndReserved + uuidClockSeqLow + DASH + uuidNode;
try {
audit = PublicAPIAccessFactory.getPublicAPIAccess().getAuditAccess();
} catch (MessagingException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
key = new MessageKey(msgUUID, MessageDirection.OUTBOUND);
try {
getTrace().addInfo("Java Mapping Program Started!");
String DocNumber = "";
String strFileName = "";
String strDirectory = "";
String dir = "";
String slash = "/";
String colon = ":";
//Get the handler for processing the attachments of source message
InputAttachments attachment = transformationInput.getInputAttachments();
String url = transformationInput.getInputParameters().getString("url");
String bucketName = transformationInput.getInputParameters().getString("bucketname");
String accesskey = transformationInput.getInputParameters().getString("accesskey");
String secretkey = transformationInput.getInputParameters().getString("secretkey");
String nodename = transformationInput.getInputParameters().getString("nodename");
String folderpath = transformationInput.getInputParameters().getString("folderpath");
String auth = "?accessKey="+accesskey+"&secretKey="+secretkey;
InputStream inputstream = transformationInput.getInputPayload().getInputStream();
OutputStream outputstream = transformationOutput.getOutputPayload().getOutputStream();
//an instance of factory that gives a document builder
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
//an instance of builder to parse the specified xml file
DocumentBuilder db = dbf.newDocumentBuilder();
Document doc = db.parse(inputstream);
doc.getDocumentElement().normalize();
getTrace().addInfo("XML Read Started");
NodeList list = doc.getElementsByTagName(nodename); //"n0:MT_ProxySender"
for (int temp = 0; temp < list.getLength(); temp++) {
Node node = list.item(temp);
if (node.getNodeType() == Node.ELEMENT_NODE) {
Element element = (Element) node;
// get value for RFQNumber
DocNumber = element.getElementsByTagName("DOCNO").item(0).getTextContent();
getTrace().addInfo("DOCNO READ" +DocNumber );
// get value for Filename
strFileName = element.getElementsByTagName("Filename").item(0).getTextContent();
getTrace().addInfo("File Name READ" +strFileName );
}
}
String ip = "INPUT";
String path = folderpath+slash+DocNumber+slash+strFileName; //"PO"; //DocNumber+slash+ip+slash+strFileName;
//Fetch the collection of attachments in the source message
for (String id : attachment.getAllContentIds(false)) {
//Read the content of the attachment and assign the content to output
outputstream.write((attachment.getAttachment(id)).getContent());
}
//Code to convert OutputStream to InputStream
ByteArrayOutputStream bos = (ByteArrayOutputStream)outputstream;
// ByteArrayInputStream inStream = new ByteArrayInputStream( bos.toByteArray());
InputStream input = new ByteArrayInputStream( bos.toByteArray());
// byte[] bytes = inStream.toString().getBytes(StandardCharsets.UTF_8);
// InputStream input = new ByteArrayInputStream(bytes);
audit.addAuditLogEntry(key, AuditLogStatus.SUCCESS, "AccessKey "+accesskey+" SecretKey "+secretkey);
MinioClient s3Client =
MinioClient.builder()
.endpoint(url)
.credentials(accesskey, secretkey)
.build();
boolean found =
s3Client.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());
if(found){
audit.addAuditLogEntry(key, AuditLogStatus.SUCCESS, "Bucket Found Successfully!!");
getTrace().addDebugMessage("Bucket " +bucketName+ " found Successfully");
s3Client.putObject(
PutObjectArgs.builder().bucket(bucketName).object(path).stream(
input, -1, 10485760)
.contentType("application/pdf")
.build());
audit.addAuditLogEntry(key, AuditLogStatus.SUCCESS, "200 - File Uploaded Successfully!!");
//GET Created File URL
Map<String, String> reqParams = new HashMap<String, String>();
reqParams.put("response-content-type", "application/json");
audit.addAuditLogEntry(key, AuditLogStatus.SUCCESS, "Path "+path);
String urlString = s3Client.getPresignedObjectUrl(GetPresignedObjectUrlArgs.builder()
.method(Method.GET)
.bucket(bucketName)
.object(path)
.expiry(1, TimeUnit.HOURS)
.extraQueryParams(reqParams)
.build());
audit.addAuditLogEntry(key, AuditLogStatus.SUCCESS, "FileURL "+urlString);
urlString = urlString.replace("%2F","/");
URL endpointUrl = new URL(urlString);
// Create Dynamic Configuration Object
DynamicConfiguration conf = transformationInput.getDynamicConfiguration();
String urlvar = endpointUrl.toString();
urlvar = urlvar.replace("%2F","/");
DynamicConfigurationKey key1 = DynamicConfigurationKey.create( "http:/"+"/sap.com/xi/XI/System/REST","endpoint");
conf.put(key1,urlvar);
}else {
getTrace().addDebugMessage("Bucket " +bucketName+ " not found");
}
}catch(Exception e) {
getTrace().addDebugMessage(e.getMessage());
audit.addAuditLogEntry(key, AuditLogStatus.ERROR, "Exception Raised: "+e.toString());
throw new StreamTransformationException(e.toString());
}
}
}
Here after file upload we are getting endpoint URL string where file can be located which will be passed as a dynamic parameter to receiver channel URL
The receiver channel config is as below,
Receiver channel_1
Receiver channel_2
Receiver channel_3
In the Download scenario, we created UDF where filenames were provided from source structure. Perform the below steps,
- Search for file in MINIO bucket
- If found, Extract file data
- Attach to receiver proxy to SAP
Mapping step,
Mapping of UDF
DownloadFileUDF
UDF Code is as below,
public String AttachFileFromMinIO(String bucketName, String accesskey, String secretkey, String url, String attachments, String tenderno, Container container) throws StreamTransformationException{
String[] arrOfStr = attachments.split(";");
String slash = "/";
String ip = "INPUT";
String op = "OUTPUT";
int DEFAULT_BUFFER_SIZE = 8192;
String attach = "";
String filename = "";
try{
getTrace().addInfo("-------------AttachFileFromMinIO Started-----------------");
// Write attachment
GlobalContainer globalContainer = container.getGlobalContainer();
OutputAttachments outputAttachments = globalContainer.getOutputAttachments();
MinioClient s3Client =
MinioClient.builder()
.endpoint(url)
.credentials(accesskey,secretkey)
.build();
OutputStream out = new ByteArrayOutputStream();
//getTrace().addInfo("Minio Bucket Found " +s3Client.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build()));
for(int i =0; i<=arrOfStr.length; i++)
{
try{
getTrace().addInfo("arrOfStr " +arrOfStr[i]);
attach = attach+","+arrOfStr[i];
filename = arrOfStr[i];
String path = tenderno+slash+op+slash+filename;
getTrace().addInfo("Filename " +filename);
try ( InputStream istream = s3Client.getObject(
GetObjectArgs.builder()
.bucket(bucketName)
.object(path)//"OUTPUT/MinIOTest.pdf"
.build())) {
// Read data from stream
getTrace().addInfo("File "+filename+" downloaded");
byte[] bytes = IOUtils.toByteArray(istream);
istream.close();
Attachment newAttachment = outputAttachments.create(filename,"application/pdf", bytes);
outputAttachments.setAttachment(newAttachment); }
}catch(Exception e){
getTrace().addInfo("Error while processing file " +filename);
}
}
}catch(Exception e){
getTrace().addInfo("Error while processing file");
}
return attachments;
}
DownloadFileOutput
We came across the blog
https://blogs.sap.com/2020/10/20/file-upload-in-aws-s3-using-rest-api/ which was using AWS S3 library but here I have referred to libraries provided by MinIO (
https://min.io/docs/minio/linux/developers/java/API.html)
I hope this blog is helpful for you all while integrating MinIO system with SAP.
If so, click on “like” or “share”. I’m looking forward to your feedback and thoughts.
Thank you.