Technology Blogs by Members
Explore a vibrant mix of technical expertise, industry insights, and tech buzz in member blogs covering SAP products, technology, and events. Get in the mix!
cancel
Showing results for 
Search instead for 
Did you mean: 
dilipkkp2412
Contributor

Business requirement:


1. Pull files from source folder (FTP or SFTP) folder and push it specific Azure-Blob-Storage folder location.
2. Read/Pull files from Azure-Blob-Storage folder locations

Technique-01:


Using PI/PO interface (FILE/SFTP TO FILE | Asynchronous) where Java Mapping is been used to push the files/Content into Azure-Blob-Storage location.
Here, The Sender File or SFTP Adapter communication channel is been used to pull the files from source location.
With help of Java Mapping/Program technique, receievd File contents/Header input stream is been sent to Azure-Blob-Storage location following all repspective Azure-Connection formats.
The Receiver File Adapter Communication Channel is been used to just complete the scenario configuration, it will keep creating/overwriting one dummy file (with dummy file-name) into the source-folder's Archive location.

Contents of the Java map program:



  1. Jar file 'aii_map_api.jar' | No need to import this into 'Enterprise Service Repository (ESR)'

  2. Jar file 'azure-storage-8.6.6.jar' | Import this in 'ESR' as an 'Imported Archive' under desired new interface namespace.

  3. Jar file 'jackson-core-2.9.4.jar' | Import this in 'ESR' as an 'Imported Archive' under desired new interface namespace.

  4. XML file 'credentialsAzureBlob.xml' | Import this into Java-Program as resource file| This has Azure-Blob-Storage credentials (Test / Production both) in below XML structure manner


<?xml version="1.0" encoding="UTF-8"?>
<AzureBlobCredentials>
<AzBlbCrd_TEST>
<AzBlb_DefaultEndpointsProtocol>https</AzBlb_DefaultEndpointsProtocol>
<AzBlb_AccountName>BlbAcNm</AzBlb_AccountName>
<AzBlb_AccountKey>BlbAcKey</AzBlb_AccountKey>
<AzBlb_EndpointSuffix>core.windows.net</AzBlb_EndpointSuffix>
<AzBlb_ContainerName1>BlobContainerName/Fldr1</AzBlb_ContainerName1>
<AzBlb_ContainerName2_Dynamic></AzBlb_ContainerName2_Dynamic>
<AzBlb_ContainerName3></AzBlb_ContainerName3>
</AzBlbCrd_TEST>
<AzBlbCrd_PRODUCTION>
<AzBlb_DefaultEndpointsProtocol>https</AzBlb_DefaultEndpointsProtocol>
<AzBlb_AccountName>BlbAcNm</AzBlb_AccountName>
<AzBlb_AccountKey>BlbAcKey</AzBlb_AccountKey>
<AzBlb_EndpointSuffix>core.windows.net</AzBlb_EndpointSuffix>
<AzBlb_ContainerName1>BlobContainerName/Fldr1/Fldr2/</AzBlb_ContainerName1>
<AzBlb_ContainerName2_Dynamic>DynamicDate_yyyy-MM-dd</AzBlb_ContainerName2_Dynamic>
<AzBlb_ContainerName3>Fldr3</AzBlb_ContainerName3>
</AzBlbCrd_PRODUCTION>
</AzureBlobCredentials>


  1. Jar File 'PushFileToAzrBlob.jar' | Import this in 'ESR' as an 'Imported Archive' under desired new interface namespace. | This is the Java-Map jar file, this will be only used in 'Operation Map' |


In Eclipse too, above jar files and xml file need to be imported like below:


Eclipse Structure of Java-Map Program



Java-Map program sample is as below:


This java program gives overview about below features:

  1. How to read attached resource file ('credentialsAzureBlob.xml') in Java-Program project and  based on PI/PO-system-ID instances, how to select the Test or Production credentials

  2. How to read the source File-Name from Dynamic-Configuration Variable (the file which is picked by Sender File CC)

  3. How to push the files into Azure-Blob-Storage

  4. How to Push the files into Azure-Blob-Storage (Dynamic Folder creation)

  5. How to pull/read files from Azure-Blob-Storage

  6. Stepwise capturing the Map-Log, printing it to map trace and resulting it as XML output stream xml while hits to File-Receiver CC



import java.io.BufferedReader;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.StringReader;
import java.net.URI;
import java.nio.charset.Charset;
import java.sql.Timestamp;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;

import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;

import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;

import com.sap.aii.mapping.api.AbstractTrace;
import com.sap.aii.mapping.api.StreamTransformation;
import com.sap.aii.mapping.api.StreamTransformationConstants;
import com.sap.aii.mapping.api.StreamTransformationException;
import com.sap.aii.mapping.api.DynamicConfiguration;
import com.sap.aii.mapping.api.DynamicConfigurationKey;

import com.microsoft.azure.storage.CloudStorageAccount;
import com.microsoft.azure.storage.blob.BlobProperties;
import com.microsoft.azure.storage.blob.CloudBlob;
import com.microsoft.azure.storage.blob.CloudBlobClient;
import com.microsoft.azure.storage.blob.CloudBlobContainer;

public class PushFileToAzrBlob implements StreamTransformation{

private Map param;
public void setParameter(Map map) {
param = map;
if (param == null) {
param = new HashMap();
}
}

private static AbstractTrace trace = null;
private static String mapTrace = "";
public void execute(InputStream in, OutputStream out) throws StreamTransformationException {
try {
mapTrace = "";
mapTrace = mapTrace + "=> Begin of JavaMap 'PushFileToAzrBlob()' | JavaMap to send File/String into Azure-Blob-Storage <=" + "\n" ;

mapTrace = mapTrace + "=> 1. Fetching the 'AzureBlobStorage-credentials' " + "\n" ;
readRscFile_AzBlbCrd();

mapTrace = mapTrace + "=> 2. Reading the 'FileName' from ASMA Property (DynamicConfiguration Variable) of Sender-FILE-Comm.Channel " + "\n" ;

//Get DatTime stamp to be appended in FileName
Date date = new Date();
Timestamp Ctimestamp2 = new Timestamp(date.getTime());
SimpleDateFormat sdf2 = new SimpleDateFormat("yyyyMMdd_HHmmss");
String sdf2Str = sdf2.format(Ctimestamp2);

String sapFileName = "FILE_"+ sdf2Str + ".csv";
try {

//Get FileName from ASMA variable
DynamicConfiguration conf = (DynamicConfiguration)param.get("DynamicConfiguration");
DynamicConfigurationKey key = DynamicConfigurationKey.create("http://sap.com/xi/XI/System/File","FileName");
sapFileName = conf.get(key); //to get the value from the 'key'

//Append DateTime stamp in FileName
int i = sapFileName.lastIndexOf(".");
if(i > -1){
String fNm1 = sapFileName.substring(0, i);
String fNm2 = sapFileName.substring(i);
sapFileName = fNm1 + "_" + sdf2Str + fNm2;
}else{
sapFileName = sapFileName + "_" + sdf2Str;
}

}catch(Exception e){
mapTrace = mapTrace + "==> Exception in FM 'execute()' while getting FileName from ASMA ==> " + e.getMessage() + "\n";
}

mapTrace = mapTrace + "=> FileName: " + sapFileName + "\n" ;
mapTrace = mapTrace + "=> 3. Uploading the file into Azure-Blob-Storage" + "\n" ;
writeFile_toAzBlbStrg(sapFileName, in);

mapTrace = mapTrace + "=> End of JavaMap 'PushFileToAzrBlob()' | JavaMap to send File/String into Azure-Blob-Storage <=" + "\n" ;
String outputXML = "<?xml version=\"1.0\"?>"
+ "<Response>" + "\n"
+ "<JavaMap_Trace><![CDATA[" + mapTrace + "]]></JavaMap_Trace>" + "\n"
+ "</Response>";
Document docOut = convertStringToDocument(outputXML);
TransformerFactory tf = TransformerFactory.newInstance();
Transformer transform = tf.newTransformer();
transform.transform(new DOMSource(docOut), new StreamResult(out));

trace = (AbstractTrace)param.get(StreamTransformationConstants.MAPPING_TRACE);
trace.addInfo(mapTrace);
//System.out.println(mapTrace);
}catch(Exception e){
trace.addInfo("==> Exception in FM execute(): " + e.getMessage());
//System.out.println("==> Exception in FM execute(): " + e.getMessage());
}
}

private static Document convertStringToDocument(String xmlStr) {
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder;
try
{
builder = factory.newDocumentBuilder();
Document doc = builder.parse( new InputSource( new StringReader( xmlStr ) ) );
return doc;
} catch (Exception e) {
e.printStackTrace();
}
return null;
}

private static String AzBlb_ContainerName1;
private static String AzBlb_ContainerName2;
private static String AzBlb_ContainerName3;
private static String AzBlb_DefaultEndpointsProtocol;
private static String AzBlb_AccountName;
private static String AzBlb_AccountKey;
private static String AzBlb_EndpointSuffix;

private void readRscFile_AzBlbCrd() throws IOException, ParserConfigurationException, SAXException{

/**Based on SAPPO-SystemIDs, fetching the TEST or Production AzureBlobStorage-credentials
* Here, XML-Element /AzureBlobCredentials/AzBlbCrd_TEST/ stores Test-System Credentials
* and XML-Element /AzureBlobCredentials/AzBlbCrd_PRODUCTION/ stores Production-System Credentials
*/
String systemID = "";
try {
//systemID = "<sysID-DEV>"; //For Eclipse-Platform-Testing
systemID = (String) System.getProperty("SAPSYSTEMNAME"); //Get the systemID of PI/PO system (DEV/TEST/QAS)
if(systemID.equals("<sysID-DEV>") || systemID.equals("<sysID-QAS>")){
readRscFile_AzBlbCrd_Elements("AzBlbCrd_TEST");
}
if(systemID.equals("<sysID-PRD>")){
readRscFile_AzBlbCrd_Elements("AzBlbCrd_PRODUCTION");
}
}catch (Exception e) {
mapTrace = mapTrace + "==> Exception in FM 'readRscFile_AzBlbCrd()'==> " + e.getMessage() + "\n";
}
}

private void readRscFile_AzBlbCrd_Elements(String XmlElementTagNm) throws IOException, ParserConfigurationException, SAXException{

//Clear Global variables of this program
AzBlb_ContainerName1 = "";
AzBlb_ContainerName2 = "";
AzBlb_ContainerName3 = "";
AzBlb_DefaultEndpointsProtocol = "";
AzBlb_AccountName = "";
AzBlb_AccountKey = "";
AzBlb_EndpointSuffix = "";

//Read AzureBlob-Credentials stored in JavaMap-RsourceFile 'credentialsAzureBlob.xml.xml'
String resFileName = "credentialsAzureBlob.xml";

//Parse the file into document
DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder dbBuilder = dbFactory.newDocumentBuilder();
InputStream is_ShpCrd = getClass().getResourceAsStream(resFileName);
Document xmlDocCrd = dbBuilder.parse(is_ShpCrd); //Parse input to create document tree

//----Start: Read AzureBlob Credentials from XmlFile ------
NodeList ndList_shp = xmlDocCrd.getElementsByTagName(XmlElementTagNm);
for(int i1 = 0; i1 < ndList_shp.getLength(); i1++){
Node nd_1 = ndList_shp.item(i1);
for(Node nd_2 = nd_1.getFirstChild(); nd_2 != null; nd_2 = nd_2.getNextSibling()){
if(nd_2.getNodeName().equals("AzBlb_DefaultEndpointsProtocol")){
if(nd_2.getFirstChild() != null){
AzBlb_DefaultEndpointsProtocol = nd_2.getFirstChild().getNodeValue();
}
}
if(nd_2.getNodeName().equals("AzBlb_AccountName")){
if(nd_2.getFirstChild() != null){
AzBlb_AccountName = nd_2.getFirstChild().getNodeValue();
}
}
if(nd_2.getNodeName().equals("AzBlb_AccountKey")){
if(nd_2.getFirstChild() != null){
AzBlb_AccountKey = nd_2.getFirstChild().getNodeValue();
}
}
if(nd_2.getNodeName().equals("AzBlb_EndpointSuffix")){
if(nd_2.getFirstChild() != null){
AzBlb_EndpointSuffix = nd_2.getFirstChild().getNodeValue();
}
}
if(nd_2.getNodeName().equals("AzBlb_ContainerName1")){
if(nd_2.getFirstChild() != null){
AzBlb_ContainerName1 = nd_2.getFirstChild().getNodeValue();
}
}
if(nd_2.getNodeName().equals("AzBlb_ContainerName2_Dynamic")){
if(nd_2.getFirstChild() != null){
AzBlb_ContainerName2 = nd_2.getFirstChild().getNodeValue();
}
}
if(nd_2.getNodeName().equals("AzBlb_ContainerName3")){
if(nd_2.getFirstChild() != null){
AzBlb_ContainerName3 = nd_2.getFirstChild().getNodeValue();
}
}
}
}
//----End : Read AzureBlob Credentials from XmlFile ------
mapTrace = mapTrace + "=> The credentials has been read from the JavaMap-RsourceFile '"+ resFileName + "'\n";
}

private static void writeFile_toAzBlbStrg(String sapFileName, InputStream fileIpstrm) throws Exception {
try {
/**
* Input Parameters:
* 1. Filename : Get the FileName using DynamicConfiguration from FileSenderChannel-ASMA
* 2. FileContent : Get the FileContent inputStream read by FileSenderChannel
* 3. AzureBlob-Storage Connection String
* 4. AzureBlob-Storage Container Name (for e.g. xyz )
*
* In Azure-Blob, Dynamic folder creation is in scope,
* where, in between, folder name should have crrentDate with yyy-MM-dd
* For example:
* Below is the path reference from Credential file:
<AzBlb_ContainerName1>BlobContainerName/Fldr1/Fldr2/
<AzBlb_ContainerName2>DynamicDate_yyyy-MM-dd
<AzBlb_ContainerName3>Fldr3
*Then final ContainerName should be /BlobContainerName/Fldr1/Fldr2/2019-06-10/Fldr3
* Here, AzBlb_ContainerName2 suggest to have current Date in format yyyy-MM-dd.

* Note: Testing-Feedback:
* During testing it's found that, 1st time, in, Azure-Blob, folder gets created having currentDate.
* On next run on same date, automatically, in same folder, file gets dropped.
* There is no error found while re-run on same date for sending 2nd file having same folder name.
*/

//Removing Leading & Training slash '/'
AzBlb_ContainerName1 = AzBlb_ContainerName1.replaceAll("^/+", "").replaceAll("/+$", "");
AzBlb_ContainerName2 = AzBlb_ContainerName2.replaceAll("^/+", "").replaceAll("/+$", "");
AzBlb_ContainerName3 = AzBlb_ContainerName3.replaceAll("^/+", "").replaceAll("/+$", "");

String AzBlb_ContainerName = "";
//Get Current Date | to be used in FileName and Blob-FolderName
if(AzBlb_ContainerName2.startsWith("DynamicDate_")) {

//Get required DateFormat to be included in Blob-Folder-path
String dtFormat = AzBlb_ContainerName2.substring(AzBlb_ContainerName2.indexOf("_")+1);
Date date = new Date();
Timestamp Ctimestamp = new Timestamp(date.getTime());
SimpleDateFormat sdf = new SimpleDateFormat(dtFormat);
String currentDateStr = sdf.format(Ctimestamp); //This will be used in AzureBlob-Storage's Dynamic Path creation

//Make Azure-Blob-Storage-Folder Path Name
AzBlb_ContainerName = AzBlb_ContainerName1 + "/" + currentDateStr +"/" + AzBlb_ContainerName3;
AzBlb_ContainerName = AzBlb_ContainerName.replaceAll("^/+", "").replaceAll("/+$", ""); //Removing Leading & Training slash '/'

}else {
AzBlb_ContainerName = AzBlb_ContainerName1 + "/" + AzBlb_ContainerName2 +"/" + AzBlb_ContainerName3;
AzBlb_ContainerName = AzBlb_ContainerName.replaceAll("^/+", "").replaceAll("/+$", ""); //Removing Leading & Training slash '/'
}

String storageConnectionString =
"DefaultEndpointsProtocol=" + AzBlb_DefaultEndpointsProtocol +
";AccountName=" + AzBlb_AccountName +
";AccountKey=" + AzBlb_AccountKey +
";EndpointSuffix=" + AzBlb_EndpointSuffix;

CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
CloudBlobContainer container = blobClient.getContainerReference(AzBlb_ContainerName);
CloudBlob blob = container.getBlockBlobReference(sapFileName);

//Set Header
BlobProperties props = blob.getProperties();
props.setContentType("text/plain");

//Upload the blob using file's inputStream and size"
blob.upload(fileIpstrm, fileIpstrm.available());
mapTrace = mapTrace + "=> The file '"+ sapFileName +"' has been successfully uploaded into Azure-Blob-Storage '" + AzBlb_AccountName + "/" + AzBlb_ContainerName +"'\n";

/*
blob.startCopy(new URI("http://www.bing.com")); //Upload the blob using file-URL
blob.uploadFromFile(sapFilePath + sapFileName); //Upload the blob using file
blockBlobClient.upload(file.getInputStream(), file.getSize()); //Upload the blob using File inputStream
BlobClient blobClient = containerClient.getBlobClient(fileName); //Get a reference to a blob
System.out.println("\nUploading to Blob storage as blob:\n\t" + blobClient.getBlobUrl());
*/

} catch (Exception e) {
mapTrace = mapTrace + "==> Exception in FM 'writeFile_toAzBlbStrg()'==> " + e.getMessage() + "\n";
}
}

private static void readFile_frmAzBlbStrg(String fName) throws Exception {
/**
* Read file from Azure-Blob-Storage
*/

try {
String storageConnectionString = "DefaultEndpointsProtocol=https;"
+ "AccountName=<BlbAcNm>;"
+ "AccountKey=<BlbAcKey>"
+ "EndpointSuffix=core.windows.net";
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
CloudBlobContainer container = blobClient.getContainerReference("BlobContainerName");
CloudBlob blob = container.getBlockBlobReference(fName);
InputStream input = blob.openInputStream();

// Read the File-Content
InputStreamReader is = new InputStreamReader(input, "UTF-8");
BufferedReader br = new BufferedReader(is);
String read = null;
StringBuffer sb = new StringBuffer();
while ((read = br.readLine()) != null) {
sb.append(read);
}
String utf8str = sb.toString();

System.out.println("Reading......\n" + utf8str);
System.out.println("File read successfully");
} catch (Exception e) {
System.out.println("Error: " + e.getMessage());
}
}
/*
public static void main(String[] args) {
try {
String filePath = "C:\\Users\\Test\\";
String Fnm = "OP_Test2.csv";
String inputFile = filePath + Fnm;
String outputFile = filePath + "OP_" + Fnm ;

PushFileToAzrBlob myClass = new PushFileToAzrBlob();
FileInputStream in = new FileInputStream(inputFile);
FileOutputStream out = new FileOutputStream(outputFile);
myClass.execute(in, out);
}catch(Exception e){
e.printStackTrace();
}
}*/
}

 

For reference, below is the map trace log (Map output):


Map output (map trace log)



Eclipse console map trace log


 

File gets received in Azure-Blob as below:


Azure-Blob receives file


 

I hope this helps to interested audience. Technique-2 coming soon.....
Labels in this area