Technology Blogs by Members
Explore a vibrant mix of technical expertise, industry insights, and tech buzz in member blogs covering SAP products, technology, and events. Get in the mix!
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Push file into Azure Blob Storage using PI/PO

dilipkkp2412
Contributor
‎2022 Sep 27 8:16 AM
4,154

Business requirement:


1. Pull files from source folder (FTP or SFTP) folder and push it specific Azure-Blob-Storage folder location.
2. Read/Pull files from Azure-Blob-Storage folder locations

Technique-01:


Using PI/PO interface (FILE/SFTP TO FILE | Asynchronous) where Java Mapping is been used to push the files/Content into Azure-Blob-Storage location.
Here, The Sender File or SFTP Adapter communication channel is been used to pull the files from source location.
With help of Java Mapping/Program technique, receievd File contents/Header input stream is been sent to Azure-Blob-Storage location following all repspective Azure-Connection formats.
The Receiver File Adapter Communication Channel is been used to just complete the scenario configuration, it will keep creating/overwriting one dummy file (with dummy file-name) into the source-folder's Archive location.

Contents of the Java map program:



  1. Jar file 'aii_map_api.jar' | No need to import this into 'Enterprise Service Repository (ESR)'

  2. Jar file 'azure-storage-8.6.6.jar' | Import this in 'ESR' as an 'Imported Archive' under desired new interface namespace.

  3. Jar file 'jackson-core-2.9.4.jar' | Import this in 'ESR' as an 'Imported Archive' under desired new interface namespace.

  4. XML file 'credentialsAzureBlob.xml' | Import this into Java-Program as resource file| This has Azure-Blob-Storage credentials (Test / Production both) in below XML structure manner


<?xml version="1.0" encoding="UTF-8"?>

<AzureBlobCredentials>

	<AzBlbCrd_TEST>

		<AzBlb_DefaultEndpointsProtocol>https</AzBlb_DefaultEndpointsProtocol>

		<AzBlb_AccountName>BlbAcNm</AzBlb_AccountName>

		<AzBlb_AccountKey>BlbAcKey</AzBlb_AccountKey>

		<AzBlb_EndpointSuffix>core.windows.net</AzBlb_EndpointSuffix>	

		<AzBlb_ContainerName1>BlobContainerName/Fldr1</AzBlb_ContainerName1>

		<AzBlb_ContainerName2_Dynamic></AzBlb_ContainerName2_Dynamic>

		<AzBlb_ContainerName3></AzBlb_ContainerName3>

	</AzBlbCrd_TEST>

	<AzBlbCrd_PRODUCTION>

		<AzBlb_DefaultEndpointsProtocol>https</AzBlb_DefaultEndpointsProtocol>

		<AzBlb_AccountName>BlbAcNm</AzBlb_AccountName>

		<AzBlb_AccountKey>BlbAcKey</AzBlb_AccountKey>

		<AzBlb_EndpointSuffix>core.windows.net</AzBlb_EndpointSuffix>	

		<AzBlb_ContainerName1>BlobContainerName/Fldr1/Fldr2/</AzBlb_ContainerName1>

		<AzBlb_ContainerName2_Dynamic>DynamicDate_yyyy-MM-dd</AzBlb_ContainerName2_Dynamic>

		<AzBlb_ContainerName3>Fldr3</AzBlb_ContainerName3>

	</AzBlbCrd_PRODUCTION>

</AzureBlobCredentials>


  1. Jar File 'PushFileToAzrBlob.jar' | Import this in 'ESR' as an 'Imported Archive' under desired new interface namespace. | This is the Java-Map jar file, this will be only used in 'Operation Map' |


In Eclipse too, above jar files and xml file need to be imported like below:


Eclipse Structure of Java-Map Program



Java-Map program sample is as below:


This java program gives overview about below features:

  1. How to read attached resource file ('credentialsAzureBlob.xml') in Java-Program project and  based on PI/PO-system-ID instances, how to select the Test or Production credentials

  2. How to read the source File-Name from Dynamic-Configuration Variable (the file which is picked by Sender File CC)

  3. How to push the files into Azure-Blob-Storage

  4. How to Push the files into Azure-Blob-Storage (Dynamic Folder creation)

  5. How to pull/read files from Azure-Blob-Storage

  6. Stepwise capturing the Map-Log, printing it to map trace and resulting it as XML output stream xml while hits to File-Receiver CC




import java.io.BufferedReader;

import java.io.ByteArrayOutputStream;

import java.io.File;

import java.io.FileInputStream;

import java.io.FileOutputStream;

import java.io.FileWriter;

import java.io.IOException;

import java.io.InputStream;

import java.io.InputStreamReader;

import java.io.OutputStream;

import java.io.StringReader;

import java.net.URI;

import java.nio.charset.Charset;

import java.sql.Timestamp;

import java.text.SimpleDateFormat;

import java.util.Date;

import java.util.HashMap;

import java.util.Map;



import javax.xml.parsers.DocumentBuilder;

import javax.xml.parsers.DocumentBuilderFactory;

import javax.xml.parsers.ParserConfigurationException;

import javax.xml.transform.Transformer;

import javax.xml.transform.TransformerFactory;

import javax.xml.transform.dom.DOMSource;

import javax.xml.transform.stream.StreamResult;



import org.w3c.dom.Document;

import org.w3c.dom.Node;

import org.w3c.dom.NodeList;

import org.xml.sax.InputSource;

import org.xml.sax.SAXException;



import com.sap.aii.mapping.api.AbstractTrace;

import com.sap.aii.mapping.api.StreamTransformation;

import com.sap.aii.mapping.api.StreamTransformationConstants;

import com.sap.aii.mapping.api.StreamTransformationException;

import com.sap.aii.mapping.api.DynamicConfiguration;

import com.sap.aii.mapping.api.DynamicConfigurationKey;



import com.microsoft.azure.storage.CloudStorageAccount;

import com.microsoft.azure.storage.blob.BlobProperties;

import com.microsoft.azure.storage.blob.CloudBlob;

import com.microsoft.azure.storage.blob.CloudBlobClient;

import com.microsoft.azure.storage.blob.CloudBlobContainer;



public class PushFileToAzrBlob implements StreamTransformation{

		

	private Map param;

	public void setParameter(Map map) {

		param = map;

		if (param == null) {

			param = new HashMap();

		}

	}

	

	private static AbstractTrace trace = null;

	private static String mapTrace = "";

	public void execute(InputStream in, OutputStream out) throws StreamTransformationException {

		try {

			mapTrace = "";			

			mapTrace = mapTrace + "=> Begin of JavaMap 'PushFileToAzrBlob()' | JavaMap to send File/String into Azure-Blob-Storage <=" + "\n" ;	

								

			mapTrace = mapTrace + "=> 1. Fetching the 'AzureBlobStorage-credentials' " + "\n" ;

			readRscFile_AzBlbCrd();

					

			mapTrace = mapTrace + "=> 2. Reading the 'FileName' from ASMA Property (DynamicConfiguration Variable) of Sender-FILE-Comm.Channel " + "\n" ;				

					

			//Get DatTime stamp to be appended in FileName

			Date date = new Date();

			Timestamp Ctimestamp2 = new Timestamp(date.getTime());			

			SimpleDateFormat sdf2 = new SimpleDateFormat("yyyyMMdd_HHmmss");			

			String sdf2Str = sdf2.format(Ctimestamp2);

			

			String sapFileName = "FILE_"+ sdf2Str + ".csv";	

			try {

				

				//Get FileName from ASMA variable

				DynamicConfiguration conf = (DynamicConfiguration)param.get("DynamicConfiguration");

				DynamicConfigurationKey key = DynamicConfigurationKey.create("http://sap.com/xi/XI/System/File","FileName");

				sapFileName = conf.get(key);	//to get the value from the 'key'		

				

				//Append DateTime stamp in FileName

			    int i = sapFileName.lastIndexOf(".");

			    if(i > -1){

			     String fNm1 = sapFileName.substring(0, i);

			     String fNm2 = sapFileName.substring(i);

			     sapFileName = fNm1 + "_" + sdf2Str + fNm2;     

			    }else{

			     sapFileName = sapFileName + "_" + sdf2Str;     

			    }

				

			}catch(Exception e){		

				mapTrace = mapTrace + "==>   Exception in FM 'execute()' while getting FileName from ASMA ==> " + e.getMessage() + "\n";

			}			

			

			mapTrace = mapTrace + "=>    FileName: " + sapFileName + "\n" ;				

			mapTrace = mapTrace + "=> 3. Uploading the file into Azure-Blob-Storage" + "\n" ;	

			writeFile_toAzBlbStrg(sapFileName, in);

							

			mapTrace = mapTrace + "=> End   of JavaMap 'PushFileToAzrBlob()' | JavaMap to send File/String into Azure-Blob-Storage <=" + "\n" ;							

			String outputXML = "<?xml version=\"1.0\"?>"

					+ "<Response>" + "\n" 

					+ "<JavaMap_Trace><![CDATA[" + mapTrace	+ "]]></JavaMap_Trace>"	+ "\n" 		

					+ "</Response>";

			Document docOut = convertStringToDocument(outputXML);

			TransformerFactory tf = TransformerFactory.newInstance();

			Transformer transform = tf.newTransformer();		

			transform.transform(new DOMSource(docOut), new StreamResult(out));

					

			trace = (AbstractTrace)param.get(StreamTransformationConstants.MAPPING_TRACE);

			trace.addInfo(mapTrace);

			//System.out.println(mapTrace);							

		}catch(Exception e){

			trace.addInfo("==> Exception in FM execute(): " + e.getMessage());

			//System.out.println("==> Exception in FM execute(): " + e.getMessage());			

		}

	}

	

    private static Document convertStringToDocument(String xmlStr) {

        DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();  

        DocumentBuilder builder;  

        try  

        {  

            builder = factory.newDocumentBuilder();  

            Document doc = builder.parse( new InputSource( new StringReader( xmlStr ) ) ); 

            return doc;

        } catch (Exception e) {  

            e.printStackTrace();  

        } 

        return null;

    }

	

	private static String AzBlb_ContainerName1;

	private static String AzBlb_ContainerName2;

	private static String AzBlb_ContainerName3;

	private static String AzBlb_DefaultEndpointsProtocol;

	private static String AzBlb_AccountName;

	private static String AzBlb_AccountKey;

	private static String AzBlb_EndpointSuffix;

		

	private void readRscFile_AzBlbCrd() throws IOException, ParserConfigurationException, SAXException{

				

		/**Based on SAPPO-SystemIDs, fetching the TEST or Production AzureBlobStorage-credentials

		 * Here, XML-Element /AzureBlobCredentials/AzBlbCrd_TEST/ 			stores Test-System Credentials

		 * and 	 XML-Element /AzureBlobCredentials/AzBlbCrd_PRODUCTION/ 	stores Production-System Credentials

		 */

		String systemID = "";	

		try {	

			//systemID = "<sysID-DEV>"; //For Eclipse-Platform-Testing

			systemID = (String) System.getProperty("SAPSYSTEMNAME");	//Get the systemID of PI/PO system (DEV/TEST/QAS)

			if(systemID.equals("<sysID-DEV>") || systemID.equals("<sysID-QAS>")){					

				readRscFile_AzBlbCrd_Elements("AzBlbCrd_TEST");

			}

			if(systemID.equals("<sysID-PRD>")){				

				readRscFile_AzBlbCrd_Elements("AzBlbCrd_PRODUCTION");

			}

		}catch (Exception e) {			

			mapTrace = mapTrace + "==>   Exception in FM 'readRscFile_AzBlbCrd()'==> " + e.getMessage() + "\n";			

		}	

	}

	

	private void readRscFile_AzBlbCrd_Elements(String XmlElementTagNm) throws IOException, ParserConfigurationException, SAXException{

							

		//Clear Global variables of this program			

		AzBlb_ContainerName1 			= "";

		AzBlb_ContainerName2 			= "";

		AzBlb_ContainerName3 			= "";	

		AzBlb_DefaultEndpointsProtocol 	= "";

		AzBlb_AccountName				= "";

		AzBlb_AccountKey				= "";

		AzBlb_EndpointSuffix			= "";

		

		//Read AzureBlob-Credentials stored in JavaMap-RsourceFile 'credentialsAzureBlob.xml.xml'			

		String resFileName = "credentialsAzureBlob.xml";	

		

		//Parse the file into document

    	DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();

 		DocumentBuilder dbBuilder = dbFactory.newDocumentBuilder();

 		InputStream is_ShpCrd = getClass().getResourceAsStream(resFileName);

		Document xmlDocCrd = dbBuilder.parse(is_ShpCrd);	//Parse input to create document tree

		

		//----Start: Read AzureBlob Credentials from XmlFile ------	

		NodeList ndList_shp = xmlDocCrd.getElementsByTagName(XmlElementTagNm);

		for(int i1 = 0; i1 < ndList_shp.getLength(); i1++){

			Node nd_1 = ndList_shp.item(i1);

			for(Node nd_2 = nd_1.getFirstChild(); nd_2 != null; nd_2 = nd_2.getNextSibling()){

				if(nd_2.getNodeName().equals("AzBlb_DefaultEndpointsProtocol")){				

					if(nd_2.getFirstChild() != null){	

						AzBlb_DefaultEndpointsProtocol  = nd_2.getFirstChild().getNodeValue();						

					}

				}

				if(nd_2.getNodeName().equals("AzBlb_AccountName")){			

					if(nd_2.getFirstChild() != null){	

						AzBlb_AccountName  = nd_2.getFirstChild().getNodeValue();						

					}

				}

				if(nd_2.getNodeName().equals("AzBlb_AccountKey")){					

					if(nd_2.getFirstChild() != null){	

						AzBlb_AccountKey  = nd_2.getFirstChild().getNodeValue();						

					}

				}

				if(nd_2.getNodeName().equals("AzBlb_EndpointSuffix")){

					if(nd_2.getFirstChild() != null){	

						AzBlb_EndpointSuffix  = nd_2.getFirstChild().getNodeValue();						

					}

				}

				if(nd_2.getNodeName().equals("AzBlb_ContainerName1")){					

					if(nd_2.getFirstChild() != null){	

						AzBlb_ContainerName1  = nd_2.getFirstChild().getNodeValue();						

					}

				}

				if(nd_2.getNodeName().equals("AzBlb_ContainerName2_Dynamic")){					

					if(nd_2.getFirstChild() != null){	

						AzBlb_ContainerName2  = nd_2.getFirstChild().getNodeValue();						

					}

				}

				if(nd_2.getNodeName().equals("AzBlb_ContainerName3")){					

					if(nd_2.getFirstChild() != null){	

						AzBlb_ContainerName3  = nd_2.getFirstChild().getNodeValue();						

					}

				}

			}						

		}

		//----End  : Read AzureBlob Credentials from XmlFile ------	

		mapTrace = mapTrace + "=>    The credentials has been read from the JavaMap-RsourceFile '"+ resFileName + "'\n";				

	}

		

	private static void writeFile_toAzBlbStrg(String sapFileName, InputStream fileIpstrm) throws Exception {

		try {		

			/**

			 * Input Parameters:

			 * 1. Filename 		: Get the FileName using DynamicConfiguration from FileSenderChannel-ASMA

			 * 2. FileContent	: Get the FileContent inputStream read by FileSenderChannel

			 * 3. AzureBlob-Storage Connection String

			 * 4. AzureBlob-Storage Container Name (for e.g. xyz )

			 * 

			 * In Azure-Blob, Dynamic folder creation is in scope, 

			 * where, in between, folder name should have crrentDate with yyy-MM-dd

			 * For example: 

			 * Below is the path reference from Credential file:

			 	<AzBlb_ContainerName1>BlobContainerName/Fldr1/Fldr2/

				<AzBlb_ContainerName2>DynamicDate_yyyy-MM-dd

				<AzBlb_ContainerName3>Fldr3	

			 *Then final ContainerName should be /BlobContainerName/Fldr1/Fldr2/2019-06-10/Fldr3	

			 * Here, AzBlb_ContainerName2 suggest to have current Date in format yyyy-MM-dd.

			 			 

			 * Note: Testing-Feedback:

			 * During testing it's found that, 1st time, in, Azure-Blob, folder gets created having currentDate.

			 * On next run on same date, automatically, in same folder, file gets dropped. 

			 * There is no error found while re-run on same date for sending 2nd file having same folder name.

			 */

						

			//Removing Leading & Training slash '/'			 

			AzBlb_ContainerName1 = AzBlb_ContainerName1.replaceAll("^/+", "").replaceAll("/+$", "");

			AzBlb_ContainerName2 = AzBlb_ContainerName2.replaceAll("^/+", "").replaceAll("/+$", "");

			AzBlb_ContainerName3 = AzBlb_ContainerName3.replaceAll("^/+", "").replaceAll("/+$", "");

			

			String AzBlb_ContainerName = "";

			//Get Current Date | to be used in FileName and Blob-FolderName

			if(AzBlb_ContainerName2.startsWith("DynamicDate_")) {	

							

				//Get required DateFormat to be included in Blob-Folder-path

				String dtFormat = AzBlb_ContainerName2.substring(AzBlb_ContainerName2.indexOf("_")+1);				

				Date date = new Date();

				Timestamp Ctimestamp = new Timestamp(date.getTime());

				SimpleDateFormat sdf = new SimpleDateFormat(dtFormat);

				String currentDateStr =  sdf.format(Ctimestamp);	//This will be used in AzureBlob-Storage's Dynamic Path creation	

			

				//Make Azure-Blob-Storage-Folder Path Name

				AzBlb_ContainerName = AzBlb_ContainerName1 + "/" + currentDateStr +"/" + AzBlb_ContainerName3;	

				AzBlb_ContainerName = AzBlb_ContainerName.replaceAll("^/+", "").replaceAll("/+$", ""); //Removing Leading & Training slash '/'

				

			}else {

				AzBlb_ContainerName = AzBlb_ContainerName1 + "/" + AzBlb_ContainerName2 +"/" + AzBlb_ContainerName3;

				AzBlb_ContainerName = AzBlb_ContainerName.replaceAll("^/+", "").replaceAll("/+$", "");  //Removing Leading & Training slash '/'

			}

				

			String storageConnectionString = 

					"DefaultEndpointsProtocol=" + AzBlb_DefaultEndpointsProtocol +

					";AccountName=" 			+ AzBlb_AccountName +

					";AccountKey=" 				+ AzBlb_AccountKey +

					";EndpointSuffix=" 			+ AzBlb_EndpointSuffix;

			

			CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);

			CloudBlobClient blobClient = storageAccount.createCloudBlobClient();

			CloudBlobContainer container = blobClient.getContainerReference(AzBlb_ContainerName);

			CloudBlob blob = container.getBlockBlobReference(sapFileName);

			

			//Set Header

			BlobProperties props = blob.getProperties();

			props.setContentType("text/plain");

			

			//Upload the blob using file's inputStream and size"

			blob.upload(fileIpstrm, fileIpstrm.available());				

			mapTrace = mapTrace + "=>    The file '"+ sapFileName +"' has been successfully uploaded into Azure-Blob-Storage '" + AzBlb_AccountName + "/" + AzBlb_ContainerName +"'\n";		

			

			/*			

			blob.startCopy(new URI("http://www.bing.com"));						//Upload the blob using file-URL

			blob.uploadFromFile(sapFilePath + sapFileName);						//Upload the blob using file		

			blockBlobClient.upload(file.getInputStream(), file.getSize()); 		//Upload the blob using File inputStream		

			BlobClient blobClient = containerClient.getBlobClient(fileName);	//Get a reference to a blob

			System.out.println("\nUploading to Blob storage as blob:\n\t" + blobClient.getBlobUrl());

			 */

			

		} catch (Exception e) {

			mapTrace = mapTrace + "==>   Exception in FM 'writeFile_toAzBlbStrg()'==> " + e.getMessage() + "\n";

		}

	}

			

	private static void readFile_frmAzBlbStrg(String fName) throws Exception {

		/**

		 * Read file from Azure-Blob-Storage

		 */

		

		try {		

			String storageConnectionString = "DefaultEndpointsProtocol=https;"

					+ "AccountName=<BlbAcNm>;"

					+ "AccountKey=<BlbAcKey>"

					+ "EndpointSuffix=core.windows.net";

			CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);

			CloudBlobClient blobClient = storageAccount.createCloudBlobClient();

			CloudBlobContainer container = blobClient.getContainerReference("BlobContainerName");

			CloudBlob blob = container.getBlockBlobReference(fName);

			InputStream input = blob.openInputStream();



			// Read the File-Content

			InputStreamReader is = new InputStreamReader(input, "UTF-8");

			BufferedReader br = new BufferedReader(is);

			String read = null;

			StringBuffer sb = new StringBuffer();

			while ((read = br.readLine()) != null) {

				sb.append(read);

			}

			String utf8str = sb.toString();



			System.out.println("Reading......\n" + utf8str);

			System.out.println("File read successfully");

		} catch (Exception e) {

			System.out.println("Error: " + e.getMessage());

		}

	}	

	/*

	public static void main(String[] args) {

		try {

			String filePath = "C:\\Users\\Test\\";				

			String Fnm = "OP_Test2.csv";

			String inputFile = filePath + Fnm;	

			String outputFile = filePath + "OP_" + Fnm ;



			PushFileToAzrBlob myClass = new PushFileToAzrBlob();

			FileInputStream in = new FileInputStream(inputFile);

			FileOutputStream out = new FileOutputStream(outputFile);

			myClass.execute(in, out);

		}catch(Exception e){

			e.printStackTrace();

		}

	}*/

}

 

For reference, below is the map trace log (Map output):


Map output (map trace log)



Eclipse console map trace log


 

File gets received in Azure-Blob as below:


Azure-Blob receives file


 

I hope this helps to interested audience. Technique-2 coming soon.....
Labels in this area
Popular Blog Posts