Parallel Data Filtering and Dropbox Integration Using SAP Cloud Integration

[Vurumind03]

Scenario Overview

Objective:
-> The goal was to build an integration flow in SAP CPI that processes structured XML data, applies conditional filtering through parallel branches, and stores the filtered outputs as separate files in Dropbox.
-> I have used Start Timer to trigger scheduled intervals, and applied filters in parallel branches to separate records, and transferred them securely to Dropbox. To ensure reliability, I added an Exception Subprocess for error handling and email notifications.

Step 1: Configuring Dropbox

1. Create a Dropbox App
   Go to https://www.dropbox.com/developers/apps (The External link is one of the processes, to setup folder in Dropbox website. To store files in the Dropbox we need to configure in that link. So, I have given direct link for not getting confusion for anyone.) and create a new app.
2. Set Redirect URL
   Add your SAP CPI tenant’s redirect URL in the app settings. This is required for OAuth authorization.
   
3. Get Credentials
   Dropbox generates an App Key and App Secret.
    -> Use App Key as Client ID in CPI.
    -> Use App Secret as Client Secret in CPI.
   
4. Set Permissions
   Select required scopes under Permissions:
   -> files.content.write
   -> files.content.read

Step 2: Configure OAuth2 in SAP CPI

->In CPI:

• Navigate to Monitor → Manage Security → Security Material.
• Create OAuth2 Authorization Code credentials.

->Fill in details:

• Authorization URL: https://www.dropbox.com/oauth2/authorize?token_access_type=offline
• Token Service URL: https://api.dropboxapi.com/oauth2/token
• Client ID / Secret: From Dropbox app
• Redirect URI: Same as in Dropbox settings

->Deploy and authorize. After successful login, CPI confirms with: "Authorization was successful: Refresh Token was added to the OAuth2 Authorization Code Credential."

Step 3: Design the IFlow

1. Start Timer
Triggers the flow at scheduled intervals (e.g., every 10 minutes).
2. Content Modifier
Add XML data in the message body. Example structure:

<students>
<student>
<id>101</id>
<name>Ravi Kumar</name>
<age>20</age>
<gender>Male</gender>
<department>Computer Science</department>
<grades>
<subject name="Math">85</subject>
<subject name="Physics">78</subject>
<subject name="Chemistry">88</subject>
</grades>
</student>
<student>
<id>102</id>
<name>Anjali Sharma</name>
<age>21</age>
<gender>Female</gender>
<department>Electronics</department>
<grades>
<subject name="Math">90</subject>
<subject name="Physics">82</subject>
<subject name="Chemistry">91</subject>
</grades>
</student>
<student>
<id>103</id>
<name>Mohit Verma</name>
<age>22</age>
<gender>Male</gender>
<department>Mechanical</department>
<grades>
<subject name="Math">75</subject>
<subject name="Physics">80</subject>
<subject name="Chemistry">70</subject>
</grades>
</student>
<student>
<id>104</id>
<name>Sneha Reddy</name>
<age>20</age>
<gender>Female</gender>
<department>Civil</department>
<grades>
<subject name="Math">88</subject>
<subject name="Physics">85</subject>
<subject name="Chemistry">89</subject>
</grades>
</student>
<student>
<id>105</id>
<name>Amit Singh</name>
<age>23</age>
<gender>Male</gender>
<department>Electrical</department>
<grades>
<subject name="Math">92</subject>
<subject name="Physics">90</subject>
<subject name="Chemistry">94</subject>
</grades>
</student>
<student>
<id>106</id>
<name>Priya Das</name>
<age>21</age>
<gender>Female</gender>
<department>Computer Science</department>
<grades>
<subject name="Math">87</subject>
<subject name="Physics">83</subject>
<subject name="Chemistry">86</subject>
</grades>
</student>
<student>
<id>107</id>
<name>Rahul Mehta</name>
<age>22</age>
<gender>Male</gender>
<department>Electronics</department>
<grades>
<subject name="Math">78</subject>
<subject name="Physics">76</subject>
<subject name="Chemistry">80</subject>
</grades>
</student>
<student>
<id>108</id>
<name>Kavita Joshi</name>
<age>20</age>
<gender>Female</gender>
<department>Mechanical</department>
<grades>
<subject name="Math">84</subject>
<subject name="Physics">79</subject>
<subject name="Chemistry">85</subject>
</grades>
</student>
<student>
<id>109</id>
<name>Suresh Nair</name>
<age>23</age>
<gender>Male</gender>
<department>Civil</department>
<grades>
<subject name="Math">81</subject>
<subject name="Physics">77</subject>
<subject name="Chemistry">82</subject>
</grades>
</student>
<student>
<id>110</id>
<name>Neha Gupta</name>
<age>21</age>
<gender>Female</gender>
<department>Electrical</department>
<grades>
<subject name="Math">89</subject>
<subject name="Physics">88</subject>
<subject name="Chemistry">90</subject>
</grades>
</student>
</students>

Also, define a dynamic property (Optional) for file naming:
filename = ${date:now:yyyyMMdd_HHmmss}.xml
3. Parallel Multicast
Routing simultaneously to multiple branches for filtering.
4. Filters
Apply XPath conditions in each branch:

1. Male students: /students/student[gender='Male']
2. Female students: /students/student[gender='Female']

5.Dropbox Receiver Adapter
Configure each branch to upload filtered files to Dropbox using OAuth credentials.
6. Exception Subprocess
Handles errors:

• Content Modifier: Capture ${exception.message}
• Mail Adapter: Send alerts via SMTP

{Mail Adapter:
Configure SMTP settings for email notification.
• Host: smtp.gmail.com
• Port: 587
• Authentication: Username/Password
• To: [email protected]
• From: [email protected]
This setup ensures immediate visibility when errors occur in processing or Dropbox connectivity.}

Step 4: Deploy and Test

Deploy the Integration Flow and monitor message processing from the Monitoring tab in Integration Suite. Check the Dropbox folders for generated files and check the email alerts for any errors.

Key Benefits

• Automated data separation and storage
• Parallel processing for better performance
• Error alerts

Conclusion

-> This integration highlights the flexibility of SAP Cloud Integration for handling structured data. By combining parallel processing, conditional filtering, and secure Dropbox connectivity, I was able to automate a process, otherwise I required manual effort. The design with error handling made the solution reusable for scenarios like reporting, batch uploads, and data archiving. This type of approach easy as well.