- SAP Managed Tags
Disclaimer
Material in this blog post is provided for information and technical features demonstration purposes only. It is strongly discouraged to create, update or delete any files on the file system of a CPI runtime node. Please use the below described technique at your own risk.
Acknowledgements
I would like to sincerely thank v.kuzyakin who worked shoulder to shoulder with me on this demo and appreciate substantial time that he invested in providing instrumental ideas and insightful comments about features and functionality, and diligent overall review which all together made the demo more advanced and feature-rich, and extended a range of supported and illustrated use cases.
Intro
Two years ago, ariel.bravoayala3 published a blog post where he demonstrated a technique and provided iFlow-based implementation for it, to browse a file system of a CPI runtime node. The tool is very handy and allows us to gain additional knowledge about a runtime node.
Over time, I though what other tools can be employed to help us in exploration endeavours and troubleshooting activities that might require access to a file system of a runtime node. In the previous year, I wrote a blog post where I described how we can execute some shell commands on a runtime node. Among other things, the technique can be used to browse a file system and access relevant content: for example, we can use an
ls command to list content of directories, and a cat command to list files content.Later on, I explored other alternatives and this is how I have come to an idea of creating an API that can be used to execute a full set of CRUD operations with files that are located on a file system of a runtime and that are accessible from a CPI application programmatically using JDK and GDK APIs. In this blog post, I would like to share some techniques and sample implementations that can be further enhanced, extended and adapted to specific requirements.
Overview
The overall implementation takes the shape of a custom-developed iFlow to which necessary commands can be sent in the form of HTTP requests. The iFlow supports several HTTP methods mapped to operations that are to be executed on a file system, where relevant arguments are passed as URL query parameters. An overview of the iFlow is provided on the illustration below:
The entire iFlow can be split into several building blocks:
An HTTPS sender connection is used to enable the iFlow to accept incoming HTTPS calls that will trigger the core logic and that can be generated by any relevant HTTP client that supports creation of requests for different HTTP methods. In the following demo, I’m going to use Postman, but we could have used alternatives (both GUI and CLI tools) – for example, cURL. In sake of simplicity, I disabled CSRF protection for this connection.
A step to parse URL query parameters contained in the incoming HTTP request. I implemented this step with the help of a Groovy script function that reads the needed message header that contains URL query parameters, composes a list of found parameter name-value pairs and places query parameter values into corresponding exchange properties that are named after query parameter names and prefixed with
QueryParameter_. This equips us with a generic and unified mechanism of parsing any arbitrary list of query parameters and enabling access to them at later processing steps in the iFlow.
A code snippet of the invoked Groovy function:
import com.sap.gateway.ip.core.customdev.util.Message
import java.nio.charset.StandardCharsets
Message parse(Message message) {
String query = message.getHeader('CamelHttpQuery', String)
query = (query) ? URLDecoder.decode(query, StandardCharsets.UTF_8.name()) : ''
Map<String, String> queryParameters = query.tokenize('&').collectEntries { entry ->
entry.tokenize('=').toSpreadMap().collectEntries { parameter ->
['QueryParameter_' + parameter.key, parameter.value]
}
}
message.setProperties(queryParameters)
return message
}While some of implemented operations can accept additional operation-specific optional parameters, there is one parameter –
path – that is mandatory and common for all of them – a file/directory path to an object that an operation shall be applied to. Therefore, we would like to have an early check that the mandatory parameter was provided in the request. I use a router step for this: if the required query parameter (and consequently, a corresponding exchange property) is missing, the iFlow produces a message with a response status code 400 (Bad Request):
An example of a request with a missing mandatory parameter and a corresponding response generated by the iFlow for it:
The next step is to check an HTTP method of the submitted request and the processed message and ensure that it is amongst those supported by the iFlow. If that isn’t the case, the iFlow produces a message with a response status code 405 (Method Not Allowed) and provides a list of supported methods in the ‘Allow’ header field:
Currently the iFlow supports following HTTP methods and query parameters for them:
| HTTP method | Query parameters | Operation | Groovy function |
| HEAD | path | Check if file exists | check |
| GET | path | Download file | download |
| POST | path, mode, permissions | Upload file | upload |
| DELETE | path | Delete file | delete |
| VIEW | path, output | Browse directory | list |
An example of a request for an unsupported HTTP method and a corresponding response generated by the iFlow for it:
Note: for safety purposes, all operations related to files (check, download, upload, delete file) can be applied to regular files only, and not to other types of objects that can be found in a file system (for example, directories). Should a directory be specified in the
path parameter for a file operation, an error will be returned in a response message.Remark: the one might note and argue that a check of an HTTP method shall logically precede more specific validations such as the one that verifies if a mandatory query parameter was submitted in the request, since generally speaking, there are HTTP methods that don’t require any specific parameterization – for example, an HTTP method OPTIONS. This is absolutely true, and if we would have aimed development and provisioning of an API that is reasonably compliant to HTTP and REST specifications, an implementation would have to get changed and enriched with some additional steps and checks, as well as a sequence of existing checks would have to get swapped around. Should we have done so in this particular iFlow, it would have resulted in a more complicated implementation or a larger degree of duplication of checks across several routes and message processing steps in the iFlow. Hence, we intentionally placed checks in the order as it appears now to keep the iFlow implementation more straightforward and given all operations that are supported by the iFlow currently, require the presence of the same parameter.
Finally, we arrive to the central part of the iFlow – a group of steps that implement necessary operations that are to be applied to objects on a file system of a runtime node. Implementation of each operation resides in a separate dedicated function, and all functions are contained in a single Groovy script.
In next sections, we will get familiar with implementation details of each of these operations, but what unites all of them is the underlying API that is used to access a file system – Java NIO.
In principle, there are two APIs that are provided by JDK (and that are further enhanced with additional methods by GDK) to facilitate operations with files:
- Java I/O. This API was introduced a while ago, it is well-supported and is still widely used in various products.
- Java NIO ("New" I/O). This API is a successor of Java I/O, it was firstly introduced in Java 1.4 (NIO), and then significantly updated, enhanced and extended in Java 7 (NIO.2). This API provides additional features and demonstrates a higher performance compared to its predecessor.
There are a number of 3rd party libraries that provide an additional abstraction layer on top of mentioned Java APIs and that can be considered as well – for example, Apache Commons IO.
In the iFlow that is used throughout this blog post, the implementation is based on pure JDK and GDK APIs, so that we can relax and avoid dependencies on additional libraries. We also intend to utilize Java NIO API, so that we can benefit from some additional features that they provide to developers.
The remaining part of the blog post will walk you through sample implementations and illustrations of their usage for the earlier mentioned operations – I will only scratch the surface of how Java NIO API can be used in the context of CPI, but it shall be indicative of how capable and flexible this API is.
Note: when using Linux shell commands, you might have come across some handy features that a shell provides – one of them are so-called shell expansions. While interpreting an entered command, a shell is capable of replacing some tokens with results of certain computations or environment variables. It is very likely that you have come across at least one of them –
~/ – that is used to denote a home directory of a currently logged user. In fact, there are several kinds of expansion that a shell performs and the above mentioned ~/ is one of examples of a tilde expansion. A comprehensive description of shell expansions that are supported by a Bash shell in Ubuntu that is used by a runtime node currently, can be found in a corresponding documentation. There is no equivalent functionality offered by JDK/GDK APIs, as shell expansions are a feature that is specific to a shell, not to underlying system APIs, and this demo doesn’t aim replication of shell expansions logic to its full extent, but given ~/ is used so often, I added a support for it here. Since this functionality is going to be used by all operations, logic that is required to expand ~/ has been moved to a separate function in the script – expandPath(String):private static String expandPath(String path) {
// Supports only tilde expansion for current user's home directory
return path?.replaceAll('~/', System.getProperty('user.home') + '/')
}Obviously, this method can be enhanced and advanced further, and can develop into a group of methods or even a separate utility class that will support further shell expansions handling, but implementation of the entire set of available shell expansions goes beyond the scope of this blog post and isn’t covered by this demo.
Check if file exists
The operation checks if a file specified in the
path parameter, exists on a file system of a runtime node.If a file exists, the iFlow produces a response message with a response status code 200 (OK). Otherwise, if a file doesn’t exist, the iFlow produces a response message with a response status code 404 (Not Found).
A code snippet of the invoked Groovy function:
import com.sap.gateway.ip.core.customdev.util.Message
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.Paths
Message check(Message message) {
Path file = Paths.get(expandPath(message.getProperty('QueryParameter_path') as String))
try {
if (Files.exists(file) && Files.isRegularFile(file)) {
message.setHeader('CamelHttpResponseCode', 200)
} else {
message.setHeader('CamelHttpResponseCode', 404)
}
} catch (Exception e) {
message.setHeader('CamelHttpResponseCode', 500)
}
return message
}Examples of requests and a corresponding responses generated by the iFlow when the specified file exists and doesn't exist:
Download file
The operation downloads content of a file specified in the
path parameter, if such file exists.The implementation is content type and file format agnostic and can be used to download both binary and text files as content is accessed and transferred in a binary mode.
The iFlow produces a response message where file content is placed in the response message body, and an application/octet-stream header contains indication of a downloaded file name.
A code snippet of the invoked Groovy function:
import com.sap.gateway.ip.core.customdev.util.Message
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.Paths
Message download(Message message) {
Path file = Paths.get(expandPath(message.getProperty('QueryParameter_path') as String))
try {
if (Files.exists(file) && Files.isRegularFile(file) && Files.isReadable(file)) {
byte[] content = Files.readAllBytes(file)
message.setBody(content)
message.setHeaders([
'CamelHttpResponseCode': 200,
'Content-Disposition' : "inline; filename=\"${file.fileName.toString()}\"",
'Content-Type' : 'application/octet-stream'
])
} else {
message.setBody("File ${file.toAbsolutePath().toString()} does not exist or is not readable")
message.setHeaders([
'CamelHttpResponseCode': 500,
'Content-Type' : 'text/plain'
])
}
} catch (Exception e) {
message.setBody("Error when reading file ${file.toAbsolutePath().toString()}: ${e.message}")
message.setHeaders([
'CamelHttpResponseCode': 500,
'Content-Type' : 'text/plain'
])
}
return message
}An example of a request and a corresponding response generated by the iFlow for it:
Alternatively, a response body can be saved to a file - this is useful for larger files or files containing binary data:
Upload file
The operation uploads content that is found in a request message body, to a file specified in the
path parameter.The implementation is content type and file format agnostic and can be used to upload both binary and text files as content is transferred and saved to a file in a binary mode.
Firstly, it is checked if a parent directory (a directory to which the file is to be uploaded) exists and is writable – if the parent directory doesn’t exist or isn’t writable, the iFlow produces a response message with an error.
Note: for safety purposes, the operation doesn’t attempt to create intermediate or parent directories if they don’t exist, but produces an error in such cases.
Next, it is checked If the file at the specified path already exists. If the file doesn’t exist yet, it will be created and content contained in the request body will be saved to it. In contrast, if the file already exists, a
mode parameter is evaluated:- If its value is
overwrite, existing file content is overwritten with uploaded content, - If its value is
append, uploaded content is appended to existing file content. This option shall be used with caution: it shall not be applied to binary content or text content in structured data formats (XML, JSON, etc.), as this may cause inconsistencies in joint content (malformed content), - If its value is
skip, an upload operation is skipped and file content remains unmodified. - If the parameter isn’t provided or its value is anything different from those mentioned above, an upload operation is skipped and file content remains unmodified (same as if the parameter value would have been equal to
skip).
When the file is created or updated using either of options above, default file permissions are set to it –
rw-r--r-- in a symbolic notation, or 644 in a numeric (octal) notation (read and write permissions for the owner, read permission for all other users). There are certain use cases when default permissions aren’t desirable and the file has to be created with some other (more or less restrictive) permissions. For such cases, a permissions parameter can be used: if it is provided and its value is a valid representation of file permissions in either symbolic or numeric notation, corresponding permissions will be assigned to the created/updated file. To retain readability of the function that is invoked for upload of the file, logic that is required to parse the value of the permissions parameter, has been moved to a separate function in the script – parseFilePermissions(String):import java.nio.file.attribute.PosixFilePermission
import java.nio.file.attribute.PosixFilePermissions
private static Set<PosixFilePermission> parseFilePermissions(String value) {
Set<PosixFilePermission> permissions = []
switch (value) {
case { value ==~ /^([-r][-w][-x]){3}$/ }: // Symbolic notation
permissions = PosixFilePermissions.fromString(value)
break
case { value ==~ /^[01234567]{3}$/ }: // Numeric notation
value.eachWithIndex { triad, triadIdx ->
Integer.toBinaryString(triad.toInteger()).padLeft(3, '0').eachWithIndex { String bit, int bitIdx ->
if (bit.toInteger()) {
if (triadIdx == 0 && bitIdx == 0) permissions.add(PosixFilePermission.OWNER_READ)
if (triadIdx == 0 && bitIdx == 1) permissions.add(PosixFilePermission.OWNER_WRITE)
if (triadIdx == 0 && bitIdx == 2) permissions.add(PosixFilePermission.OWNER_EXECUTE)
if (triadIdx == 1 && bitIdx == 0) permissions.add(PosixFilePermission.GROUP_READ)
if (triadIdx == 1 && bitIdx == 1) permissions.add(PosixFilePermission.GROUP_WRITE)
if (triadIdx == 1 && bitIdx == 2) permissions.add(PosixFilePermission.GROUP_EXECUTE)
if (triadIdx == 2 && bitIdx == 0) permissions.add(PosixFilePermission.OTHERS_READ)
if (triadIdx == 2 && bitIdx == 1) permissions.add(PosixFilePermission.OTHERS_WRITE)
if (triadIdx == 2 && bitIdx == 2) permissions.add(PosixFilePermission.OTHERS_EXECUTE)
}
}
}
break
// default: // Invalid value
}
return permissions
}A code snippet of the invoked Groovy function:
import com.sap.gateway.ip.core.customdev.util.Message
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.Paths
import java.nio.file.StandardOpenOption
import java.nio.file.attribute.PosixFilePermission
Message upload(Message message) {
Path file = Paths.get(expandPath(message.getProperty('QueryParameter_path') as String))
String mode = (message.getProperty('QueryParameter_mode') as String)?.toLowerCase()
Set<PosixFilePermission> permissions = parseFilePermissions(message.getProperty('QueryParameter_permissions') as String)
byte[] content = message.getBody(byte[])
Path directory = file.parent
try {
if (Files.exists(directory) && Files.isDirectory(directory) && Files.isWritable(directory)) {
if (Files.exists(file) && Files.isRegularFile(file)) {
switch (mode) {
case 'overwrite':
Files.write(file, content, StandardOpenOption.TRUNCATE_EXISTING)
if (permissions) {
Files.setPosixFilePermissions(file, permissions)
}
message.setBody("File ${file.toAbsolutePath().toString()} already exists, content was overwritten")
break
case 'append':
Files.write(file, content, StandardOpenOption.APPEND)
if (permissions) {
Files.setPosixFilePermissions(file, permissions)
}
message.setBody("File ${file.toAbsolutePath().toString()} already exists, content was appended")
break
case 'skip':
message.setBody("File ${file.toAbsolutePath().toString()} already exists, upload was skipped")
break
default:
message.setBody("File ${file.toAbsolutePath().toString()} already exists, upload was skipped")
}
} else {
Files.write(file, content)
if (permissions) {
Files.setPosixFilePermissions(file, permissions)
}
message.setBody("File ${file.toAbsolutePath().toString()} was uploaded")
}
message.setHeaders([
'CamelHttpResponseCode': 200,
'Content-Type' : 'text/plain'
])
} else {
message.setBody("Directory ${directory.toAbsolutePath().toString()} does not exist or is not writable")
message.setHeaders([
'CamelHttpResponseCode': 500,
'Content-Type' : 'text/plain'
])
}
} catch (Exception e) {
message.setBody("Error when creating file ${file.toAbsolutePath().toString()}: ${e.message}")
message.setHeaders([
'CamelHttpResponseCode': 500,
'Content-Type' : 'text/plain'
])
}
return message
}An example of a request to upload a new file with default permissions and a corresponding response generated by the iFlow for it:
Note that the file was processed in a binary mode. Instead of a text file, we could have sent a binary file in the similar way. Text data can be also provided directly in a request body.
As it has been mentioned, in a default mode, if an attemp to upload a file that already exists on a file system of a runtime node is performed, the attempt will be skipped:
Below is an example of an advanced usage of a file upload operation where teh file is uploaded in an overwrite mode and custom defined permissions are set to it:
Delete file
The operation deletes a file specified in the
path parameter, if such file exists.A code snippet of the invoked Groovy function:
import com.sap.gateway.ip.core.customdev.util.Message
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.Paths
Message delete(Message message) {
Path file = Paths.get(expandPath(message.getProperty('QueryParameter_path') as String))
try {
if (Files.exists(file) && Files.isRegularFile(file)) {
Files.delete(file)
message.setBody("File ${file.toAbsolutePath().toString()} was deleted")
message.setHeaders([
'CamelHttpResponseCode': 200,
'Content-Type' : 'text/plain'
])
} else {
message.setBody("File ${file.toAbsolutePath().toString()} does not exist")
message.setHeaders([
'CamelHttpResponseCode': 500,
'Content-Type' : 'text/plain'
])
}
} catch (Exception e) {
message.setBody("Error when deleting file ${file.toAbsolutePath().toString()}: ${e.message}")
message.setHeaders([
'CamelHttpResponseCode': 500,
'Content-Type' : 'text/plain'
])
}
return message
}An example of a request and a corresponding response generated by the iFlow for it:
Browse directory
The operation lists content of the directory specified in the
path parameter, including hidden directories and files.Note: in sake of performance considerations, the operation doesn’t list directories and files recursively (doesn’t list directories and files in the specified directory’s subdirectories).
In the produced output list, directories and files are distinguished in such a way that a file name appears identically to its original name on a file system, while a directory name appears with a trailing
/ symbol. Following sorting order is applied to entries that are listed in the output:- Firstly, hidden directories and files (in Linux system, these are directories and files which names start with
.) are listed, followed by other directories and files, - Within each of the above mentioned two groups, directories are listed first, followed by files,
- Remaining sorting is in the alphabetical order.
Given that the
path parameter might contain a relative path, while it might be also useful to get to now an absolute path for it, a response message contains an additional X-Directory header that indicates an absolute path to a listed directory.Sometimes it might be useful to get more details about directories and files, and not only their names. For such cases, an
output parameter can be used: if it is provided and its value is equal to verbose, an output will be verbose and will include additional details about listed directories and files – such as owner, group, permissions, size, last modified time. In a verbose mode, an output might not be easily readable due to absence of proper output formatting (an output is produced as plain text and lacks columns alignment, text colouring, etc.), but since it is a pipe-delimited output, it can be copied from or saved a response message and then imported, parsed and analyzed in tools that are capable of handling CSV-like data formats.A code snippet of the invoked Groovy function:
import com.sap.gateway.ip.core.customdev.util.Message
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.Paths
import java.nio.file.attribute.PosixFileAttributes
import java.nio.file.attribute.PosixFilePermissions
import java.time.format.DateTimeFormatter
import java.util.stream.Collectors
Message list(Message message) {
Path directory = Paths.get(expandPath(message.getProperty('QueryParameter_path') as String))
boolean isOutputVerbose = ((message.getProperty('QueryParameter_output') as String)?.toLowerCase() == 'verbose')
try {
if (Files.exists(directory) && Files.isDirectory(directory)) {
StringBuilder builder = new StringBuilder()
List<Path> entries = Files.walk(directory, 1).skip(1).collect(Collectors.toList())
if (entries) {
if (isOutputVerbose) {
builder.append(new StringJoiner('|', '', '\n').with {
add('Owner')
add('Group')
add('Permissions')
add('Size (bytes)')
add('Last modified time')
add('Name')
}.toString())
}
entries
.toSorted(new OrderBy([
{ Path entry -> entry.fileName.toString().startsWith('.') },
{ Path entry -> Files.isDirectory(entry) },
{ Path entry -> !entry.fileName.toString() }
]).reversed())
.each { Path entry ->
PosixFileAttributes attributes = Files.readAttributes(entry, PosixFileAttributes)
StringJoiner line = new StringJoiner('|', '', '\n')
if (isOutputVerbose) {
line.with {
add(attributes.owner().name)
add(attributes.group().name)
add(PosixFilePermissions.toString(attributes.permissions()))
add(attributes.size().toString())
add(DateTimeFormatter.ISO_INSTANT.format(attributes.lastModifiedTime().toInstant()))
}
}
line.add(entry.fileName.toString() + (attributes.directory ? '/' : ''))
builder.append(line.toString())
}
}
message.setBody(builder.toString())
message.setHeaders([
'CamelHttpResponseCode': 200,
'Content-Type' : 'text/plain',
'X-Directory' : directory.toAbsolutePath().toString()
])
} else {
message.setBody("Directory ${directory.toAbsolutePath().toString()} does not exist")
message.setHeaders([
'CamelHttpResponseCode': 500,
'Content-Type' : 'text/plain'
])
}
} catch (Exception e) {
message.setBody("Error when listing directory ${directory.toAbsolutePath().toString()}: ${e.message}")
message.setHeaders([
'CamelHttpResponseCode': 500,
'Content-Type' : 'text/plain'
])
}
return message
}An example of a request to list a directory using a simple output and a corresponding response generated by the iFlow for it, inlcuding the
X-Directory header:
An example of a request to list a directory using a verbose output and a corresponding response generated by the iFlow for it:
As a last example for this blog post, I saved the above response body content to a file, opened it in Microsoft Visual Studio Code and aligned its columns to illustrate how it can be presented in a better formatted output (as it has been mentioned, a plethora of simple and advanced editors can be used to process retrieved data):
Bluesky| User | Count |
|---|---|
| 27 | |
| 24 | |
| 20 | |
| 19 | |
| 13 | |
| 13 | |
| 12 | |
| 12 | |
| 12 | |
| 11 |
