AI-powered Summarization Tool with Programmatical Pod Spawning in Kyma for Asynchronous Processing

[ヨアヒム]

AI-Powered Summarization Tool in SAP Kyma

SummurAI - Get your 

In today's fast-paced world, information overload is a common challenge. Whether it’s for business analysts, academia, or everyday reading, lengthy documents can often be overwhelming and time-consuming. To address this issue, I have developed a scalable yet simple AI-powered summarization service that easily extracts the most important information from large documents and writes a consolidated summary. With the help of the python kubernetes library, I was able to run the summarization processes asynchronously, allowing for multiple summaries to be written simultaneously. Just send in your document and let the summurai do it's business, while you deal with other tasks until you receive the summary by email. This makes the process of information consumption very simple and more effective and efficient.

Simple UX Workflow

Options and LLM Parameters

• Model: Chose from a variety of large language models to get different results
• Summarization Method: Chose from 3 different summarization methods:
  • Stuff: Summarizing the whole document in one single request, which can be problematic with very large documents.
  • Map Reduce: Summarizing chunks of a document seperately, before producing a final consolidated summary from all the summarized chunks.
  • Refine: Starting with the summarization of the first chunk, then repeatedly takin in the already summarized text, adding the next chunk of the document and creating a new summary from that until all the chunks of the document were processed.
• Summarization Language: Configures the desired language of the summary.
• Chunk Settings and Temperature: Set the chunk size for the splitting of the document and an overlap to smoothen the text chunks. You may also set a temperature for the language model, but it is recommended to set it very low, since a summarization is very objective and doesn't require any creativity from the llm.

Architecture

• API Trigger: User actions as described in the UX Workflow call an endpoint of the SummurAI service, starting the summarization process with uploaded document and the configured options and parameters
• File and Parameter Management: The uploaded PDF is saved, and necessary parameters and credentials are stored as environment variables for subsequent processing in the pod that will be spawned for the summarization.
• Kubernetes Integration: A Kubernetes job is created dynamically with the python kubernetes library to be able to run multiple summarization processes simultaneously since they usually take some time to finish, especially with larger documents. This procedure will be explained further below -> 
• Summarization Process: An LLM chain is initialized and invoked with the python langchain library with the specified model, the set parameters and the document, split into smaller chunks. After some time, the chain will produce a summary in html format as output, which will be attached to an email and sent using a python email library.
• Resource Cleanup: Upon completion, the pod is deleted after 5 minutes of inactivity and the corresponding files in the shared mounted volume are deleted.

Detailed Walkthrough of my Approach for Programmatic Pod-Spawning in Kyma

The Motivation for this approach first of all the long summarization process, which, depending on the method used, can easily take an hour if the document is really big.

So, to prevent the user from having to stay on the UI the whole time, and, more importantly, to allow multiple users to use the service at the same time, I decided that it would be interesting to spawn a new K8s Job for every summarization process. That Job then automatically spawns a Pod to run it's logic in. When the pod finished it's logic, after 5 minutes of inactivity, it is being deleted.

In the following I'll explain, how this works based on some code snippets. The python kubernetes library makes it really comfortable. Basically, the yaml structure has to be transformed into the following structure of objects from the kubernetes library.

The following modules will be needed throughout the procedure:

from kubernetes import client, config

First, load the config of your current cluster:

try:
    config.load_incluster_config()
except config.ConfigException as e:
    logging.error(f"Couldn't load in cluster config: {e}")

Then define the necessary kubernetes objects, that you want your job to be able to use. This is the example from my code:

    # Define the shared volume
    shared_volume = client.V1Volume(
        name="summurai-pv",
        persistent_volume_claim=client.V1PersistentVolumeClaimVolumeSource(
            claim_name="summurai-pvc"
        )
    )

    # Define the volume mount for the worker pod
    volume_mount = client.V1VolumeMount(
        mount_path="/mnt/shared",
        name="summurai-pv"
    )

    # Define the job
    job = client.V1Job(
        api_version="batch/v1",
        kind="Job",
        metadata=client.V1ObjectMeta(name=job_name, namespace=namespace),
        spec=client.V1JobSpec(
            template=client.V1PodTemplateSpec(
                spec=client.V1PodSpec(
                    containers=[
                        client.V1Container(
                            name=pod.name,
                            image=pod.image,
                            env=env_vars,
                            command=["python", "-c", "import job; job.main()"],
                            volume_mounts=[volume_mount]
                        )
                    ],
                    volumes=[shared_volume],
                    restart_policy="Never"
                )
            ),
            ttl_seconds_after_finished=300  # 5 minutes
        )
    )

Again, you're defining a Job that contains a Pod configuration which again contains configurations for all the necessary kubernetes elements. In my case that is a Container and a Volume.
For the Container you want to pass in a self chosen name, a docker image, a list of the needed environment variables (use the type V1EnvVar(name="", value="") from the kubernetes.client module) and a list of the Volume Mounts, as well as a command that get's executed upon pod startup.

Eventually, you will need to actually create the job with the configuration that you just defined, like this:

# Create the job
batch_v1 = client.BatchV1Api()
batch_v1.create_namespaced_job(body=job, namespace=namespace)

Now some helper functions and extra information:

For the pod image, I'm retrieving the image of the "parent pod" (as I will call the pod, that receives the requests and spawns the Jobs) and use that, since it contains all the necessary code.

def get_pod_image(pod_name): 
    v1 = client.CoreV1Api()
    try:
        pod = v1.read_namespaced_pod(name=pod_name, namespace="aiplayground")
        return pod.spec.containers[0].image
    except ApiException as e:
        logging.error(f"Exception while trying to get image for pod {pod_name}: {e}")
        return None

Pay attention that your pod has the right Role and Role Binding configured for all the actions. The following YAML file covers all the necessary permissions that I need for my procedure:

# Role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: aiplayground
  name: summurai-job-spawner
rules:
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["create", "get", "list", "watch", "delete"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create", "get", "list", "watch", "delete"]
---
# Role Binding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: summurai-job-spawner-binding
  namespace: aiplayground
subjects:
- kind: ServiceAccount
  name: summurai-job-spawner
  namespace: aiplayground
roleRef:
  kind: Role
  name: summurai-job-spawner
  apiGroup: rbac.authorization.k8s.io
---

Also, for the Job and Pod names, make sure you have a logic like the following, to avoid errors due to duplication.

def job_name_exists(name, namespace)
    config.load_incluster_config()
    batch_v1 = client.BatchV1Api()
    jobs = batch_v1.list_namespaced_job(namespace) 
    for job in jobs.items:
        if job.metadata.name == name:
            return True
    return False

def pod_name_exists(name, namespace):
    config.load_incluster_config()
    v1 = client.CoreV1Api()
    pods = v1.list_namespaced_pod(namespace=namespace)
    for pod in pods.items:
        if pod.metadata.name == name:
            return True
    return False

The described procedure allows you to programmatically spawn Kubernetes Jobs from within another Pod, so you can run tasks simultaneously, ansynchronously. You can always read further in the kubernetes library documentation.

I didn't experiment with the following, but it would be interesting to know if there even is a chance to callback to the original pod. If so, one would be able to outsource only the most resource consuming tasks and then continue with the result back in the "parent pod". If you happen to know something about that, I would be all ears!

Benefits of AI-Powered Summarization

• Time-Saving and Efficient: Drastically reduces the time required to read through lengthy documents. You don't have to watch the summary going. Deal with some other topics or go grab a coffee, since you will be automatically notified when the summary is finished.
• Simple UX: Since there are default settings for the LLM parameters and summarization options, there is no need to set any of those. Just upload a document, send it and wait for the summary.
• Flexibility: If wished for, You have the ability to customize summarization options and other parameters to meet your specific needs or improve the summary output if you're not entirely satisfied with what the AI gives you.

Conclusion: Transforming Information Consumption

In a world where time is of the essence, our AI-powered summarization tool integrated with the Apachat AI Playground stands out as an indispensable asset. By transforming the way users consume and comprehend information, it paves the way for increased productivity and informed decision-making. Embrace the future of smart summarization and let this tool do the heavy lifting for you.