- SAP Managed Tags:
Introduction
This year been a very much trouble for everyone and the reason is Covid-19. So i was going through lots of Tech Hackathon to see How other Techies are trying their Hands to Solve this in different forums. I was really amazed to see Data Scientists are really working hard to solve problem around this. Though i am not a Medical Student i was really surprised to see some really great research on this and one of the research is 'Detecting Covid-19 based on Chest X-Ray' It's because as per studies available over internet Covid-19 create impact in lungs which is pretty much identificable in Chest X-ray or Chest CT scan and many people have started already working on this. So i was thinking why cant we make AI system which detect Covid-19 Based on Chest X-Ray Image and which would affordable to use for Normal People. Because Its only Technology which could make things simpler. And trust me when simplycity comes in my mind automatically SAP Conversational AI comes into my mind. You could curse me and tell me Why I promote SAP Conversational AI so much but trust me according to me they are Next Generation UX.
Wait a Minute you must be thinking What, SAP Conversational AI How it will detect Covid-19 based on X-ray Image when It is meant for Building AI Based Conversational System which understand human Language the way Human understand, well some extend we can infuse emotion too in it. But Detecting Covid-19 Based on X-Ray Image is no way, this guy is so high.
Well definately you are right, but i am not high. We can also integrate a Digital Radiologist also with Conversational AI, A Digital Radiologist who is educated enough to differentiate between Covid-19, Pneumonia and Normal. But How it is Possible, Well read the whole blog.
** In the end of the blog dont forget to watch the Demo Video.
What is Digital Radiologist Here
A Tensorflow based Custom Trained Machine Learning Model with the help of expert radiologist, which would be able to differentiate between Covid-19. For this example I have trained this model with public covid-19 image datasets are availavle in Kaggle and Google. Almost 442 Covid-19 X-ray Image, 1263 Normal X-Ray Image and 3295 Pneumonia X-ray Image.
Below are the some research in this direction
Evaluation of an AI system for detection of COVID-19 on Chest X-Ray images
COVID-19 on the Chest Radiograph: A Multi-Reader Evaluation of an AI System
COVID-19 detector using X-ray images
Automated detection of COVID-19 cases using deep neural networks with X-ray images
Covid-19 Detection using X-ray Images
So What exactly I am talking about
Very simple we are going to build a Conversational AI System which would accept a X-Ray Image and send that image to Custom Tensor Flow based Machine Learning Model which would analyse the X-ray Image and Predict Weather its Normal, or Pneumonia or Covid-19. And we are going to host this SAP Conversational AI based bot over WhatsApp so it would be very easy to use because everyone almost use WhatsApp.
Let's Connect the all Block
So if you look at the Architecture of the above Solution, it is clear you need expert radiologist to classify the x-ray image and train your machine learning model continuously, for building machine learning model and training your model you could use Google Cloud Platform, AWS or Azure. Today i am not going to discuss how to build Image classification Model using TensorFlow, Open CV and Flask. in another blog i might cover. In this example i have exported TensorFlow based machine learning model in proto buffer format, then created a Python script to resize and feed the input to cusrom trained machine learning model and then created a Python flask based Rest API to Integrate with SAP Conversational AI or Web Application or Mobile Application. In this Blog you would also learn how you could bring your own custom built machine learning model and deploy that model as rest based API into SAP Cloud Platform.
Let's connect the dots
Data is the new oil
Well in order to build any custom machine learning model and train your model you first need good data and definately a validator who would validate the data. And in this case Data is as much as X-ray image i could collect. So i went to Kaggle, Github and Google image to collect the data and classified the data into three categories COVID19, PNEUMONIA, NORMAL. Then i have divided whole data into 85% and 15%. 85% Data i have used for training and 15% data for testing. In below google drive all the data i have uploaded. If you want you can download.
Collection of COVID-19, Pneumonia and Normal X-Ray Image Datasets for Training the machine learning...
I have built and trained this Machine Learning Model using Azure, but you can use Aws or Google Cloud Platform also.
Packaging the machine learning App
After building and training the machine learning model we need to package it so that it could be deployed anywhere, and if you ask me to generalise then first things comes in my mind is docker. below is how my application structure looks like.
Code snippet for app.py which is responsible for routing your HTTP request
import json
import os
import io
# Imports for the REST API
from flask import Flask, request, jsonify
# Imports for image procesing
from PIL import Image
# Imports for prediction
from predict import initialize, predict_image, predict_url
app = Flask(__name__)
# 4MB Max image size limit
app.config['MAX_CONTENT_LENGTH'] = 4 * 1024 * 1024
# Default route just shows simple text
@app.route('/')
def index():
return 'GET Request is not supported'
@app.route('/imageurl', methods=['POST'])
def predict_url_handler(project='COVID19TK', publishedName='covid19tk_optimized_sudip'):
try:
image_url = json.loads(request.get_data().decode('utf-8'))['url']
results = predict_url(image_url)
return jsonify(results)
except Exception as e:
print('EXCEPTION:', str(e))
return 'Error processing image'
if __name__ == '__main__':
# Load and intialize the model
initialize()
# Run the server
app.run(host='0.0.0.0', port=80)
Codesnippet for Python script which would resize the image, input the data to model and further calculation.
from urllib.request import Request, urlopen
from datetime import datetime
import tensorflow as tf
from PIL import Image
import numpy as np
import sys
try:
import cv2
use_opencv = True
print("Using OpenCV resizing...")
except:
use_opencv = False
print("Using CVS resizing...")
filename = 'model.pb'
labels_filename = 'labels.txt'
network_input_size = 0
output_layer = 'loss:0'
input_node = 'Placeholder:0'
graph_def = tf.compat.v1.GraphDef()
labels = []
def initialize():
print('Loading model...',end=''),
with open(filename, 'rb') as f:
graph_def.ParseFromString(f.read())
tf.import_graph_def(graph_def, name='')
# Retrieving 'network_input_size' from shape of 'input_node'
with tf.compat.v1.Session() as sess:
input_tensor_shape = sess.graph.get_tensor_by_name(input_node).shape.as_list()
assert len(input_tensor_shape) == 4
assert input_tensor_shape[1] == input_tensor_shape[2]
global network_input_size
network_input_size = input_tensor_shape[1]
print('Success!')
print('Loading labels...', end='')
with open(labels_filename, 'rt') as lf:
global labels
labels = [l.strip() for l in lf.readlines()]
print(len(labels), 'found. Success!')
def log_msg(msg):
print("{}: {}".format(datetime.now(),msg))
def extract_bilinear_pixel(img, x, y, ratio, xOrigin, yOrigin):
"""
Custom implementation of bilinear interpolation when opencv is not available
img: numpy source image array
x,y: target pixel coordinates
ratio: scaling factor
xOrigin, yOrigin: source image offset
returns interpolated pixel value (RGB)
"""
xDelta = (x + 0.5) * ratio - 0.5
x0 = int(xDelta)
xDelta -= x0
x0 += xOrigin
if x0 < 0:
x0 = 0;
x1 = 0;
xDelta = 0.0;
elif x0 >= img.shape[1]-1:
x0 = img.shape[1]-1;
x1 = img.shape[1]-1;
xDelta = 0.0;
else:
x1 = x0 + 1;
yDelta = (y + 0.5) * ratio - 0.5
y0 = int(yDelta)
yDelta -= y0
y0 += yOrigin
if y0 < 0:
y0 = 0;
y1 = 0;
yDelta = 0.0;
elif y0 >= img.shape[0]-1:
y0 = img.shape[0]-1;
y1 = img.shape[0]-1;
yDelta = 0.0;
else:
y1 = y0 + 1;
#Get pixels in four corners
bl = img[y0, x0]
br = img[y0, x1]
tl = img[y1, x0]
tr = img[y1, x1]
#Calculate interpolation
b = xDelta * br + (1. - xDelta) * bl
t = xDelta * tr + (1. - xDelta) * tl
pixel = yDelta * t + (1. - yDelta) * b
return pixel
def extract_and_resize(img, targetSize):
"""
resize and cropn when opencv is not available
img: input image numpy array
targetSize: output size
returns resized and cropped image
"""
determinant = img.shape[1] * targetSize[0] - img.shape[0] * targetSize[1]
if determinant < 0:
ratio = float(img.shape[1]) / float(targetSize[1])
xOrigin = 0
yOrigin = int(0.5 * (img.shape[0] - ratio * targetSize[0]))
elif determinant > 0:
ratio = float(img.shape[0]) / float(targetSize[0])
xOrigin = int(0.5 * (img.shape[1] - ratio * targetSize[1]))
yOrigin = 0
else:
ratio = float(img.shape[0]) / float(targetSize[0])
xOrigin = 0
yOrigin = 0
resize_image = np.empty((targetSize[0], targetSize[1], img.shape[2]), dtype=np.float32)
for y in range(targetSize[0]):
for x in range(targetSize[1]):
resize_image[y, x] = extract_bilinear_pixel(img, x, y, ratio, xOrigin, yOrigin)
return resize_image
def extract_and_resize_to_256_square(image):
"""
extracts image central square crop and resizes it to 256x256
image: input image numpy array
returns resized 256x256 central crop as numpy array
"""
h, w = image.shape[:2]
log_msg("crop_center: " + str(w) + "x" + str(h) +" and resize to " + str(256) + "x" + str(256))
if use_opencv:
min_size = min(h, w)
image = crop_center(image, min_size, min_size)
return cv2.resize(image, (256, 256), interpolation = cv2.INTER_LINEAR)
else:
return extract_and_resize(image, (256, 256))
def crop_center(img,cropx,cropy):
"""
extracts central crop
img: input image numpy array
cropx, cropy: crop size
returns central crop as numpy array
"""
h, w = img.shape[:2]
startx = max(0, w//2-(cropx//2))
starty = max(0, h//2-(cropy//2))
log_msg("crop_center: " + str(w) + "x" + str(h) +" to " + str(cropx) + "x" + str(cropy))
return img[starty:starty+cropy, startx:startx+cropx]
def resize_down_to_1600_max_dim(image):
"""
resized image to 1600px in max dimension if image exceeds 1600 by width or height
image: input image numpy array
returns downsized image
"""
w,h = image.size
if h < 1600 and w < 1600:
return image
new_size = (1600 * w // h, 1600) if (h > w) else (1600, 1600 * h // w)
log_msg("resize: " + str(w) + "x" + str(h) + " to " + str(new_size[0]) + "x" + str(new_size[1]))
if use_opencv:
# Convert image to numpy array
image = convert_to_nparray(image)
return cv2.resize(image, new_size, interpolation = cv2.INTER_LINEAR)
else:
if max(new_size) / max(image.size) >= 0.5:
method = Image.BILINEAR
else:
method = Image.BICUBIC
image = image.resize(new_size, method)
return image
def predict_url(imageUrl):
"""
predicts image by url
"""
log_msg("Predicting from url: " +imageUrl)
imgrequest = Request(imageUrl, headers={"User-Agent": "Mozilla/5.0"})
with urlopen(imgrequest) as testImage:
image = Image.open(testImage)
return predict_image(image)
def convert_to_nparray(image):
"""
converts PIL.Image to numpy array and changes RGB order to BGR
image: inpout PIL image
returns image as a numpy array
"""
# RGB -> BGR
log_msg("Convert to numpy array")
image = np.array(image)
return image[:, :, (2,1,0)]
def update_orientation(image):
"""
corrects image orientation according to EXIF data
image: input PIL image
returns corrected PIL image
"""
exif_orientation_tag = 0x0112
if hasattr(image, '_getexif'):
exif = image._getexif()
if exif != None and exif_orientation_tag in exif:
orientation = exif.get(exif_orientation_tag, 1)
log_msg('Image has EXIF Orientation: ' + str(orientation))
# orientation is 1 based, shift to zero based and flip/transpose based on 0-based values
orientation -= 1
if orientation >= 4:
image = image.transpose(Image.TRANSPOSE)
if orientation == 2 or orientation == 3 or orientation == 6 or orientation == 7:
image = image.transpose(Image.FLIP_TOP_BOTTOM)
if orientation == 1 or orientation == 2 or orientation == 5 or orientation == 6:
image = image.transpose(Image.FLIP_LEFT_RIGHT)
return image
def preprocess_image_opencv(image_pil):
"""
image_pil: PIL Image, already converted to 'RGB' and correctly oriented
returns: nparray of extracted crop
"""
image = convert_to_nparray(image_pil)
h, w = image.shape[:2]
min_size = min(h,w)
crop_size = min(min_size, int(min_size * network_input_size / 256.0))
startx = max(0, int(max(0, w//2-(crop_size//2))))
starty = max(0, int(max(0, h//2-(crop_size//2))))
new_size = (network_input_size, network_input_size)
log_msg(f"crop: {w}x{h} to {crop_size}x{crop_size}, origin at {startx}, {starty}, target = {network_input_size}")
return cv2.resize(image[starty:starty+crop_size, startx:startx+crop_size], new_size, interpolation = cv2.INTER_LINEAR)
def preprocess_image(image_pil):
"""
image_pil: PIL Image, already converted to 'RGB' and correctly oriented
returns: nparray of extracted crop
"""
# If the image has either w or h greater than 1600 we resize it down respecting
# aspect ratio such that the largest dimention is 1600
image_pil = resize_down_to_1600_max_dim(image_pil)
# Convert image to numpy array
image = convert_to_nparray(image_pil)
# Crop the center square and resize that square down to 256x256
resized_image = extract_and_resize_to_256_square(image)
# Crop the center for the specified network_input_Size
return crop_center(resized_image, network_input_size, network_input_size)
def predict_image(image):
"""
calls model's image prediction
image: input PIL image
returns prediction response as a dictionary. To get predictions, use result['predictions'][i]['tagName'] and result['predictions'][i]['probability']
"""
log_msg('Predicting image')
try:
if image.mode != "RGB":
log_msg("Converting to RGB")
image = image.convert("RGB")
w,h = image.size
log_msg("Image size: " + str(w) + "x" + str(h))
# Update orientation based on EXIF tags
image = update_orientation(image)
if use_opencv:
cropped_image = preprocess_image_opencv(image)
else:
cropped_image = preprocess_image(image)
tf.compat.v1.reset_default_graph()
tf.import_graph_def(graph_def, name='')
with tf.compat.v1.Session() as sess:
prob_tensor = sess.graph.get_tensor_by_name(output_layer)
predictions, = sess.run(prob_tensor, {input_node: [cropped_image] })
result = []
for p, label in zip(predictions, labels):
truncated_probablity = np.float64(round(p,8))
if truncated_probablity > 1e-8:
result.append({
'type': label,
'chances': truncated_probablity * 100
})
response = {
'created': datetime.utcnow().isoformat(),
'predictions': result
}
log_msg("Results: " + str(response))
return response
except Exception as e:
log_msg(str(e))
return 'Error: Could not preprocess image for prediction. ' + str(e)
Code Snippet for Docker file
FROM python:3.7-slim
RUN pip install -U pip
RUN pip install numpy==1.17.3 tensorflow==2.0.0 flask pillow
COPY app /app
# Expose the port
EXPOSE 80
# Set the working directory
WORKDIR /app
# Run the flask server for the endpoints
CMD python -u app.pyRequirements
tensorflow==1.14.0
pillow==6.1.0
numpy==1.16.4
flask==0.12.4Classification Labels
COVID19
NORMAL
PNEUMONIADocker build
docker build -t codersudip/covid19xraysudip:aarini . Docker run Locally
docker run -p 127.0.0.1:80:80 -d codersudip/covid19xraysudip:aarini Test it in Locally using Postman
Push to Docker Hub and Deploy Machine Learning Model into SAP Cloud Platform CF
docker push codersudip/covid19xraysudip:aarini Deploy into SAP Cloud Platform Cloud Foundry
cf push digitalcovid19tk --docker-image codersudip/covid19xraysudip:aariniBuilding Conversational AI and Integrate with WhatsApp
Created a Intent called covid test and below is how all expression looks like
Below is how my skills look like
Trigger
Requirements
Action
Integrating Conversational AI with WhatsApp Follow below blogs
To understand How Conversational AI can be Integrated with WhatsApp follow this blog
In order to understand How Image also could be sent over WhatsApp follow this blog by vandanagupta09
Now let's have a look how the demo looks like, Thats Important because it only give more clear understanding.
DEMO
I hope you liked this blog, if you like then like, share and in comment let me know how it is. Meanwhile enjoy your day, be safe and Play with SAP Cloud Platform.
**N.B In order to use this in Production it needs lot of validation , need to train with more and more data, evaluate the model , approval from different Medical Regulatory Board, its just an idea or showcase which i wanted to present.
14 Comments
