Image Search Engine Bot with SAP Cloud Foundry, Python and NodeJS

[ferrygun18]

In this post, I would like to share how to build an image search engine bot with Facebook Messenger platform, SAP Cloud Foundry and some Python & NodeJS codes.

User interacts with the bot on Facebook Messenger by uploading the photo of shoes and the bot will response with the similar images.

Setup and Configuration

We will have a basic setup with the following components and configuration.

• SAP Cloud Foundry & Cloud Platform


• SAP Cloud Connector


• Python App


• NodeJS App


• Facebook Messenger Bot

There are three main parts that we need to build: Python app, NodeJS app and HTML5 app. Let's go through the details.

Python App

We will create two Python apps:

• extract_feature.py
  We start by taking the shoes images (I have populated 100 images) and put in the Static folder. It would be the dataset of images. We then extract the features from each image and store the features (feature.pkl) in Feature folder.Run with command python extract_feature.py

• server.py 
  A user must submit a query image to our server via Facebook Messenger Bot platform. We then take a query image and extract the features from it.
  These features from the query image are compared with the features of the images that we already indexed in our dataset (in our case is pickle feature.pkl).
  The results are then sorted by relevancy and returns top 5 results in JSON format.
  
  Run the server with command python server.py

NodeJS App

We will create NodeJS ShoesBotApp.js to handle user's query when submitting the image and response back with the relevant top 5 images.

The following snippet codes do the job.

if (event.message && event.message.attachments) {

    if (event.message.attachments[0].type === "image") {

        let imageURL = event.message.attachments[0].payload.url;

        console.log(imageURL);



        let options = {

            host: host,

            path: '/img?url=' + imageURL,

            headers: {

                'Authorization': 'Basic ' + new Buffer(uid + ':' + pwd).toString('base64')

            }

        };



        https.get(options, function(res) {

            let body = "";

            res.on('data', function(data) {

                body += data;

            });

            res.on('end', function() {

                let result = JSON.parse(body);

                ArrayData = [];



                for (let k = 0; k < result.result.distances.length; k++) {

                    let image_url = result.result.img_path[k];

                    image_url = image_url.split('\\')[1];



                    image_url = 'https://xxxxtrial.dispatcher.hanatrial.ondemand.com/img/' + image_url;

                    console.log(image_url);



                    ArrayData.push({

                        'title': 'Product',

                        'subtitle': 'Sub Title',

                        'image_url': image_url,

                        "buttons": [{

                            "type": "web_url",

                            "url": 'https://dummy.com',

                            "title": 'Detail Product'

                        }]

                    });

                }



                if (ArrayData.length > 0) {

                    sendTextMessage(sender, 'Showing the result...', function(returnValue) {

                        groups = ArrayData.map(function(e, i) {

                                return i % chunk_size === 0 ? ArrayData.slice(i, i + chunk_size) : null;

                            })

                            .filter(function(e) {

                                return e;

                            });

                        for (let p = 0; p < groups.length; p++) {

                            sendMsg(groups[p], sender, function(returnValue) {});

                        }

                    });

                } else

                    sendTextMessages(sender, aritem, false, 0);

            })

            res.on('error', function(e) {

                console.log("Got error: " + e.message);

            });

        });



    }

}

Run the app with command node ShoesBotApp.js. The app runs on port 5000.

HTML5 App on SAP Cloud Platform

The final part is to create an HTML app to host the image files so it can be accessible by the bot.

• Upload the shoes images to SAP Web IDE by creating the dummy SAPUI5 app


• Deploy it to SAP Cloud Platform.

• I have set the authenticationMethod to None in neo-app.json so the image file can be accessible by the bot from the app URL: https://fd-pxxxxxxtrial.dispatcher.hanatrial.ondemand.com/img/i52.jpg.

That's all that we need to do and now you can open the Facebook Messenger app to test the bot. You can find the complete source code here.

We can enhance the app and put another function for the image segmentation to recognize and zoom in the image of shoes itself. You can do this by creating another Python app or you can also use SAP Leonardo Framework.

References

• Digital Transformation for SMBs – the Intelligent Enterprise


• The complete guide to building an image search engine with Python and OpenCV


• Facebook Messenger OCR Receipt Bot