Project Description


I wanted to see whether it was possible to write a bot to be able to answer questions on my data.
At the time I was also starting to explore NLP with services like Google's DialogFlow and wanted to try this out.

What I did


The project involved many components, as it required an understanding for several parts, C#, several REST API's, NLP/AI and Qlik.
To start I needed data, so I simply built a quick app on Qlik Desktop to scrape some data from Wikipedia.
I chose something basic like Population per country. This got pulled in and visualized on the app.
I then had to create the .NET application that could speak to Qlik Desktop API....this was no easy feat due to the lack of documentation.
So having spent MANY MANY nights on the forums chatting to people around the world...
[Community]
[You don't get to the top of the leader board without keyboard blood, sweat and tears!]


I eventually figured out how to connect to the Qlik Desktop API (worked a bit different to the Qlik Sense Enterprise Edition).
Once I was able to send requests and retrieve information one simply had to connect to some kind of communication channel with the user.
Having investigated Telegram, WhatsApp, Slack etc I found to be Telegram great, well documented, well written and of course very free.

Telegram was much easier and had a lot more support for it, so the design for this went faster.
After connecting Telegram and Qlik I still sat with the problem of understanding what the user was asking...
If you couldn't translate "normal" language into something computer's could understand then it means that
the user would simply send through robotic instructions...so of course [entering the room] was Google's dialogflow (also no charges).
With Dialogflow you simply had to build an intent and train the model with it.
Although this seems very vast with all the possibilities, in reality my app would only have 2 questions (what is the population and what is the weather there).
The weather was another separate free API from http://api.openweathermap.org (which works really well).
Being able to add something like the weather meant the bot started to feel a bit more complete, especially when adding some referral words like "there".
For example


This was the whole app essentially, a user can communicate via 'normal' language through their phone, asking the dataset some questions.
DialogFlow will analyze the intent and parameters, and pass this back to the .NET App.
.NET will send this on to qlik and create the filters onto the Qlik app, which will result in a value.
The .NET app will return the values from Qlik and finally inform the user.
This technology seems limitless with possibilities, as sales staff can quickly ask their apps, "What was ABC sales last year?" or
"How many products did they sell?", whilst still sitting in the meeting.

So in summary, the Telegram message received from the user, gets sent to the .NET App which sends it off to Google to
identify an intent. Once the intent is established the details can be sent to the Qlik API which returns the data
and finally closing the loop - gets sent to the Telegram user again.
An image is shown below of the complete process flow of the application.

At the bottom of the page is some conversations/examples held with the "Bot".

Also something interesting

What's also interesting is that I would log every request sent through Telegram to the bot.
In this way you don't just analyze the main data, but now data built from data.
Below is a word cloud which shows the most common words and phrases the users were typing for requests.
If I noticed that there was a phrase used very often that I had not trained the model on yet then
iteratively I would update the model to become more and more intuitive and user-friendly.



Overview





Technology involved


Examples














Created by RH [2020]