NLP, NLU, and NLG
Images used in my articles are Properties of the Respective Organisations and are used here solely for Reference, Illustrative and Educational Purposes Only. [Images Source: Google (Aside from some images, in whose case, the source is specifically mentioned below the image)]
We’ve understood what Natural Language Processing, (NLP) is right? But we haven’t understood much about what Natural Language Understanding (NLU), and Natural Language Generation (NLG) are. Let’s understand both of them, along with NLP in this article.
This article will be divided into 3 Parts-
- NLP
- NLU
- NLG
Natural Language Processing (NLP)
Natural Language Processing (NLP) refers to the branch of Computer Science, and more specifically, the branch of Artificial Intelligence (AI)concerned with giving computers the ability to understand the text and spoken words in much the same way human beings can.
A computer program can understand human language as it is spoken and written — which is known as Natural Language.
It is the technology that is used by machines to understand, analyze, manipulate, and interpret human languages.
It helps in performing tasks like —
- Translation
- Automatic Summarisation
- Speech Recognition
etc.
Advantages of NLP
- NLP helps users to ask questions about any subject and get a direct response within seconds.
- NLP offers exact answers to the question means it does not offer unnecessary and unwanted information (Filler Words)
- NLP helps computers to communicate with humans in their languages.
- It is very time efficient.
- Most companies use NLP to improve the efficiency of documentation processes, accuracy of documentation, and identify the information from large databases.
Disadvantages of NLP
- NLP may not show context, which makes it hard to understand.
- NLP is unpredictable.
- NLP has a limited function, and therefore NLP is built for a single and specific task only.
Subsets of NLP
There are 2 Subsets of NLP —
- Natural Language Understanding (NLU)
- Natural Language Generation (NLG)
Applications of NLP
There are a few ways we use NLP in the real life. Let’s take a look at a few of those.
Question Answering (Virtual Assistants):
- Automatically answering the questions asked by us humans in a Natural Language, the language which we understand.
Spam Detection
- Spam Detection is used to detect unwanted e-mails which are sometimes spam, or promotional, and filter them out of the main inbox. They are filtered and sent into another place, known as the junk/spam folder.
- The NLP sometimes may classify important e-mails as junk/promotional, because it’s not a perfect model, which means it can’t classify every single spam e-mail as spam, and every single important e-mail as important.
Sentiment Analysis
- It is used on applications, and social media like Twitter, Instagram, Facebook, etc to analyze the attitude, behavior, and emotional state of the sender.
- This application is implemented through a combination of NLP (Natural Language Processing) and Statistics by assigning the values to the text (positive, negative, or natural), and identifying the mood of the context (happy, sad, angry, etc.)
- This way the Model will know which tweets might be hate speech etc.
Language Translation
NLP can also translate speech or text from one Natural Language to another Natural Language, like how it’s being done here, from English to French.
Spelling Error Correction
Whenever you misspell some word while typing, the reason you’re getting that Red/blue underline, is because NLP has identified that you might have misspelled a word, and is suggesting a replacement word for it, usually the correctly spelled word.
And many more use cases, but that’s all we’re going to get to today because I don’t want the article to be too long :)
Ambiguities in NLP
NLP is great, but sometimes there is a percentage of uncertainty as well.
Look at these examples.
- Ex.1- John is looking for a Match.
In this case, does it mean that John is looking to get married, or does it mean that John is looking for a Cricket Match happening somewhere?
This is an example of Lexical Ambiguity — The confusion that exists in the presence of two or more possible meanings of the sentence within a single word.
- Ex.2- James went to Kat. She said “I am Hungry”
In this case, who is hungry? James, or Kat? Think about it.
This is an example of Referential Ambiguity — The Confusion that exists when you are referring to something using the pronoun.
- Ex.3- I saw my dad with a pair of binoculars.
Now, in this case, does the sentence mean that I saw my dad with the help of a pair of binoculars, or does it mean that I saw my dad while he was using a pair of binoculars?
This is an example of Syntactic Ambiguity — The Confusion that exists in the presence of two or more possible meanings within the sentence.
Now, let’s move on to NLU.
Natural Language Understanding (NLU)
- Natural Language Understanding (NLU) is a subset of Natural Language Processing (NLP), which involves transforming human language into a machine-readable format.
- With the help of NLU and ML, computers can automatically analyze data in seconds, saving businesses countless hours and resources when analyzing troves of customer feedback.
- NLU focuses on a machine’s ability to understand the natural, human language. NLU refers to how unstructured data is rearranged so that machines may “understand”, interpret and analyze it.
Before a computer can process unstructured text into a machine-readable format, first machines need to understand the specifics, and uniqueness of the human language. (Slang, context, sarcasm, etc)
Gartner defines NLU as “It is the comprehension by computers of the structure and meaning of human language (e.g., English, Spanish, Japanese), allowing users to interact with the computer using natural sentences” — Gartner
In other words, NLU is AI that uses computer software to interpret text and any type of unstructured data. NLU can digest a text, translate it into computer language and produce an output in a language that humans can understand.
The main focus of NLU is to make machines:
- Interpret the Natural Language
- Derive Meaning
- Identify Context
- Draw Insights
**The Above 4 Points are from the Source: BMC Blogs**
Why is NLU Important Now?
(Source: AIMultiple)
- NLU can be used as a tool that will support the analysis of an unstructured text. There are various ways that people can express themselves, and sometimes this can vary from person to person.
- Especially for personal assistants to be successful, an important point is the correct understanding of the user. NLU transforms the complex structure of the language into a machine-readable structure.
- This enables text analysis and enables machines to respond to human queries.
- The amount of unstructured text that needs to be analyzed is increasing. Computers can perform language-based analysis for 24/7 in a consistent and unbiased manner.
- Considering the amount of raw data produced every day, NLU and hence NLP are critical for efficient analysis of this data. A well-developed NLU-based application can read, listen to, and analyze this data.
Organisations leading in NLU
(Source: AIMultiple)
As in many emerging areas, technology giants take a big place also in NLU. Some startups as well as open-source API’s are also part of the ecosystem.
Microsoft
- Knowledge Exploration Service
- Language Understanding Intelligent Service (LUIS)
- Azure Translator APIs
- Dialogflow
- Translate API
- Cloud Natural Language API
IBM
- Watson Conversation Service
- Watson Tone Analyzer
Amazon
- Comprehend
- Lex
Open-source alternatives are:
- Facebook’s Wit.ai
- Rasa NLU
- FuzzyWuzzy
- PyNLPl
- Stanford CoreNLP
Natural Language Generation (NLG)
(Source: Marketing AI Institute)
Natural language generation (NLG) is the process of transforming data into natural language using AI.
NLG software does this by using AI models powered by ML and deep learning to turn numbers into natural language text or speech that humans can understand.
Chatbots, Voice Assistants, and AI blog writers (to name a few) all use natural language generation. NLG systems can turn numbers into narratives based on pre-set templates. They can predict which words need to be generated next (in, say, an email you’re actively typing). Or, the most sophisticated systems can formulate entire summaries, articles, or responses.
Applications of Natural Language Generation (NLG)
(Source: Marketing AI Institute)
The following are some of the most common applications of NLG
- Chatbots that automatically answer questions on websites.
- Voice assistants like Alexa or Siri that respond to commands.
- Machine translation tools that translate one language into another.
- Conversational AI assistants that use advanced NLG and NLU to carry on two-way conversations.
- Analytics platforms can use NLG to explain insights from your data in easy-to-understand language.
- AI blog writers for content creation can use a language model to automatically write anything from a sentence to an entire article.
- Sentiment analysis platforms use NLU to understand which language resonates with customers, then employ NLG to create messages they’re likely to respond to.
- AI-powered transcription tools use speech recognition to understand audio, then NLG to turn it into text.
- Narrative generation tools use structured data (often in the form of a spreadsheet) to automatically generate a text narrative.
How does NLG Work?
(Source: TechTarget)
NLG is a multi-stage process, with each step further refining the data being used to produce content with natural-sounding language. The six stages of NLG are as follows:
- Content analysis. Data is filtered to determine what should be included in the content produced at the end of the process. This stage includes identifying the main topics in the source document and the relationships between them.
- Data understanding. The data is interpreted, patterns are identified and its put into context. Machine learning is often used at this stage.
- Document structuring. A documented plan is created and a narrative structure is chosen based on the type of data being interpreted.
- Sentence aggregation. Relevant sentences or parts of sentences are combined in ways that accurately summarise the topic.
- Grammatical structuring. Grammatical rules are applied to generate natural-sounding text. The program deduces the syntactical structure of the sentence. It then uses this information to rewrite the sentence in a grammatically correct manner.
- Language presentation. The final output is generated based on a template or format the user or programmer has selected.
How is NLG Used?
(Source: TechTarget)
Natural language generation is being used in an array of ways. Some of the many uses include the following:
- Generating the responses of chatbots and voice assistants such as Google’s Alexa and Apple’s Siri;
- Converting financial reports and other types of business data into easily understood content for employees and customers;
- Automating lead nurturing email, messaging, and chat responses;
- Personalising responses to customer emails and messages;
- Generating and personalising scripts used by customer service representatives;
- Aggregating and summarising news reports;
- Reporting on the status of internet of things devices; and
- Creating product descriptions for e-commerce web pages and customer messaging.
Check out this YouTube video discussing what chatbots are, and how they’re used.
Redirections for Further Research Enthusiasts
Natural language generation — Wikipedia
Natural Language Generation (NLG): What it is & How it works (aimultiple.com)
A Quick Guide to Natural Language Generation (NLG) (phrazor.ai)
Before we end this Article, if you have read my previous article, you might have remembered that i’ve given 5 questions, and the answers to which, will be in this article, at the end, so here are those! Please attempt them first yourselves, and then check these out :)
Q1. Difference Between NLU & NLP.
You would’ve gotten the answer in this article :)
Q2. How do organisations benefit from using Conversational AI?
- Accuracy
- Efficiency
- Contactless Customer Service
- Upsell Opportunities
- Better Customer Experience
- Reducing Operational Costs
For further detailed explanation on each of the above points, refer to this article.
Q3.If the Input given by the Customer is Speech, then what is used to analyse the data?
Automated Speech Recognition (ASR) (Refer to my Previous Article)
Q4.What do we do in the Stage of Dialogue Management?
During this stage, Natural Language Generation (NLG), a component of NLP, forms an appropriate response. (Refer to my Previous Article)
Q5.What is one of the biggest challenges faced by Conversational AI?
Constantly Changing forms of Communication
Security and Privacy
Discovery and Adoption
(Refer to my Previous Article)
P.S- There’s no questions for this Article :)
Thanks for Reading, Happy Learning!
