___STEADY_PAYWALL___.

12 Mar 2018

Jarbas Guidelines for skill development 2 - Continuous Dialog

Mycroft makes it super easy to make simple conversations

Hey Mycroft, tell me a joke?
"Joke here"

Hey Mycroft, when will the world end?
"Wolfram answer here"

Hey Mycroft, what is the price of bitcoin?
"Bitcoin is worth 8814 u s dollars"

But did you notice it’s all single shot question/answers pairs?

What about continuous dialog, how do i handle questions that depend on what was said previously on my skill?

Hey Mycroft, How tall is John Cleese?

"John Cleese is 196 centimeters"

Hey Mycroft, Where's he from?

"He's from England"

The official way - Conversational Context

The favorite way to handle this is to use conversational context, on the second question if mycroft remembers the first question he will know what you are talking about

The way this works is that you can add an adapt keyword whenever you want, then it will be considered in the next question

Hey Mycroft, How old is John Cleese?

(mycroft thinking)
- i think he wants me to say how old something is
- john cleese is a person, i will trigger the person age intent

"John Cleese is 20 year old"

(mycroft thinking)
- we are talking about the person named John Cleese, i will remember it

Hey Mycroft, How tall is he?

(mycroft thinking)
- i think he wants me to say how tall something is
- is it a building or a person? which intent do i pick?
- oh, we were talking about a person name JohnCleese

"John Cleese is 196 centimeters"

If the context is not used in the next question mycroft will give it a lower priority, until after 5 questions the context is removed, making mycroft forget that and resume normal functionality

This will only work with adapt intents, padatious intents do not support this yet

You can learn more about conversational context in the official docs, they do a good job explaining how to use context

In the end every question is still considered like a single shot question/answers pair, we are just remembering info, this can get messy quick

Enabling and Disabling

Let’s try another approach

Hey Mycroft, Let's talk about John Cleese

(mycroft thinking)
- john cleese is a person, i will trigger the person size intent

"Ok, Ask me something about John Cleese"

(mycroft thinking)
- i think he next may ask me some questions related to this
- enabling john_age, john_cousin, john_dog, ..... , intents

 Hey Mycroft, How old is he?

(mycroft thinking)
- i think he wants me to say how old a person is
- John is the only person with intents enabled

"John Cleese is 20 year old"

This was a simple scenario, for sequential things like games with levels this can be very handy!

Here is a sample skill demonstrating sequential entry of konami code

class KonamiCodeSkill(MycroftSkill):
    def __init__(self):
        super(KonamiCodeSkill, self).__init__(name="KonamiCode")
        # UP UP DOWN DOWN LEFT RIGHT LEFT RIGHT B A
        self.counter = 0
        self.next_cheat = "up"

    def initialize(self):

        up_intent = IntentBuilder('KonamiUpIntent'). \
            require("KonamiUpKeyword").build()

        down_intent = IntentBuilder('KonamiDownIntent'). \
            require("KonamiDownKeyword").build()

        left_intent = IntentBuilder('KonamiLeftIntent'). \
            require("KonamiLeftKeyword").build()

        right_intent = IntentBuilder('KonamiRightIntent'). \
            require("KonamiRightKeyword").build()

        b_intent = IntentBuilder('KonamiBIntent'). \
            require("KonamiBKeyword").build()

        a_intent = IntentBuilder('KonamiAIntent'). \
            require("KonamiAKeyword").build()

        self.register_intent(up_intent, self.handle_up_intent)
        self.register_intent(down_intent, self.handle_down_intent)
        self.register_intent(left_intent, self.handle_left_intent)
        self.register_intent(right_intent, self.handle_right_intent)
        self.register_intent(b_intent, self.handle_b_intent)
        self.register_intent(a_intent, self.handle_a_intent)
        self.disable_intent('KonamiDownIntent')
        self.disable_intent('KonamiLeftIntent')
        self.disable_intent('KonamiRightIntent')
        self.disable_intent('KonamiBIntent')
        self.disable_intent('KonamiAIntent')

    def handle_up_intent(self, message):
        self.speak_dialog("up")
        self.counter += 1
        if self.counter == 2:
            self.disable_intent('KonamiUpIntent')
            self.enable_intent("KonamiDownIntent")
            self.counter = 0
            self.next_cheat = "down"

    def handle_down_intent(self, message):
        self.speak_dialog("down")
        self.counter += 1
        if self.counter == 2:
            self.disable_intent('KonamiDownIntent')
            self.enable_intent("KonamiLeftIntent")
            self.counter = 0
            self.next_cheat = "left"

When you say “up” , mycroft will disable the “up” intent and activate the “down” intent, you effectively stopped an answer ( “up” ) and allowed another ( “down” )

This is a good use case when you need to block things from triggering and allow new ones to trigger after, conversational context does not allow you to stop old intents from triggering

the con? Conversational context wears out, in this model if you stop using mycroft it will be stuck there, this may be desired for a permanent setting change, but if you leave mid way you may have screwed up any further interaction

Converse Method

Before we go any further let’s try to understand a little better how things happen

  • Mycroft receives an utterance
  • Mycroft asks the active skills if they want to handle the utterance
  • If no active skill consumed the utterance Mycroft checks adapt for an intent
  • If no adapt intent triggered Mycroft will check the Fallback skills
  • Padatious intents are considered the first fallback skill
  • If a skill triggered it becomes active

pic

Since our last used skill became active, it can reset the intents!

This is accomplished by using the converse method, if this method returns True the utterance handling cycle will end there, if it returns False, the next active skill is given a shot at it

Skills become inactive after 5 minutes

Hey Mycroft, How tall is John Cleese?

(mycroft thinking)
- i think he wants me to say how tall something is
- is it a building or a person? which intent do i pick?
- john cleese is a person, i will trigger the person size intent

"John Cleese is 196 centimeters"

 Hey Mycroft, How old is he?

(mycroft thinking)
- let me check if the JohnInfo skill wants to do something

"John Cleese is 20 year old"

(mycroft thinking)
- JohnInfo skill handled this question, don't need to do anything

Converse method is crucial for some use cases, let’s imagine a skill that turns mycroft into a parrot, you need to capture all utterances and not trigger intents

The way to do this is in the converse method, we check if the user asked to stop parroting, in this case we let the utterance go because the stop intent will trigger, else we speak the utterance back

  def converse(self, utterances, lang="en-us"):
    if self.parroting:
        # check if stop intent will trigger
        for stop in self.stop_words:
            if stop in utterances[0]:
                return False
        # if not parrot utterance back
        self.speak(utterances[0], expect_response=True)
        return True
    else:
        return False

Going back to the konami code, if the user says the wrong thing we may want to reset the konami code intents back to “up”, in this case it is a passive ability, we just reset intents and do not consume the utterance, therefore we always return false

def converse(self, utterances, lang="en-us"):
    if self.next_cheat not in utterances:
        self.enable_intent('KonamiUpIntent')
        self.disable_intent('KonamiDownIntent')
        self.disable_intent('KonamiLeftIntent')
        self.disable_intent('KonamiRightIntent')
        self.disable_intent('KonamiBIntent')
        self.disable_intent('KonamiAIntent')
        self.next_cheat = "up"
        self.counter = 0
    return False

There is a pending PR to allow a skill to know when it was kicked from the active list, this means the user left after 5 minutes of inactivity, since the skill is no longer active we abort the konami code entry sequence so everything works normally

def on_deactivate(self):
    """
    Invoked when the skill is removed from active skill list
    """
    self.enable_intent('KonamiUpIntent')
        self.disable_intent('KonamiDownIntent')
        self.disable_intent('KonamiLeftIntent')
        self.disable_intent('KonamiRightIntent')
        self.disable_intent('KonamiBIntent')
        self.disable_intent('KonamiAIntent')
        self.next_cheat = "up"
        self.counter = 0

This will allow you to do things like passive skills that are always active, for example always mutating an utterance

def on_deactivate(self):
    # reactivate to keep skill always active
    self.make_active()

def converse(self, utterances, lang="en-us"):
    if utterances[0] != self.last_mutation:
        self.last_mutation = self.mutate(utterances[0])
        self.emitter.emit(Message("recognizer_loop:utterance", {"utterances": [self.last_mutation])
        return True
    return False

Other examples would be monitoring utterances and setting contexts, or displaying them in a web UI, or training a chatbot…

the same could be achieved by setting a listener to recognizer_loop:utterance, but this way you can consume the utterance, which otherwise you could not do

Get response - The other official way

By doing some clever stuff with the converse method mycroft devs added a get_response method

Inside an intent you may want to get the next utterance as a string, for example to clarify something you did not understand or ask for more information

 Hey Mycroft, How old is he?

(mycroft thinking)
- i think he wants me to say how old a person is
- trigger HowOldIntent

"Which person's age do you want to know?"

John Cleese

"John Cleese is 20 years old"

this can be done very easily inside an intent handler

person = self.get_response('what.person.are.we.talking.about')
if person:
    age = self.get_age(person)
    self.speak(person + ' is ' + age + ' years old')

Intent Layers

it can also become messy fast when you enable and disable lots of intents, i made an helper class to enable and disable intents for you!

This depends on PR#860

from mycroft_jarbas_utils.intents.layers import IntentLayers

def initialize():
    layers = [["KonamiUpIntent"], ["KonamiUpIntent"], ["KonamiDownIntent"], ["KonamiDownIntent"],
            ["KonamiLeftIntent"], ["KonamiRightIntent"], ["KonamiLeftIntent"], ["KonamiRightIntent"],
            ["KonamiBIntent"], ["KonamiAIntent"]]

    self.layers = IntentLayers(self.emitter, layers)

each layer can have any number of intents, you just decide what layers is active at any time, this is usefull when you want to enable several intents at once

to activate next layer/state -> self.layers.next()
to activate previous layer/state -> self.layers.previous()
to activate layer/state 0 -> self.layers.reset()
to get current layer/state -> state = self.layers.current_layer

to go directly to a layer do -> self.layers.activate_layer(layer_num)

on converse -> parse intent/utterance and manipulate layers if needed (bad sequence)

I like using IntentLayers for games, check the moon game example

Once the intent data PR is merged you can use the IntentParser helper class in converse method to know if a intent from this skill will trigger, this is super useful to decide if we are going to consume the utterance or not, this is also used in the moon game skill

Currently padatious is not available for this kind of interaction, i will be expanding the intent enabling/disabling to also support it, conversational context is a bit harder to do but will eventually also be supported, in a next post i will talk about using padatious in continuous conversations and a local listener with limited commands!

Liked this and want more?

Become a patron

Donate on Paypal

Thank You For Reading

This page is not about me, we need open source AI, for that reason i support mycroft, in this website you can find my contributions

These include skills, helper packages, forks, lots of Pull Requests, blog posts and videos!

If you find my work useful consider becoming a supporter.

Need help? - ask me

Need someone to hack away at mycroft? - hire me

jarbasai@mailfence.com

Become a patron

Please Donate To Bitcoin Address: [[address]]

Donation of [[value]] BTC Received. Thank You.
[[error]]
comments powered by Disqus