ThingTalk is the programming language that lies at the core of any virtual assistant powered by Genie technology. It provides a higher level abstraction for connecting IoT devices, web services, and database queries, while hiding the details of configuration and networking. This document presents a brief guide to ThingTalk for Genie developers.
A ThingTalk program is composed of statements, terminated by a semicolon. Each statement connects one or more calls to functions in Thingpedia, connected by =>
.
Here is an example of ThingTalk program which gets a joke and shows it to the user:
@com.icanhazdadjoke.get();
The program invokes the get
function from the Dad Jokes service which returns a random joke. The result is not passed to any other function, so the program shows the result to the user.
The function we're calling is declared in Thingpedia with the syntax:
class @com.icanhazdadjoke {
query get(out text: String,
out joke_id: Entity(com.icanhazdadjoke:id));
}
This indicates that the function takes no input parameters, but has two output parameters, "text" and "joke_id". You can learn more about the ThingTalk syntax to declare classes in the Skill Manifest Guide.
You can issue commands in ThingTalk directly in Web Genie. To tell the system that your input is code and not a natural language sentence, type \t
with a space before the ThingTalk code.
In our first example, we have seen a query @com.icanhazdadjoke.get()
. It talks to the joke API and gets a random joke from it.
The query in a program returns data to the user. The data can also be used for the action to consume.
Instead of querying some data, a program can run an action function in Thingpedia, to perform some side-effects. For example, the following command sends a message to Slack:
@com.slack.send();
Queries and actions can be combined using the =>
operator. In that case, the result of the query can be used to invoke the action. For example, instead of being notified inside Almond, you can choose to send a joke to Slack as follows:
@com.icanhazdadjoke.get() => @com.slack.send();
So far, we have not used any parameters for functions in our examples. In the following we will introduce how to handle parameters in ThingTalk.
Parameters in ThingTalk are passed by keyword, using the names of the parameters defined in Thingpedia. Let's look at one example, using the Weather API. The API is declared in Thingpedia as:
class @org.thingpedia.weather {
monitorable query current(in req location: Location
out temperature: Measure(C),
out wind_speed: Measure(mps),
out humidity: Number,
out cloudiness: Number,
out fog: Number,
out status: Enum(raining,cloudy,sunny,snowy,sleety,drizzling,windy) ,
out icon: Entity(tt:picture));
}
With that declaration, we can use the API as:
@org.thingpedia.weather.current(location=new Location("palo alto"));
The current
function for @org.thingpedia.weather
has an input parameter location
, which specifies where we want the weather for. In this example, we set location=new Location("palo alto")
.
For every device in Thingpedia, you can find the list of input parameters in the device details page (Almond 1,99 version).
You can find the syntax of constructors for each type in the ThingTalk reference.
In addition to constant values, we can also pass the value returned by previous functions, by specifying the name of one output parameter of the previous function.
@com.foxnews.get() => @com.slack.send(channel="general", message=title);
In this example, the message sent to slack is the value of the output parameter title
from Fox News.
Parameters can be passed from queries to actions, and from queries to other queries. For example:
@com.foxnews.get() => @com.yandex.translate.translate(target_language="ch", text=title);
In this case, each news from Fox News is combined with the translate
query from Yandex Translate, passing text=title
. In practice, this program means that get the title of news from Fox News, translate it to Chinese, and show the translated title to the user.
In addition to calling APIs like Weather and Jokes, ThingTalk can be used to perform complex queries on databases.
To perform a database query, we first need to declare the database table in Thingpedia; this is done by declaring a special type of query
API that statisfies the following requisites:
list
keyword)id
parameter whose type is an Entity
with the same name as the query.For example, here is the declaration of the "restaurant" table in Yelp:
class @com.yelp {
entity restaurant #[name="Restaurant on Yelp"];
entity restaurant_cuisine #[name="Yelp Cuisines"];
list query restaurant(out id: Entity(com.yelp:restaurant),
out image_url: Entity(tt:picture),
out link: Entity(tt:url),
out cuisines: Array(Entity(com.yelp:restaurant_cuisine)),
out price: Enum(cheap,moderate,expensive,luxury),
out rating: Number,
out review_count: Number,
out geo: Location,
out phone: Entity(tt:phone_number));
}
(Database tables can also optionally be monitorable
, if monitoring for changes is meaningful.)
With this declaration, we can query the table in the simplest form as:
@com.yelp.restaurant();
This command returns the full list of restaurants. Because the list might be very long, the agent only shows the top results.
You should observe that this looks very similar to just calling an API on Yelp. Indeed, ThingTalk is designed to unify databases and APIs and make the distinction transparent to the user. This helps with translating from natural language, as the user can refer to "restaurants" without knowledge of implementation details.
In the following, you will learn how to refine the query to return only a subset of the whole table.
The most important operation in a database is selection: choosing a subset of the rows to return. In ThingTalk this is accomplished with a filter:
@com.yelp.restaurant() filter contains(cuisines, "latin"^^com.yelp:restaurant_cuisine("Latin American"));
This program returns the restaurants that serve Latin American food.
The filter is specified with a comma following the corresponding stream or query, followed by a boolean predicate that uses the output parameters. In this example, we filter on the cuisines
parameter; the program will be triggered only if the list of cuisine contains an entity of type com.yelp:restaurant_cuisine
with ID latin
(The string "Latin American" is the name of the entity displayed to the user).
Multiple filters can be combined with &&
(and) and ||
(or):
@com.yelp.restaurant() filter contains(cuisines, "latin"^^com.yelp:restaurant_cuisine("Latin American")) && geo == $location.current_location;
The program returns the restaurants that serve Latin American food around the same location as the user. $location
is a special variable that allows to refer to the user's locations (current location, home address, and work address).
Filters are automatically normalized when possible. For example, the following program, that returns either cheap restaurants or moderately priced restaurants:
@com.yelp.restaurant() filter price == enum(cheap) || price == enum(moderate);
Would be automatically normalized to:
@com.yelp.restaurant() filter in_array(price, [enum(cheap), enum(moderate)]);
For the full list of predicates supported by ThingTalk, see the ThingTalk reference.
By default, the query returns all fields in the table (output parameters of the query function). To return only a specific field, one uses a projection.
The syntax of a projection is:
[<field1>, <field2>, ...] of <query>;
For example, the question "how expensive are restaurants around" can be expressed as:
[price] of @com.yelp.restaurant() filter geo == $location.current_location;
When the agent answers this question, it will show the name of the restaurant (the id
parameter, always present in the reply) as well as the price range. Because the id
parameter is always present, you should not specify it explicitly.
One can project on multiple fields, for example, "where are restaurants that serve Italian food, and how expensive are they" can be expressed as:
[price, geo] of @com.yelp.restaurant() filter contains(cuisines, "italian"^^com.yelp:restaurant_cuisines("Italian"));
The order of fields in the projection does not matter and is normalized automatically.
Note: the operation order of projection vs selection is normalized (first selection then projection), so parenthesis are also not significant.
In addition to existing fields, you can compute new value using an arithmetic expression for every row in the table, by specifying the expression in a projection clause:
[<expr>] of <query>;
For example, the following computes the square of the product of the review count and the average review (the total number of stars ever awarded to the restaurant):
[(review_count * review)] of @com.yelp.restaurant();
And the following computes the distance of the restaurant to the user's current location:
[distance(geo, $location.current_location)] of @com.yelp.restaurant();
The result of a computation function is stored in a variable with the same name as the function (so in the example, a variable called distance
). If the computation uses an arithmetic expression and not a function, the result is stored in the variable value
.
Computation functions and expressions can also be used in filters. For example, the following finds "restaurants within 3 miles":
@com.yelp.restaurant() filter distance(geo, $location.current_location) <= 3mi;
To sort the result of a query, one uses the sort
keyword:
sort(<expr> <direction> of <query>);
For example, the following returns all restaurants by number of reviews, from least to most:
sort(review_count asc of @com.yelp.restaurants());
The direction can be asc
or desc
.
The result of a query can be limited by index using the slice operator, which has two forms:
<query> [<idx1>, <idx2>, ...];
<query> [<base> : <delta>];
The first form returns the elements with indices idx1, idx2, etc. The second form returns delta elements starting from base. Indices are 1-based (the first element is 1). Negative indices are taken from the end of the list instead of the beginning.
Indexing can be combined with sorting. For example, the following returns the restaurant with the most reviews:
sort(review_count desc of @com.yelp.restaurants())[1];
The previous program is equivalent to the following:
sort(review_count asc of @com.yelp.restaurants())[-1];
(the latter program sorts in reverse order and then chooses the last element in the sort).
When applying an index to a sort operation, negative indices are normalized to positive ones.
The following program returns the restaurant with the second largest number of reviews:
sort(review_count desc of @com.yelp.restaurants())[2];
And the following program returns the top 3 restaurants with most reviews:
sort(review_count desc of @com.yelp.restaurants())[1 : 3];
As before, in addition to a single field, you can specify a computed expression as the value to sort on, so for example the following program finds the closest restaurant:
sort(distance(geo, $location.current_location) asc of @com.yelp.restaurants())[1];
Aggregation is a way to compute a single result for all rows in the table. Aggregation is expressed as:
<aggr-op>(<field> of <query>);
<aggr-op>
can be sum
, min
, max
, avg
and count
.
For example, the following computes the average number of reviews of restaurants nearby:
avg(review_count of @com.yelp.restaurant() filter geo == $location.current_location);
Note the difference between aggregation and ranking. The following computes the minimum number of reviews of restaurants nearby:
min(review_count of @com.yelp.restaurant() filter geo == $location.current_location);
Compare that to the following instead, which returns the restaurant with the minimum number of reviews:
sort(review_count desc of @com.yelp.restaurant() filter geo == $location.current_location))[1];
The former returns a single number, while the latter returns the full restaurant row (with name, location, etc.).
For the count
operator, aggregation can be specified with a field (which returns the number of distinct values) or with no fields (which returns the number of rows). For example, the following returns the number of restaurants around:
count(@com.yelp.restaurant() filter geo == $location.current_location);
In addition to specifing a single value in a filter, a program can specify an entire query inside a filter, which is then known as a "subquery". For every row of in the main query, the subquery is evaluated to compute the comparison value for the filter.
There are two forms of subqueries. The first form uses a comparison:
<query1> filter <expr> <comp-op> any ( <query2> )
In that case, the filter is true if <expr>
compares correctly to any value returned by <query2>
.
For example, the following returns "all songs in the latest two albums by Taylor Swift":
@com.spotify.song() filter album == any (
sort(release_date desc of @com.spotify.album(), artist == null^^com.spotify.artist("taylor swift"))[1 : 2]
);
The alternative form of subquery is a direct existential subquery:
<query1> filter any ( <query2> )
In the frist case, the filter is true if <query2>
returns any result at all.
For example, the following program returns restaurants that have had at least review in the last month:
@com.yelp.restaurant() filter any (@com.yelp.review(restaurant=id) filter date >= $now - 1mon);
Note that the parameter id
is passed from the restaurant query to the review query.
Subqueries use existential quantification only. The effect of universal quantification can be obtained by adding negation.
So we far, we seen how to represent a single command as a ThingTalk program. To represent a whole conversation, composed of multiple turns, ThingTalk introduces the notion of a formal dialogue state The dialogue state represents the accumulated state of a conversation, and contains the ThingTalk statements corresponding to the previous commands, as well as their results.
At any time, the agent keeps track of the current dialogue state. There is a state immediately before the agent speaks, immediately before the user speaks, and immediately after the user speaks. The agent advances the state by adding any ThingTalk statement corresponding to the new commands that the user issues, and recording the result of executing those statements.
A dialogue state has syntax:
$dialogue <policy>.<dialogue-act>(<params>);
<history-item>*
The policy is an identifier of a specific dialogue policy to use to interpret the state. At the moment, there is only one such policy: @org.thingpedia.dialogue.transaction
. The dialogue act is a high-level representation of the purpose of the last command issued. In the state immediately before the user speaking, the dialogue act represents what the agent says. In the state immediately after the user speaks, the dialogue act represents what the user just said. The list of dialogue acts for the transaction policy is here.
Each <history-item>
consists of a ThingTalk statement, optionally followed by a #[results]
or #[error]
annotation, indicating this statement was executed, with the given result or error, or by a #[confirm]
annotation, indicating this statement is about to be executed, with the given confirmation level. #[confirm]
can be enum(proposed)
(the statement was just proposed by the agent), enum(accepted)
(the statement was just issued by the user) or enum(confirmed)
(the statement was first issued by the user and later confirmed again by the user).
For example, the following dialogue state represents the agent asking if the user is interested in restaurants of a specific price, after the user asked for a restaurant nearby:
$dialogue @org.thingpedia.dialogue.transaction.sys_search_question(price_range);
@com.yelp.restaurant() filter geo == new Location(...)
#[results=[{
{ id="..."^^com.yelp:restaurant("Ramen Nagi"), image_url="https://s3-media3.fl.yelpcdn.com/bphoto/OKCXWIEFIkNdvkqETl0Bqw/o.jpg"^^tt:picture, cuisines=["ramen"^^com.yelp:restaurant_cuisine("Ramen"), "noodles"^^com.yelp:restaurant_cuisine("Noodles")], price=enum(moderate), rating=4.5, review_count=1625, geo=new Location(37.445523, -122.1607073261), phone=""^^tt:phone_number },
...
]];
(note that the location value was normalized to a specific location, which is omitted for clarity)
The following dialogue state represents the agent offering to search for a fine-dining restaurant in response to the same query:
$dialogue @org.thingpedia.dialogue.transaction.sys_propose_refined_query;
@com.yelp.restaurant() filter geo == new Location(...)
#[results=[{
{ id="..."^^com.yelp:restaurant("Ramen Nagi"), image_url="https://s3-media3.fl.yelpcdn.com/bphoto/OKCXWIEFIkNdvkqETl0Bqw/o.jpg"^^tt:picture, cuisines=["ramen"^^com.yelp:restaurant_cuisine("Ramen"), "noodles"^^com.yelp:restaurant_cuisine("Noodles")], price=enum(moderate), rating=4.5, review_count=1625, geo=new Location(37.445523, -122.1607073261), phone=""^^tt:phone_number },
...
]]
@com.yelp.restaurant() filter geo == new Location(...) && price==enum(luxury)
#[confirm=enum(proposed)];
Primitive templates are composable snippets of ThingTalk with parameters. They are expressed with the syntax:
<type> (<params>) := <statement>;
For example:
query (p_location : Number) := @org.thingpedia.weather.current(location=p_location);
stream (p_keyword : String) := monitor(@com.foxnews.get() filter title =~ p_keyword);
Primitive templates, annotated with natural language expressions, are used to provide composable commands for devices in Thingpedia, as detailed in their guide.
In addition to executing commands immediately, ThingTalk allows a program to stay active in the background and trigger the rest of the program at a future time when a certain condition is met.
It does so with a stream. The stream is a clause that is specified before the query or action in a program, separated by =>
. The supported stream types include monitor stream and timer.
Query functions in Thingpedia can be monitored and turned into a stream, called monitor stream. A monitor stream will trigger the rest of the program once the returned data from the query changes.
For example, one can set up a monitor for Fox News with the following command
monitor(@com.foxnews.get());
This program starts a monitor on get
query function from Fox news produces a notification to the user every time Fox News publishes something new. Note that in this example, there is no query part. When the monitor gets triggered, it returns the result to the user, using the default action.
To support monitoring queries, a query must be declared as monitorable
in Thingpedia. For example, Fox News is declared as:
class @com.foxnews {
monitorable list query get(out title: String,
out url: Entity(tt:url),
out author: String,
out description: String);
#[poll_interval=1h];
}
Monitorable queries have a #[poll_interval]
annotation that indicates how often the query is checked.
A monitor stream can also be applied to multiple queries connected by =>
:
monitor (@com.foxnews.get() => @com.bing.web_search(query=title));
In that case, Almond will monitor the combination of the two queries, that is, it will continuously query Bing based on the news title and notify if the search results change.
Note that the semantics of this command is different from the following one:
monitor(@com.foxnews.get()) => @com.bing.web_search(query=title);
Instead of monitoring changes of the search results, this program only monitors the changes on Fox News, and then runs the Bing search if it detects a new article.
In addition to operating on changes of data, programs can be fired at specific times using timer streams, using the syntax:
timer(base=$now, interval=1h) => @com.icanhazdadjoke.get();
This syntax creates a timer that starts now and fires every hour. The notation $now
indicates the current time: in ThingTalk, variables that start with $
refer to the context of the calling user (time, location, accounts, preferences, etc.)
Another form of timer triggers every day at a specific time:
attimer(time=new Time(8, 0)) => @com.icanhazdadjoke.get();
This syntax creates a timer that triggers every day at 8 AM.
Filters can also be applied in the stream clause. In that case, the stream triggers only when if the filter was previously false and is now true.
For example:
monitor(@thermostat.get_temperature()) filter value >= 70F;
The above continuously monitors the temperature, and triggers when the temperature was previously below 70 F, and is now above (i.e., when it crosses the threshold). Further changes of the temperature above 70 F (e.g. going from 71 to 72 F) do not cause additional notification, until the value goes below the threshold and then above again.
It is possible to obtain level trigger behavior for streams as well, in addition to the above specified edge trigger, by specifying the filter inside the monitor clause instead of outside. This is rarely useful though.