Connect machine learning services

You are currently viewing the legacy GraphQL Transformer documentation. View latest documentation

@predictions

The @predictions directive allows you to query an orchestration of AI/ML services such as Amazon Rekognition, Amazon Translate, and/or Amazon Polly.

Note: Support for adding the @predictions directive uses the s3 storage bucket which is configured via the CLI. At the moment this directive works only with objects located within public/.

Definition

The supported actions in this directive are included in the definition.

1directive @predictions(actions: [PredictionsActions!]!) on FIELD_DEFINITION
2enum PredictionsActions {
3  identifyText # uses Amazon Rekognition to detect text
4  identifyLabels # uses Amazon Rekognition to detect labels
5  convertTextToSpeech # uses Amazon Polly in a lambda to output a presigned url to synthesized speech
6  translateText # uses Amazon Translate to translate text from source to target language
7}

Usage

Given the following schema a query operation is defined which will do the following with the provided image.

Identify text from the image
Translate the text from that image
Synthesize speech from the translated text.

1type Query {
2  speakTranslatedImageText: String
3    @predictions(actions: [identifyText, translateText, convertTextToSpeech])
4}

An example of that query will look like:

1query SpeakTranslatedImageText($input: SpeakTranslatedImageTextInput!) {
2  speakTranslatedImageText(
3    input: {
4      identifyText: { key: "myimage.jpg" }
5      translateText: { sourceLanguage: "en", targetLanguage: "es" }
6      convertTextToSpeech: { voiceID: "Conchita" }
7    }
8  )
9}

A code example of this using the JS Library:

1import React, { useState } from 'react';
2import { Amplify, Storage, API, graphqlOperation } from 'aws-amplify';
3import awsconfig from './aws-exports';
4import { speakTranslatedImageText } from './graphql/queries';
5
6/* Configure Exports */
7Amplify.configure(awsconfig);
8
9function SpeakTranslatedImage() {
10  const [src, setSrc] = useState('');
11  const [img, setImg] = useState('');
12
13  function putS3Image(event) {
14    const file = event.target.files[0];
15    Storage.put(file.name, file)
16      .then(async (result) => {
17        setSrc(await speakTranslatedImageTextOP(result.key));
18        setImg(await Storage.get(result.key));
19      })
20      .catch((err) => console.log(err));
21  }
22
23  return (
24    <div className="Text">
25      <div>
26        <h3>Upload Image</h3>
27        <input
28          type="file"
29          accept="image/jpeg"
30          onChange={(event) => {
31            putS3Image(event);
32          }}
33        />
34        <br />
35        {img && <img src={img}></img>}
36        {src && (
37          <div>
38            {' '}
39            <audio id="audioPlayback" controls>
40              <source id="audioSource" type="audio/mp3" src={src} />
41            </audio>{' '}
42          </div>
43        )}
44      </div>
45    </div>
46  );
47}
48
49async function speakTranslatedImageTextOP(key) {
50  const inputObj = {
51    translateText: {
52      sourceLanguage: 'en',
53      targetLanguage: 'es'
54    },
55    identifyText: { key },
56    convertTextToSpeech: { voiceID: 'Conchita' }
57  };
58  const response = await client.graphql(
59    graphqlOperation(speakTranslatedImageText, { input: inputObj })
60  );
61  return response.data.speakTranslatedImageText;
62}
63function App() {
64  return (
65    <div className="App">
66      <h1>Speak Translated Image</h1>
67      <SpeakTranslatedImage />
68    </div>
69  );
70}
71export default App;

How it works

From example schema above, @predictions will create resources to communicate with Amazon Rekognition, Translate and Polly. For each action the following is created:

IAM Policy for each service (e.g. Amazon Rekognition detectText Policy)
An AppSync VTL function
An AppSync DataSource

Finally a resolver is created for speakTranslatedImageText which is a pipeline resolver composed of AppSync functions which are defined by the action list provided in the directive.

Actions

Each of the actions described in the @predictions definition section can be used individually, as well as in a sequence. Sequence of actions supported today are as follows:

identifyText -> translateText -> convertTextToSpeech
identifyLabels -> translateText -> convertTextToSpeech
translateText -> convertTextToSpeech