Identify text
Set up the backend
If you haven't already done so, run amplify init
inside your project and then amplify add auth
(we recommend selecting the default configuration).
Run amplify add predictions
and select Identify. Then use the following answers:
1? What would you like to identify? (Use arrow keys)2❯ Identify Text3 Identify Entities4 Identify Labels5 Learn More6
7? Would you also like to identify documents? Yes8? Who should have access? Auth and Guest users
NOTE: Due to a known issue, you will need to update your amplify backend configuration with the properties below before pushing to the cloud:
amplify/backend/identify-text-resource/parameters.json
1"identifyDoc": true,2"access": "auth | authAndGuest",3"format": "PLAIN | FORM | TABLE | ALL"
Now run amplify push
which will generate your amplifyconfiguration.json
and create resources in the cloud. You can now either add this to your backend or skip and add more features to your app.
Services used: Amazon Rekognition (default for plain text) and Amazon Textract (default for documents)
Working with the API
Detect text in an input image. Input can be sent directly from the browser or an Amazon S3 key from project bucket.
1import { Predictions } from '@aws-amplify/predictions';2
3const response = await Predictions.identify({4 text: {5 source: {6 file7 }8 }9});10console.log({ response });
Identify image stored in Amazon S3
1import { Predictions } from '@aws-amplify/predictions';2
3const response = await Predictions.identify({4 text: {5 source: {6 key: pathToPhoto,7 level?: 'guest' | 'private' | 'protected', //optional, default is configured on Storage category8 }9 }10})11console.log({ response });
The following options are independent of which
source
is specified. For demonstration purposes we will reference afile
but it can be an S3 Key as well.Predictions.identify({text : {...}})
can detect unstructured textPLAIN
, structured text from tablesTABLE
or text from formsFORM
.
Identify plain text
For detecting plain text, you can see the whole detected text, the lines detected, the position of each line of text, and each word.
1import { Predictions } from '@aws-amplify/predictions';2
3const response = await Predictions.identify({4 text: {5 source: {6 file7 },8 format: 'PLAIN'9 }10});11
12const {13 text: {14 fullText, // String15 lines, // Array of String ordered from top to bottom16 linesDetailed /* Array of objects that contains17 text, // String18 boundingBox: {19 width, // ratio of overall image width20 height, // ratio of overall image height21 left, // left coordinate as a ratio of overall image width22 top // top coordinate as a ratio of overall image height23 },24 polygon // Array of { x, y } coordinates as a ratio of overall image width and height25 */,26 words // Array of objects that contains { text, boundingBox, polygon}27 }28} = response;
Identify structured forms
For detecting structured forms (documents, tables, etc.) from an image, keyValues
will return a string of the entity found in the image as well as metadata such as selected checkboxes or the relative location in the image using a boundingBox
.
1import { Predictions } from '@aws-amplify/predictions';2
3const response = await Predictions.identify({4 text: {5 source: {6 file7 },8 format: 'FORM'9 }10});11
12const {13 text: {14 // same as PLAIN +15 keyValues // Array of { key: string, value: { text: string, selected: boolean}, polygon, boundingBox }16 }17} = response;
For example the below image would return keyValues
with "Test" or "Checked" as a key, and true
since they are selected. The location of these elements would be returned in the boundingBox
value.
Identify structured tables
For detecting structured tables from an image
1import { Predictions } from '@aws-amplify/predictions';2
3const response = await Predictions.identify({4 text: {5 source: {6 file7 },8 format: 'TABLE'9 }10});11
12const {13 text: {14 // same as PLAIN +15 tables: [16 {17 size: { rows, columns },18 table // Matrix Array[ Array ] of size rows19 // each element of the array contains { text, boundingBox, polygon, selected, rowSpan, columnSpan}20 }21 ]22 }23} = response;
For detecting tables and forms on the image just select format "ALL"
1import { Predictions } from '@aws-amplify/predictions';2
3const response = await Predictions.identify({4 text: {5 source: {6 file7 },8 format: 'ALL'9 }10});11
12const {13 text: {14 // same as PLAIN + FORM + TABLE15 }16} = response;