Page updated Jan 16, 2024

Identify text

Set up the backend

If you haven't already done so, run amplify init inside your project and then amplify add auth (we recommend selecting the default configuration).

Run amplify add predictions and select Identify. Then use the following answers:

? What would you like to identify? (Use arrow keys) ❯ Identify Text Identify Entities Identify Labels Learn More ? Would you also like to identify documents? Yes ? Who should have access? Auth and Guest users
1? What would you like to identify? (Use arrow keys)
2❯ Identify Text
3 Identify Entities
4 Identify Labels
5 Learn More
6
7? Would you also like to identify documents? Yes
8? Who should have access? Auth and Guest users

NOTE: Due to a known issue, you will need to update your amplify backend configuration with the properties below before pushing to the cloud:

amplify/backend/identify-text-resource/parameters.json

"identifyDoc": true, "access": "auth | authAndGuest", "format": "PLAIN | FORM | TABLE | ALL"
1"identifyDoc": true,
2"access": "auth | authAndGuest",
3"format": "PLAIN | FORM | TABLE | ALL"

Now run amplify push which will generate your amplifyconfiguration.json and create resources in the cloud. You can now either add this to your backend or skip and add more features to your app.

Services used: Amazon Rekognition (default for plain text) and Amazon Textract (default for documents)

Working with the API

Detect text in an input image. Input can be sent directly from the browser or an Amazon S3 key from project bucket.

import { Predictions } from '@aws-amplify/predictions'; const response = await Predictions.identify({ text: { source: { file } } }); console.log({ response });
1import { Predictions } from '@aws-amplify/predictions';
2
3const response = await Predictions.identify({
4 text: {
5 source: {
6 file
7 }
8 }
9});
10console.log({ response });

Identify image stored in Amazon S3

import { Predictions } from '@aws-amplify/predictions'; const response = await Predictions.identify({ text: { source: { key: pathToPhoto, level?: 'guest' | 'private' | 'protected', //optional, default is configured on Storage category } } }) console.log({ response });
1import { Predictions } from '@aws-amplify/predictions';
2
3const response = await Predictions.identify({
4 text: {
5 source: {
6 key: pathToPhoto,
7 level?: 'guest' | 'private' | 'protected', //optional, default is configured on Storage category
8 }
9 }
10})
11console.log({ response });

The following options are independent of which source is specified. For demonstration purposes we will reference a file but it can be an S3 Key as well. Predictions.identify({text : {...}}) can detect unstructured text PLAIN, structured text from tables TABLE or text from forms FORM.

Identify plain text

For detecting plain text, you can see the whole detected text, the lines detected, the position of each line of text, and each word.

import { Predictions } from '@aws-amplify/predictions'; const response = await Predictions.identify({ text: { source: { file }, format: 'PLAIN' } }); const { text: { fullText, // String lines, // Array of String ordered from top to bottom linesDetailed /* Array of objects that contains text, // String boundingBox: { width, // ratio of overall image width height, // ratio of overall image height left, // left coordinate as a ratio of overall image width top // top coordinate as a ratio of overall image height }, polygon // Array of { x, y } coordinates as a ratio of overall image width and height */, words // Array of objects that contains { text, boundingBox, polygon} } } = response;
1import { Predictions } from '@aws-amplify/predictions';
2
3const response = await Predictions.identify({
4 text: {
5 source: {
6 file
7 },
8 format: 'PLAIN'
9 }
10});
11
12const {
13 text: {
14 fullText, // String
15 lines, // Array of String ordered from top to bottom
16 linesDetailed /* Array of objects that contains
17 text, // String
18 boundingBox: {
19 width, // ratio of overall image width
20 height, // ratio of overall image height
21 left, // left coordinate as a ratio of overall image width
22 top // top coordinate as a ratio of overall image height
23 },
24 polygon // Array of { x, y } coordinates as a ratio of overall image width and height
25 */,
26 words // Array of objects that contains { text, boundingBox, polygon}
27 }
28} = response;

Identify structured forms

For detecting structured forms (documents, tables, etc.) from an image, keyValues will return a string of the entity found in the image as well as metadata such as selected checkboxes or the relative location in the image using a boundingBox.

import { Predictions } from '@aws-amplify/predictions'; const response = await Predictions.identify({ text: { source: { file }, format: 'FORM' } }); const { text: { // same as PLAIN + keyValues // Array of { key: string, value: { text: string, selected: boolean}, polygon, boundingBox } } } = response;
1import { Predictions } from '@aws-amplify/predictions';
2
3const response = await Predictions.identify({
4 text: {
5 source: {
6 file
7 },
8 format: 'FORM'
9 }
10});
11
12const {
13 text: {
14 // same as PLAIN +
15 keyValues // Array of { key: string, value: { text: string, selected: boolean}, polygon, boundingBox }
16 }
17} = response;

For example the below image would return keyValues with "Test" or "Checked" as a key, and true since they are selected. The location of these elements would be returned in the boundingBox value.

Image

Identify structured tables

For detecting structured tables from an image

import { Predictions } from '@aws-amplify/predictions'; const response = await Predictions.identify({ text: { source: { file }, format: 'TABLE' } }); const { text: { // same as PLAIN + tables: [ { size: { rows, columns }, table // Matrix Array[ Array ] of size rows // each element of the array contains { text, boundingBox, polygon, selected, rowSpan, columnSpan} } ] } } = response;
1import { Predictions } from '@aws-amplify/predictions';
2
3const response = await Predictions.identify({
4 text: {
5 source: {
6 file
7 },
8 format: 'TABLE'
9 }
10});
11
12const {
13 text: {
14 // same as PLAIN +
15 tables: [
16 {
17 size: { rows, columns },
18 table // Matrix Array[ Array ] of size rows
19 // each element of the array contains { text, boundingBox, polygon, selected, rowSpan, columnSpan}
20 }
21 ]
22 }
23} = response;

For detecting tables and forms on the image just select format "ALL"

import { Predictions } from '@aws-amplify/predictions'; const response = await Predictions.identify({ text: { source: { file }, format: 'ALL' } }); const { text: { // same as PLAIN + FORM + TABLE } } = response;
1import { Predictions } from '@aws-amplify/predictions';
2
3const response = await Predictions.identify({
4 text: {
5 source: {
6 file
7 },
8 format: 'ALL'
9 }
10});
11
12const {
13 text: {
14 // same as PLAIN + FORM + TABLE
15 }
16} = response;