Identify text

The following APIs will allow you to identify text (words, tables, pages from a book) from an image.

For identifying text on iOS we use both AWS backend services as well as Apple's on-device Core ML Vision Framework to provide you with the most accurate results. If your device is offline, we will return results only from Core ML. On the other hand, if you are able to connect to AWS Services, we will return a unioned result from both the service and Core ML. Switching between backend services and Core ML is done automatically without any additional configuration required.

Set up the backend

If you haven't already done so, run amplify init inside your project and then amplify add auth (we recommend selecting the default configuration).

Run amplify add predictions, then use the following answers:

1? Please select from one of the categories below
2❯ Identify
3  Convert
4  Interpret
5  Infer
6  Learn More
7
8? What would you like to identify? (Use arrow keys)
9❯ Identify Text
10  Identify Entities
11  Identify Labels
12
13? Provide a friendly name for your resource
14    <Enter a friendly name here>
15
16? Would you also like to identify documents?
17    <Enter 'y'> 
18
19? Who should have access?
20  Auth users only
21❯ Auth and Guest users

Run amplify push to create the resources in the cloud.

Identify text from image

Amplify will make calls to both Amazon Textract and Rekognition depending on the type of text you are looking to identify (i.e. image or document).

If you are detecting text from an image you would send in .plain as your text format as shown below. Using .plain with PredictionsIdentifyRequest.Options() combines results from on device results from Core ML and AWS services to yield more accurate results.

1func detectText(_ image: URL) async throws -> [Predictions.IdentifiedWord]? {
2    do {
3        let result = try await Amplify.Predictions.identify(.text, in: image)
4        print("Identified text: \(result)")
5        return result.words
6    }  catch let error as PredictionsError {
7        print("Error identifying text: \(error)")
8        throw error
9    } catch {
10        print("Unexpected error: \(error)")
11        throw error
12    }
13  }
14}

1func detectText(_ image: URL) -> AnyCancellable {
2    Amplify.Publisher.create {
3        try await Amplify.Predictions.identify(.text, in: image)
4    }
5    .sink(receiveCompletion: { completion in
6        if case let .failure(error) = completion {
7            print("Error identifying text: \(error)")
8        }
9    }, receiveValue: { value in
10        print("Identified text: \(value)")
11    })
12}

To get results that utilize on-device capabilities (Core ML), without combining results from the backend, you can use the following to pass into the options argument of the Amplify.Predictions.identify function.

1let options = Predictions.Identify.Options(defaultNetworkPolicy: .offline)

Identify text in a document

Sending in .form or .table or .all will do document analysis as well as text detection to detect tables and forms in a document. See below for an example with .form.

1func detectDocumentText(_ image: URL) async throws -> Predictions.Identify.DocumentText.Result {
2    do {
3        let result = try await Amplify.Predictions.identify(
4            .textInDocument(textFormatType: .form), in: image
5        )
6        print("Identified document text: \(result)")
7        return result
8    } catch let error as PredictionsError {
9        print("Error identifying text in document: \(error)")
10        throw error
11    } catch {
12        print("Unexpected error: \(error)")
13        throw error
14    }
15}

1func detectDocumentText(_ image: URL) -> AnyCancellable {
2    Amplify.Publisher.create {
3        try await Amplify.Predictions.identify(
4            .textInDocument(textFormatType: .form), in: image
5        )
6    }
7    .sink(receiveCompletion: { completion in
8        if case let .failure(error) = completion {
9            print("Error identifying text in document: \(error)")
10        }
11    }, receiveValue: { value in
12        print("Identified text in document: \(value)")
13    })
14}