Using TransferUtility

You are currently viewing the AWS SDK for Mobile documentation which is a collection of low-level libraries. Use the Amplify libraries for all new app development. Learn more

To make it easy to upload and download objects from Amazon S3, we provide a TransferUtility component with built-in support for background transfers, progress tracking, and MultiPart uploads. The Transfer Utility component set includes a Service called the TransferService, which monitors network connectivity changes. When the device goes offline, the TransferService will pause all ongoing transfers; when the device is back online, the Transfer Service will resume paused transfers.

Starting with version 2.7.0, the TransferService will not be automatically started or stopped by TransferUtility. You have to start TransferService manually from your application. A recommended way is to start the service upon Application startup, by including the following line in the onCreate method of your app's Application class.

1getApplicationContext().startService(new Intent(getApplicationContext(), TransferService.class));

This section explains how to implement upload and download functionality and a number of additional storage use cases.

Note: If you use the transfer utility MultiPart upload feature, take advantage of automatic cleanup features by setting up the AbortIncompleteMultipartUpload action in your Amazon S3 bucket life cycle configuration.

Upload a File

The following example shows how to use the TransferUtility to upload a file. Instantiate the TransferUtility object using the provided TransferUtility builder function. Use the AWSMobileClient to get the AWSConfiguration and AWSCredentialsProvider to pass into the builder. See Authentication for more details.

The TransferUtility checks the size of the file being uploaded and automatically switches over to using multi-part uploads if the file size exceeds 5 MB.

1import android.app.Activity;
2import android.content.Intent;
3import android.os.Bundle;
4import android.util.Log;
5
6import com.amazonaws.mobile.client.AWSMobileClient;
7import com.amazonaws.mobile.client.Callback;
8import com.amazonaws.mobile.client.UserStateDetails;
9import com.amazonaws.mobileconnectors.s3.transferutility.TransferListener;
10import com.amazonaws.mobileconnectors.s3.transferutility.TransferObserver;
11import com.amazonaws.mobileconnectors.s3.transferutility.TransferService;
12import com.amazonaws.mobileconnectors.s3.transferutility.TransferState;
13import com.amazonaws.mobileconnectors.s3.transferutility.TransferUtility;
14import com.amazonaws.services.s3.AmazonS3Client;
15
16import java.io.BufferedWriter;
17import java.io.File;
18import java.io.FileWriter;
19
20public class MainActivity extends Activity {
21    private static final String TAG = MainActivity.class.getSimpleName();
22
23    @Override
24    protected void onCreate(Bundle savedInstanceState) {
25        super.onCreate(savedInstanceState);
26        setContentView(R.layout.activity_main);
27
28        getApplicationContext().startService(new Intent(getApplicationContext(), TransferService.class));
29
30        // Initialize the AWSMobileClient if not initialized
31        AWSMobileClient.getInstance().initialize(getApplicationContext(), new Callback<UserStateDetails>() {
32            @Override
33            public void onResult(UserStateDetails userStateDetails) {
34                Log.i(TAG, "AWSMobileClient initialized. User State is " + userStateDetails.getUserState());
35                uploadWithTransferUtility();
36            }
37
38            @Override
39            public void onError(Exception e) {
40                Log.e(TAG, "Initialization error.", e);
41            }
42        });
43
44    }
45
46    public void uploadWithTransferUtility() {
47
48        TransferUtility transferUtility =
49                TransferUtility.builder()
50                        .context(getApplicationContext())
51                        .awsConfiguration(AWSMobileClient.getInstance().getConfiguration())
52                        .s3Client(new AmazonS3Client(AWSMobileClient.getInstance()))
53                        .build();
54
55        File file = new File(getApplicationContext().getFilesDir(), "sample.txt");
56        try {
57            BufferedWriter writer = new BufferedWriter(new FileWriter(file));
58            writer.append("Howdy World!");
59            writer.close();
60        }
61        catch(Exception e) {
62            Log.e(TAG, e.getMessage());
63        }
64
65        TransferObserver uploadObserver =
66                transferUtility.upload(
67                        "public/sample.txt",
68                        new File(getApplicationContext().getFilesDir(),"sample.txt"));
69
70        // Attach a listener to the observer to get state update and progress notifications
71        uploadObserver.setTransferListener(new TransferListener() {
72
73            @Override
74            public void onStateChanged(int id, TransferState state) {
75                if (TransferState.COMPLETED == state) {
76                    // Handle a completed upload.
77                }
78            }
79
80            @Override
81            public void onProgressChanged(int id, long bytesCurrent, long bytesTotal) {
82                float percentDonef = ((float) bytesCurrent / (float) bytesTotal) * 100;
83                int percentDone = (int)percentDonef;
84
85                Log.d(TAG, "ID:" + id + " bytesCurrent: " + bytesCurrent
86                        + " bytesTotal: " + bytesTotal + " " + percentDone + "%");
87            }
88
89            @Override
90            public void onError(int id, Exception ex) {
91                // Handle errors
92            }
93
94        });
95
96        // If you prefer to poll for the data, instead of attaching a
97        // listener, check for the state and progress in the observer.
98        if (TransferState.COMPLETED == uploadObserver.getState()) {
99            // Handle a completed upload.
100        }
101
102        Log.d(TAG, "Bytes Transferred: " + uploadObserver.getBytesTransferred());
103        Log.d(TAG, "Bytes Total: " + uploadObserver.getBytesTotal());
104    }
105}

If you run this code, login to your AWS console, and go to the S3 service, you'll see a bucket and file structure like this (in this example the friendly name specified was dev and the bucket name was storagedemo):

Shows S3 service menu on AWS Console

Download a File

The following example shows how to use the TransferUtility to download a file. Instantiate the TransferUtility object using the provided TransferUtility builder function. Use the AWSMobileClient to get the AWSConfiguration and AWSCredentialsProvider to pass into the builder. See Authentication for more details.

1import android.app.Activity;
2import android.content.Intent;
3import android.os.Bundle;
4import android.util.Log;
5
6import com.amazonaws.mobile.client.AWSMobileClient;
7import com.amazonaws.mobile.client.Callback;
8import com.amazonaws.mobile.client.UserStateDetails;
9import com.amazonaws.mobileconnectors.s3.transferutility.TransferListener;
10import com.amazonaws.mobileconnectors.s3.transferutility.TransferObserver;
11import com.amazonaws.mobileconnectors.s3.transferutility.TransferService;
12import com.amazonaws.mobileconnectors.s3.transferutility.TransferState;
13import com.amazonaws.mobileconnectors.s3.transferutility.TransferUtility;
14import com.amazonaws.services.s3.AmazonS3Client;
15
16import java.io.BufferedWriter;
17import java.io.File;
18import java.io.FileWriter;
19
20public class MainActivity extends Activity {
21    private static final String TAG = MainActivity.class.getSimpleName();
22
23    @Override
24    protected void onCreate(Bundle savedInstanceState) {
25        super.onCreate(savedInstanceState);
26        setContentView(R.layout.activity_main);
27
28        getApplicationContext().startService(new Intent(getApplicationContext(), TransferService.class));
29
30        // Initialize the AWSMobileClient if not initialized
31        AWSMobileClient.getInstance().initialize(getApplicationContext(), new Callback<UserStateDetails>() {
32            @Override
33            public void onResult(UserStateDetails userStateDetails) {
34                Log.i(TAG, "AWSMobileClient initialized. User State is " + userStateDetails.getUserState());
35                downloadWithTransferUtility();
36            }
37
38            @Override
39            public void onError(Exception e) {
40                Log.e(TAG, "Initialization error.", e);
41            }
42        });
43
44    }
45
46    private void downloadWithTransferUtility() {
47
48        TransferUtility transferUtility =
49                TransferUtility.builder()
50                        .context(getApplicationContext())
51                        .awsConfiguration(AWSMobileClient.getInstance().getConfiguration())
52                        .s3Client(new AmazonS3Client(AWSMobileClient.getInstance()))
53                        .build();
54
55        TransferObserver downloadObserver =
56                transferUtility.download(
57                        "public/sample.txt",
58                        new File(getApplicationContext().getFilesDir(), "download.txt"));
59
60        // Attach a listener to the observer to get state update and progress notifications
61        downloadObserver.setTransferListener(new TransferListener() {
62
63            @Override
64            public void onStateChanged(int id, TransferState state) {
65                if (TransferState.COMPLETED == state) {
66                    // Handle a completed upload.
67                }
68            }
69
70            @Override
71            public void onProgressChanged(int id, long bytesCurrent, long bytesTotal) {
72                float percentDonef = ((float)bytesCurrent/(float)bytesTotal) * 100;
73                int percentDone = (int)percentDonef;
74
75                Log.d("Your Activity", "   ID:" + id + "   bytesCurrent: " + bytesCurrent + "   bytesTotal: " + bytesTotal + " " + percentDone + "%");
76            }
77
78            @Override
79            public void onError(int id, Exception ex) {
80                // Handle errors
81            }
82
83        });
84
85        // If you prefer to poll for the data, instead of attaching a
86        // listener, check for the state and progress in the observer.
87        if (TransferState.COMPLETED == downloadObserver.getState()) {
88            // Handle a completed upload.
89        }
90
91        Log.d("Your Activity", "Bytes Transferred: " + downloadObserver.getBytesTransferred());
92        Log.d("Your Activity", "Bytes Total: " + downloadObserver.getBytesTotal());
93    }
94}

Track Transfer Progress

With the TransferUtility, the download and upload methods return a TransferObserver object. This object gives access to:

The transfer state, as an enum
The total bytes that have been transferred so far
The total bytes remaining to transfer
A unique ID that you can use to keep track of each transfer

Given the transfer ID, the TransferObserver object can be retrieved from anywhere in your app, even if the app was terminated during a transfer. It also lets you create a TransferListener, which will be updated on changes to transfer state, progress, and when an error occurs.

To get the progress of a transfer, call setTransferListener() on your TransferObserver. This requires you to implement onStateChanged, onProgressChanged, and onError as shown in the example.

1TransferObserver transferObserver = download(MY_BUCKET, OBJECT_KEY, MY_FILE);
2transferObserver.setTransferListener(new TransferListener(){
3
4    @Override
5    public void onStateChanged(int id, TransferState state) {
6        // do something
7    }
8
9    @Override
10    public void onProgressChanged(int id, long bytesCurrent, long bytesTotal) {
11        int percentage = (int) (bytesCurrent/bytesTotal * 100);
12        //Display percentage transferred to user
13    }
14
15    @Override
16    public void onError(int id, Exception ex) {
17        // do something
18    }
19});

The transfer ID can be retrieved from the TransferObserver object that is returned from the upload or download function. You can also query for TransferObservers using the getTransfersWithType(transferType) or the getTransfersWithTypeAndState(transferType, transferState) method.

1// Gets id of the transfer.
2int transferId = transferObserver.getId();

Pause a Transfer

Transfers can be paused using the pause(transferId) method. If your app is terminated, crashes, or loses Internet connectivity, transfers are automatically paused.

To pause a single transfer:

1transferUtility.pause(idOfTransferToBePaused);

To pause all uploads:

1transferUtility.pauseAllWithType(TransferType.UPLOAD);

To pause all downloads:

1transferUtility.pauseAllWithType(TransferType.DOWNLOAD);

To pause all transfers of any type:

1transferUtility.pauseAllWithType(TransferType.ANY);

Resume a Transfer

In the case of a loss in network connectivity, transfers will automatically resume when network connectivity is restored. If the app crashed or was terminated by the operating system, transfers can be resumed with the resume(transferId) method.

To resume a single transfer:

1transferUtility.resume(idOfTransferToBeResumed);

To resume all uploads:

1transferUtility.resumeAllWithType(TransferType.UPLOAD);

To resume all downloads:

1transferUtility.resumeAllWithType(TransferType.DOWNLOAD);

To resume all transfers of any type:

1transferUtility.resumeAllWithType(TransferType.ANY);

Cancel a Transfer

To cancel an upload, call cancel() or cancelAllWithType() on the TransferUtility object.

To cancel a single transfer, use:

1transferUtility.cancel(idToBeCancelled);

To cancel all transfers of a certain type, use:

1transferUtility.cancelAllWithType(TransferType.DOWNLOAD);

Background Transfers

The SDK uploads and downloads objects from Amazon S3 using background threads. These transfers will continue to run regardless of whether your app is running in the foreground or background.

Long-running Transfers

When you want your app to perform long-running transfers in the background, you can initiate the transfers from a background service that you can implement within your app. A recommended way to use a service to initiate the transfer is demonstrated in the Transfer Utility sample application.

Supporting TransferService on Oreo and above

TransferNetworkLossHandler, a broadcast receiver that listens for network connectivity changes is introduced in 2.11.0. TransferNetworkLossHandler pauses the on-going transfers when the network goes offline and resumes the transfers that were paused when the network comes back online. TransferService registers the TransferNetworkLossHandler when the service is created and de-registers the handler when the service is destroyed.

TransferService will be moved to the foreground state when the device is running Android Oreo (API Level 26) and above.
- Transitioning to the foreground state requires a valid on-going Notification object, identifier for on-going notification and the flag that determines the ability to remove the on-going notification when the service transitions out of foreground state. If a valid notification object is not passed in, the service will not be transitioned into the foreground state.
- The TransferService can now be started using startForegroundService method to move the service to foreground state. The service can be invoked in the following way to transition the service to foreground state.

1Intent tsIntent = new Intent(getApplicationContext(), TransferService.class);
2tsIntent.putExtra(TransferService.INTENT_KEY_NOTIFICATION, <notification-object>);
3tsIntent.putExtra(TransferService.INTENT_KEY_NOTIFICATION_ID, <notification-id>);
4tsIntent.putExtra(TransferService.INTENT_KEY_REMOVE_NOTIFICATION, <remove-notification-when-service-stops-foreground>);
5getApplicationContext().startForegroundService(tsIntent);

Supporting Unicode characters in key-names

Upload/download objects

Since 2.4.0 version of the SDK, the key name containing characters that require special handling are URL encoded and escaped ( space, %2A, ~, /, :, ', (, ), !, [, ] ) by the AmazonS3Client, after which the AWS Android Core Runtime encodes the URL resulting in double encoding of the key name.
Starting 2.11.0, the additional layer of encoding and escaping done by AmazonS3Client is removed. The key name will not be encoded and escaped by AmazonS3Client. Now, the key name that is given to AmazonS3Client or TransferUtility will appear on the Amazon S3 console as is.

List Objects

When a S3 bucket contains objects with key names containing characters that require special handling, and since the SDK has an XML parser, (XML 1.0 parser) which cannot parse some characters, the SDK is required to request that Amazon S3 encode the keys in the response. This can be done by passing in url as encodingType in the ListObjectsRequest.

1AmazonS3Client s3 = new AmazonS3Client(credentials);
2final ObjectListing objectListing = s3.listObjects(
3                new ListObjectsRequest(bucketName, prefix, null, null, null)
4                    .withEncodingType(Constants.URL_ENCODING));

Since 2.4.0, there was a bug where the SDK did not decode the key names which are encoded by S3 when url is requested as the encodingType. This is fixed in 2.11.0, where the SDK will decode the key names in the ListObjectsResponse sent by S3.
If you have objects in S3 bucket that has a key name containing characters that require special handling, you need to pass the encodingType as url in the ListObjectsRequest.

Transfer with Object Metadata

To upload a file with metadata, use the ObjectMetadata object. Create a ObjectMetadata object and add in the metadata headers and pass it to the upload function.

1import com.amazonaws.services.s3.model.ObjectMetadata;
2
3ObjectMetadata myObjectMetadata = new ObjectMetadata();
4
5//create a map to store user metadata
6Map<String, String> userMetadata = new HashMap<String,String>();
7userMetadata.put("myKey","myVal");
8
9//call setUserMetadata on your ObjectMetadata object, passing it your map
10myObjectMetadata.setUserMetadata(userMetadata);

Then, upload an object along with its metadata:

1TransferObserver observer = transferUtility.upload(
2  MY_BUCKET,        /* The bucket to upload to */
3  OBJECT_KEY,       /* The key for the uploaded object */
4  MY_FILE,          /* The file where the data to upload exists */
5  myObjectMetadata  /* The ObjectMetadata associated with the object*/
6);

To download the metadata, use the S3 getObjectMetadata method. See the API Reference and Object Key and Metadata for more information.

Transfer Utility Options

You can use the TransferUtilityOptions object to customize the operations of the TransferUtility.

TransferThreadPoolSize

This parameter allows you to specify the number of transfers that can run in parallel. By increasing the number of threads, you will be able to increase the number of parts of a multi-part upload that will be uploaded in parallel. By default, this is set to 2 * (N + 1), where N is the number of available processors on the mobile device. The minimum allowed value is 2.

1TransferUtilityOptions options = new TransferUtilityOptions();
2options.setTransferThreadPoolSize(8);
3
4TransferUtility transferUtility = TransferUtility.builder()
5    // Pass-in S3Client, Context, AWSConfiguration/DefaultBucket Name
6    .transferUtilityOptions(options)
7    .build();

TransferNetworkConnectionType

The TransferNetworkConnectionType option allows you to restrict the type of network connection (WiFi / Mobile / ANY) over which the data can be transferred to Amazon S3.

1TransferUtilityOptions options = new TransferUtilityOptions(10, TransferNetworkConnectionType.WIFI);
2
3TransferUtility transferUtility = TransferUtility.builder()
4    // Pass-in S3Client, Context, AWSConfiguration/DefaultBucket Name
5    .transferUtilityOptions(options)
6    .build();

By specifying TransferNetworkConnectionType.WIFI , data transfers to and from S3 will only happen when the device is on a WiFi connection