This was part of the documentation for a library I’m currently prototyping to simplify accessing media files. It turned out to be helpful to write down all the facts in a condensed way first. But this is so useful I’m releasing it into the wild. Please enjoy!

Quick history lesson

In the early days of Android, the user’s data was always stored on an external SD card because the internal memory of phones was too limited. Thus, the default file storage area is also called “external storage”, or “shared storage” more recently. In contrast, there is the “internal storage”, which actually is always on the internal flash memory of a phone, but also is private for each app. It always was relatively easy to read and write data on the external storage, as it is the primary storage location for user data. However, with Android 3.0, the concept of emulated storage was introduced. Here, the external storage is no longer an actual FAT32 file system on a SD card, but instead an emulated layer which is part of the internal storage. (Nowadays, emulated storage is extremely common.) As convenience feature of 3.0, MediaStore also returns files from removable storage (actual external SD cards) and the File API can be used to read the files on removable storage as well. However, an official way to discover or write to removable storage was never made available, so OEM-specific guesswork and assumption the File API can also be used for writing was commonplace until Google locked it down in Android 4.4 (technically, the lockdown was in 4.2, but it wasn’t enforced so no OEM actually followed through with that as it broke a lot of apps). In Android 4.4, the storage access framework was the official alternative, but it lacked a feature to grant access to an entire directory or SD card, only individual files could be overwritten one-by-one. This was rectified with Android 5.0 and the introduction of Intent.ACTION_OPEN_DOCUMENT_TREE, which was the first official API for batch writes on removable storage that Android ever got. So, as of Android 5.0, reading both external (which is actually the internal flash memory, because it’s emulated) storage and removable storage was possible using File API. For writing to external storage, you have the choice to use File API or Storage Access Framework, but for removable storage, SAF is the only option.

Introducing scoped storage

However, with the Android 10 release, a lot of things changed. Google wanted to get away from every app having full access to the storage, so the APIs were split for different use cases. The File API is used for app-private storage, MediaStore is used to access media files and SAF is used for document and miscellaneous files. This means a music player getting permission to read music files wouldn’t be able to read your PDF invoices in your Download folder. But they went a bit further by also restricting file write access via MediaStore to only files the app owns. If an app tries to write to / delete a file it did not create (-> does not own), the user has to be asked for permission with a special dialog. SAF did not change much, so that did not cause issues for app developers, however, the lockdown of the File API was extremely disruptive to the app ecosystem, so Google added an escape hatch which returns the File API to the former state (can be used for reading & writing to external storage and just reading for removable storage) while also enabling full view of files in MediaStore, while preserving new scoped-storage compatible features in MediaStore, such as allowing MediaStore to write to removable storage.

This escape hatch turned out to be a good idea because the scoped storage implementation in Android 10 is severely undercooked. Android 11 rectified many issues, among them are:

• You cannot batch write/delete files your app did not create, you have to show a dialog for every single file. If a gallery app tries to delete 2000 pictures shot by the Camera app, the user will go insane very quickly after being forced to click “Allow” 2000 times.
• Files such as subtitle/song lyric or playlist files were not counted as media files allowed to be accessed via MediaStore, so apps had to use an API not intended for their use case which was causing poor UX.
• The new “Trash” feature did not hide files from apps using the escape hatch, which made it not obvious to the user this file was actually deleted.
• The MediaStore database was not guaranteed to be complete because apps were able to create files using the File API if they use the escape hatch (or are old apps) without updating MediaStore, which punished new apps that were now unable to read the files created by older apps, causing poor UX.
• The playlist parser from MediaStore was using very old code that was prone to creating issues where the cached playlist did not match the version on disk, which hindered collaboration between apps as scoped storage music players could only see the MediaStore playlist, while File API players often parsed the playlist from disk themselves instead of using the MediaStore version (which was already available to them).
• Because the SAF API was not restricted at all, apps commonly requested access to all storage devices via SAF instead of trying to minimize the permission requirements. This lead to poor performance because the SAF API is not designed for high performance, and it also made the privacy efforts useless because every app was still going to have full access to storage.
• There was no API optimized for file management or backup apps, they were forced to use the slow SAF APIs even if suboptimal.
• There is no way for an app on Android 10 to write the MediaStore.Files SQL data of a non-media file on removable storage at all (though this is not too much of a problem because none of the SQL columns of earlier versions that apply to non-media files are still writable, they’re all auto-detected. The only missing features for removable storage are thus IS_PENDING and IS_TRASHED which are both new). If a media file’s SQL data is to be modified, care must be taken to not accidentally use a MediaStore.Files Uri which would be valid in Android 11 for example, but not in Android 10.

The new ways

Android 11 reintroduced support for the File API by using the Linux FUSE feature, which allows the system to implement the File API but process all interactions just as the app was actually using MediaStore. This means that apps running on Android 11 using the File API see a filtered view of storage - they can only see what MediaStore can also see. In reverse, writing using File API would also update the MediaStore database, so it is now always up to date. The write permission aspect was reworked entirely to depend on whether the app owns the file, both for File and MediaStore API, and it no longer differentiates between removable and external storage. Requesting write access was made possible in batches of thousands of files to improve UX. Additionally, playlist, lyric and subtitle files were added to MediaStore, and the playlist parser and Trash features were reworked. Additionally, a favourite feature was added. A new permission just for file management apps was introduced which grants full read and write access to both external and removable storage using both File and MediaStore API. Last but not least, SAF was restricted to no longer allow entire volumes or the Download directory to be granted to apps.

The bad treatment of removable storage was finally stopped, it’s now fully equal to external storage. But on the way, there were a lot of different Android versions with their own quirks:

• Android 5-9 require the user to pick the SD card in the SAF folder picker (the more user friendly yes-no dialog is available from Android 7 to 9) and the app has to find out the SAF URI for a file somehow, for which an API was only added in Android 8.
• Android 10 allows media files to be written on SD card without permission prompt if the app has the classic storage permission through the escape hatch, but the definition of media files is too tight to be universally useful: playlists, lyric and subtitle files are missing. Also, due to old apps not using MediaStore, a File might be missing from the MediaStore database and as such cannot be written using MediaStore even if it is visible to us using File API. The SQL write Uris have to be fixed for Android 10 for media files, and SQL writes must be avoided for non-media files on removable storage.
• On Android 11 and later, it must first be ensured we either own the file or obtained write permission from the user. After obtaining write permission, we can only use the MediaStore API, but for files we own (including files we are going to create) we can use the File API.

Implementing this in a way that always results in the best UX (low amount of permission prompts) requires using all three of File, MediaStore and SAF APIs on Android 10 alone.

What I so far didn’t cover is reading MediaStore SQL data. This doesn’t require special permission handling, instead, you need to account for the fact some files might not be displayed:

• On Android 10 and earlier, a file might not be indexed because it was created by an old app. This can only be fixed by using the File API and walking the tree, using android.media.MediaScannerConnection.scanFile on missing files.
• On Android 11 and later, a file might be marked as pending-by-FUSE if it was created by an old app (or adb push). This means that the app didn’t notify the system that this file was fully created, but also that the app was using the File API instead of the MediaStore API, which often means it’s an old app that just didn’t support notifying the MediaStore API for some reason or another. These files have to be explicitly included by setting MediaStore.QUERY_ARG_MATCH_PENDING to MediaStore.MATCH_INCLUDE and then using a SQL query to exclude files that are pending-by-MediaStore (created by new apps and explicitly to be ignored due to being work-in-progress, such as an ongoing download). (for example, pseudocode: IS_PENDING=0 OR NOT (DISPLAY_NAME LIKE '.pending-%'))

Do note that all files in a folder with .nomedia are also hidden from MediaStore and:

• on Android 10 or earlier, are only visible with the File API or SAF
• on Android 11 and later, are only visible with SAF, or with the “all files access” permission for file managers, or when owning the file. In the latter two cases, you’ll find it in MediaStore.Files instead of one of the normal collections even if it’s usually a media file. (Also, you can only create files in a folder with .nomedia using the File API. Trying to use insert() and then openFileDescriptor() will crash at the latter saying the Uri doesn’t exist. Using MediaStore API to write or delete already existing files in a folder with .nomedia however works, same as the File API.)

Recap / how to do this in your app

For the external storage, the File API is used to write and delete files until Android 11. That is because MediaStore did not actually support deleting files until Android 10, and in Android 10 it might still be outdated which would cause both write or delete to fail, so the File API is more reliable. It also doesn’t need any additional prompts for permission other than the storage permission that’s used anyways to read files. MediaStore SQL writes are also possible with the same permission without additional complexity. (A app using SAF for reading would not be able to use the File API here, but such an app should just use SAF to delete files, too.)

For removable storage on Android 5-10, use StorageVolume.createAccessIntent (for Android 7-9) and Intent.ACTION_OPEN_DOCUMENT_TREE to gain access to the removable storage volume using SAF and uses it to write or delete files. MediaStore SQL writes are possible on Android 5-9 for all files using the normal storage permission used to read, and the same applies for Android 10, except for non-media files where it’s impossible (the URIs need to be fixed for Android 10 from MediaStore.Files to a specific table like MediaStore.Audio.Media to avoid false negatives). On Android 10 only, if the file happens to be a media file per Android 10’s definition, it can be deleted without permission prompt using MediaStore.

For both external and removable storage on Android 11 and later, the appropriate write/delete request dialogs are used if required (file is now owned by us and we don’t have the “all files access” special permission for file managers), and both SQL writes and writes via MediaStore file descriptor API (but not the File API) become possible once granted via dialog. If we didn’t need to request permission in the first place, the File API, MediaStore API and MediaStore SQL writes are already immediately possible.