CTS FAQ

The Android Compatibility Program is the key driver to sustain the positive feedback for Android ecosystem. CTS is the key tool to ensure the quality of compatibility in the scale. The Android team continues to improve CTS tool and test coverage. The regular addition of test cases has significant improvement on the quality of compatible devices.

General questions

This section provides general CTS FAQs.

What kinds of things does the CTS test?

The CTS tests that all of the supported Android strong-typed APIs are present and behave correctly. The CTS also tests other non-API system behaviors such as app lifecycle and performance.

How is the CTS licensed?

The CTS is licensed under the same Apache Software License 2.0 that the bulk of Android uses.

Are codecs verified by CTS?

Yes. All mandatory codecs are verified by CTS.

Test-specific questions

This section provides FAQs that help run CTS tests more efficiently.

What is the difference between CTS Sharding and TF Sharding?

CTS Sharding and TF Sharding are totally different test plans powered by different test infrastructure codebase. While the run command is the same across different versions, the sharding result behaves differently. CTS Sharding statically assigns test cases to Devices Under Test (DUTs) as follows:

TF Sharding dynamically assigns test cases to available DUTs as follows:

What is expected from a device supporting multiple ABIs?

The device has to pass all CTS and CTS Verifier tests for each ABI mode it claims to support. Therefore, it is necessary to execute an app for the particular ABIs. The guidelines for multiple ABIs are as follows:

  • For CTS and CTS Verifier, there are ARM and x86 releases for each architecture. Each of them can support to 32- or 64-bit mode.
  • For CTS tests, if a device supports both ARM and x86, it has to run and pass both ARM and x86 CTS tests respectively.

See CDD 3.3.1. Application Binary Interfaces for CDD requirements on ABI.

Is it sufficient to run a test only on the primary ABI (for example, 64 bits) to reduce test execution time?

No. An Android app runs on its own 32 -bit or 64-bit runtimes. The actual machine code, code path, and state are different between 32 and 64. If you skip one mode, you are only covering 50% of the device ABI.

Why are there so many test cases reported as Not Executed?

You should check the Module Done number instead of Not Executed number.

In the previous versions, CTS modules were reported as Module Done too aggressively before being completed. Therefore, a Modules Done number was reported without all the test case complete even when some devices had problems. The new test harness is more conservative and reports a higher number of Not Executed tests when a problem occurs.

A module run to completion reports Module Not Done in the most recent invocation (done="false") in the report during the following:

  • A test run for the module was interrupted by a device connection issue.
  • Not all expected test runs for the module were performed.

  • Retried (using option -r/--retry) with additional filtering options, such as:

    • --include-filter
    • --exclude-filter
    • -t/--test (Option not yet supported on retry)
    • --retry-type failed
    • --subplan

To obtain a status of Module Done (done="true") for these modules, retry the following for the most recent invocation:

run retry --retry <session_id> for Android 9 and later versions

run cts --retry <session_id> for Android 8.1 and previous versions

A module executed without any of the problems mentioned previously (even with 0 remaining tests) is marked Module Done in the new report.

Exceptions

  • CtsNNAPITestCases has a known issue due to linux/OS limitation of args. The module can be rerun in isolation through run cts -m CtsNNAPITestCases directly.

How can I avoid test preparation failing behind corporate firewall?

All automated test suites try to download either the CTS media files or the business logic files during runtime. In many corporate environments, a firewall and proxy is typical, which makes the test preparation fail. Execute the following line or add it to .profile (on Ubuntu).

export JAVA_TOOL_OPTIONS='-Djava.net.useSystemProxies=true'

Do I need a SIM card for CTS for Secure Element?

Whether a SIM card is needed for the test depends on the understanding if the feature is supported in the test device.

  • If your device DOES NOT need to support the Android apps accessing secure elements—either in the UICC (e.g., a SIM card) distributed by the mobile network operators (carriers) or embedded in the device—you can configure the HIDL manifest to not include the android.hardware.secure_element HAL element. In this case, the android.se.omapi.SEService.getReaders() API reports an empty list and the CTS test automatically passes and reports a pass for the CTS.
  • If your device DOES need to support Android apps accessing either secure elements—either in the UICC (e.g., a SIM card) distributed by the mobile network operators (carriers) or embedded in the device—you need to implement secure element properly and test it in-house. CTS Test for Secure Element outlines how to prepare to run the CTS tests that ensure android.se.omapi API package added in Android 9 is functional. We also recommend performing additional testing on your own since the CTS test coverage is minimal.

Where can I get the SIM cards for CTS for Secure Element?

You can reach out to your preferred SIM vendor.

Why is Orange SIM on the lock screen during CTS execution with token sharding?

The test case doesn't start because testing the SIM card is locked. Disable the Lock SIM card in **SIM card lock settings before executing the CTS with token sharding.