Cuttlefish: Multi-tenancy

Cuttlefish multi-tenancy allows for your host machine to launch multiple virtual guest devices with a single launch invocation.

These multiple Cuttlefish virtual machines can share some of the host disk resources, allowing you to conserve disk space. Each virtual machine gets a distinct disk overlay covering the Android partitions (such as super, userdata, boot). Any modification to the guest disk is placed in a guest-specific disk overlay, giving each Cuttlefish virtual machine a copy-on-write view of its disk partitions. As each virtual device writes more data to its view of the disk, the overlay files on the host disk increase in size because they're capturing the delta between the original state and their current state.

The overlay files can be reset to return the Cuttlefish disk to its original state. This process is known as powerwashing.

Instance numbers

Cuttlefish devices create and consume resources on the host tied to their instance number. For example, an adb connection is exposed through a TCP server socket at port 6520 for the Cuttlefish device with instance number 1. When multiple instances are launched, it creates a TCP server socket at port 6520 for the first device, 6521 for the second device, and increments the port number for each additional device.

Modify maximum number of supported instances

Host-side resources, such as TAP network devices, must be preallocated for Cuttlefish virtual machines. By default, the cuttlefish-base Debian package provisions resources for instance numbers 1 through 10.

This can be changed through modifying num_cvd_accounts in /etc/default/cuttlefish-host-resources and restarting the cuttlefish-host-resources service by running the following command:

sudo systemctl restart cuttlefish-host-resources

Canonical configurations

Canonical configurations represent the configuration for Cuttlefish devices in JSON format. You can create canonical configuration JSON files to describe the configuration of multiple Cuttlefish devices in a multi-tenant scenario.

The following describes the format used for canonical configuration JSON files (replace placeholders with device configurations).

{
  "common": {
    CONFIGURATIONS_APPLYING_TO_ALL_DEVICES
  },
  "instances": [
    {
      FIRST_DEVICE_CONFIGURATION
    },
    {
      NTH_DEVICE_CONFIGURATION
    }
  ]
}

The following is an example configuration file of a setup with two devices, a phone device and a wearable device:

{
  "instances": [
    {
      "@import": "phone",
      "vm": {
        "memory_mb": 8192,
        "setupwizard_mode": "OPTIONAL",
        "cpus": 4
      },
      "disk": {
        "default_build": "/home/username/devices/cf_x86_64_phone-userdebug"
      }
    },
    {
      "@import": "wearable",
      "vm": {
        "memory_mb": 8192,
        "setupwizard_mode": "REQUIRED",
        "cpus": 4
      },
      "disk": {
        "default_build": "/home/username/devices/cf_gwear_x86-userdebug"
      }
    }
  ]
}

This table describes the elements used in the example configuration with two devices:

Element	Type	Description
`instances`	Array	Array of JSON objects, each describing a single device in the multi-tenant group of devices to be launched.
`@import`	String	Indicates the device type and sets default values for the device configuration. The supported device types are `phone` and `wearable`.
`vm`	Object	Specifies properties that apply to the VM environment specific to the device instance.
`memory_mb`	Number	The amount of memory to allocate for a specific VM in megabytes.
`setupwizard_mode`	String	Specifies whether the device setup wizard is run for the user at first run of the device. The supported options for setup wizard mode are `DISABLED`, `OPTIONAL`, and `REQUIRED`.
`cpus`	Number	The number of virtual CPUs allocated to the VM.
`disk`	Object	Specifies properties relevant to locating file system artifacts used for running a single VM instance.
`default_build`	String	Local file system path to a folder containing a Cuttlefish virtual device image.

For more examples of configuration files distributed with cvd, see the cvd_test_configs folder in the Cuttlefish code tree.

Control multi-tenant instances

This section describes how to create and destroy devices in a multi-tenant scenario. You can launch multiple Cuttlefish devices using the following options:

Canonical config: Launch customized configurations for each device using a JSON config file describing the list of instances to launch.
Single guest image: Launch multiple devices from a single guest image.

Start multiple VMs from canonical config

To create multiple VMs using a canonical configuration that describes the multi-tenant scenario, invoke the cvd create command with the --config_file= flag prefix as follows:

cvd create --config_file=CONFIG_FILE

The following is an example invocation referencing a canonical config file named /etc/phone.json.

cvd create --config_file=/etc/phone.json

Start multiple VMs from single guest image

To launch multiple Cuttlefish devices using a single guest image, use the --num_instances=N flag, where N is the number of devices to launch. By default, devices start from the instance number 1.

cvd create --num_instances=N

To change the starting instance number, do one of the following:

Add the --base_instance_num=N flag with N as the first instance number.
```
cvd create --base_instance_num=N
```
Launch Cuttlefish with a user account matching vsoc-NN, where NN is the base instance number in two digit form. For example, to launch Cuttlefish with a user account vsoc-01, run:
```
cvd create --base_instance_num=1
```

How to stop devices

To stop all the devices launched by the last cvd create invocation, run:

cvd stop

To restart devices that have been stopped, run:

cvd start

To completely remove a device from the system run:

cvd remove