ArFrame

Attempts to acquire a "raw", mostly unfiltered, depth image that corresponds to the current frame.

The raw depth image is sparse and does not provide valid depth for all pixels. Pixels without a valid depth estimate have a pixel value of 0 and a corresponding confidence value of 0 (see ArFrame_acquireRawDepthConfidenceImage ).

The depth image has format HardwareBuffer.D_16 , which is a single 16-bit plane at index 0, stored in little-endian format. Each pixel contains the distance in millimeters along the camera principal axis, with the representable depth range between 0 millimeters and 65535 millimeters, or about 65 meters.

To extract distance from a depth map, see the Depth API developer guide .

The actual size of the depth image depends on the device and its display aspect ratio. The size of the depth image is typically around 160x120 pixels, with higher resolutions up to 640x480 on some devices. These sizes may change in the future. The outputs of ArFrame_acquireDepthImage16Bits , ArFrame_acquireRawDepthImage16Bits and ArFrame_acquireRawDepthConfidenceImage will all have the exact same size.

Optimal depth accuracy is achieved between 500 millimeters (50 centimeters) and 15000 millimeters (15 meters) from the camera, with depth reliably observed up to 25000 millimeters (25 meters). Error increases quadratically as distance from the camera increases.

Depth is primarily estimated using data from the motion of world-facing cameras. As the user moves their device through the environment, 3D depth data is collected and cached, improving the quality of subsequent depth images and reducing the error introduced by camera distance. Depth accuracy and robustness improves if the device has a hardware depth sensor, such as a time-of-flight (ToF) camera.

Not every raw depth image contains a new depth estimate. Typically there are about 10 updates to the raw depth data per second. The depth images between those updates are a 3D reprojection which transforms each depth pixel into a 3D point in space and renders those 3D points into a new raw depth image based on the current camera pose. This effectively transforms raw depth image data from a previous frame to account for device movement since the depth data was calculated. For some applications it may be important to know whether the raw depth image contains new depth data or is a 3D reprojection (for example, to reduce the runtime cost of 3D reconstruction). To do that, compare the current raw depth image timestamp, obtained via ArImage_getTimestamp , with the previously recorded raw depth image timestamp. If they are different, the depth image contains new information.

When the Geospatial API and the Depth API are enabled, output images from the Depth API will include terrain and building geometry when in a location with VPS coverage. See the Geospatial Depth Developer Guide for more information.

The image must be released via ArImage_release once it is no longer needed.