This developer guide walks you through the steps of enabling your app to switch seamlessly between exclusive control of the camera via the Android Camera2 API and sharing camera access with ARCore.
This topic assumes you:
-
Have completed the ARCore Quickstart
-
Are familiar with the Android Camera2 API (review the Android-specific Camera2 sample to learn more)
Build and run the sample app
When you build and run the Shared Camera Javasample app, it creates an ARCore session that supports shared camera access. The app starts in non-AR mode, with ARCore paused.
When the app operates in non-AR mode, the camera viewer displays a sepia color effect. When switching to AR mode, the sepia effect switches off as the app returns camera control to ARCore by resuming the paused session.
You can use the AR switch in the app to change modes. During preview, both modes display the number of continuous frames captured by Camera2.
To build and run the Shared Camera Java sample app:
-
Download and extract the Google ARCore SDK for Android .
-
Open the
samples/shared_camera_javaproject. -
Make sure that your Android device is connected to the development machine via USB. See ARCore Supported devices for detailed information.
-
In Android Studio, click Run .
-
Choose your device as the deployment target, and click OK to launch the sample app on your device.
-
On the device, confirm that you want to allow the app to take pictures and record video.
-
If prompted to do so, update or install the latest version of ARCore.
-
Use the AR switch to change between non-AR and AR modes.
Overview of enabling an app to share camera access with ARCore
Follow these steps to implement shared camera access with ARCore in your app.
All code snippets are available in the SharedCameraActivity.java
within the shared_camera_java
sample.
Request CAMERA
permission
In order to be able to use the device's camera, the user must grant your app the CAMERA
permission
.
The ARCore samples include a CameraPermissionHelper
,
which provides utilities to request the correct permission for your app.
Java
protected
void
onResume
()
{
// Request the camera permission, if necessary.
if
(
!
CameraPermissionHelper
.
hasCameraPermission
(
this
))
{
CameraPermissionHelper
.
requestCameraPermission
(
this
);
}
}
Kotlin
override
fun
onResume
()
{
// Request the camera permission, if necessary.
if
(
!
CameraPermissionHelper
.
hasCameraPermission
(
this
))
{
CameraPermissionHelper
.
requestCameraPermission
(
this
)
}
}
Ensure ARCore is installed and up to date
ARCore must be installed and up-to-date before it can be used. The following snippet shows how to request an installation of ARCore if it has not already been installed on the device.
Java
boolean
isARCoreSupportedAndUpToDate
()
{
// Make sure that ARCore is installed and supported on this device.
ArCoreApk
.
Availability
availability
=
ArCoreApk
.
getInstance
().
checkAvailability
(
this
);
switch
(
availability
)
{
case
SUPPORTED_INSTALLED
:
return
true
;
case
SUPPORTED_APK_TOO_OLD
:
case
SUPPORTED_NOT_INSTALLED
:
// Requests an ARCore installation or updates ARCore if needed.
ArCoreApk
.
InstallStatus
installStatus
=
ArCoreApk
.
getInstance
().
requestInstall
(
this
,
userRequestedInstall
);
switch
(
installStatus
)
{
case
INSTALL_REQUESTED
:
return
false
;
case
INSTALLED
:
return
true
;
}
return
false
;
default
:
// Handle the error. For example, show the user a snackbar that tells them
// ARCore is not supported on their device.
return
false
;
}
}
Kotlin
// Determine ARCore installation status.
// Requests an ARCore installation or updates ARCore if needed.
fun
isARCoreSupportedAndUpToDate
():
Boolean
{
when
(
ArCoreApk
.
getInstance
().
checkAvailability
(
this
))
{
Availability
.
SUPPORTED_INSTALLED
-
>
return
true
Availability
.
SUPPORTED_APK_TOO_OLD
,
Availability
.
SUPPORTED_NOT_INSTALLED
-
>
{
when
(
ArCoreApk
.
getInstance
().
requestInstall
(
this
,
userRequestedInstall
))
{
InstallStatus
.
INSTALLED
-
>
return
true
else
-
>
return
false
}
}
else
-
>
{
// Handle the error. For example, show the user a snackbar that tells them
// ARCore is not supported on their device.
return
false
}
}
}
Create an ARCore session that supports camera sharing
This involves creating the session and storing the reference and ID of ARCore shared camera:
Java
// Create an ARCore session that supports camera sharing.
sharedSession
=
new
Session
(
this
,
EnumSet
.
of
(
Session
.
Feature
.
SHARED_CAMERA
))
// Store the ARCore shared camera reference.
sharedCamera
=
sharedSession
.
getSharedCamera
();
// Store the ID of the camera that ARCore uses.
cameraId
=
sharedSession
.
getCameraConfig
().
getCameraId
();
Kotlin
// Create an ARCore session that supports camera sharing.
sharedSession
=
Session
(
this
,
EnumSet
.
of
(
Session
.
Feature
.
SHARED_CAMERA
))
// Store the ARCore shared camera reference.
sharedCamera
=
sharedSession
.
sharedCamera
// Store the ID of the camera that ARCore uses.
cameraId
=
sharedSession
.
cameraConfig
.
cameraId
(Optional) Inform ARCore of any custom surfaces
Requesting additional custom surfaces increases the performance demands of the device. To ensure it performs well, test your app on the devices that your users will use.
ARCore will request two streams by default:
- 1x YUV CPU stream, currently always
640x480.
ARCore uses this stream for motion tracking . - A 1x GPU stream, typically
1920x1080
UseSession#getCameraConfig()to determine the current GPU stream resolution.
You can change the resolution of the GPU stream on supported devices by using getSupportedCameraConfigs()
and setCameraConfig()
.
As rough indicator, you can expect:
- 2x YUV CPU streams
, e.g.
640x480and1920x1080 - 1x GPU stream
, e.g.
1920x1080 - 1x occasional high res still image
(JPEG), e.g.
12MP
- 2x YUV CPU streams
, e.g.
640x480and1920x1080 - 1x GPU stream
, e.g.
1920x1080
- 1x YUV CPU streams
, e.g.
640x480–or–1920x1080 - 1x GPU stream
, e.g.
1920x1080 - 1x occasional high res still image
(JPEG), e.g.
12MP
To use custom surfaces, such as a CPU image reader surface, make sure to add it
to the list of surfaces that need to be updated
(for example, an ImageReader
).
Java
sharedCamera
. ">setAppSurfaces
(this.cameraId, Arrays.asList(imageReader.getSurface()));
Kotlin
sharedCamera
. ">setAppSurfaces
(this.cameraId, listOf(imageReader.surface))
Open the camera
Open the camera using an ARCore-wrapped callback:
Java
// Wrap the callback in a shared camera callback.
CameraDevice
.
StateCallback
wrappedCallback
=
sharedCamera
.
createARDeviceStateCallback
(
cameraDeviceCallback
,
backgroundHandler
);
// Store a reference to the camera system service.
cameraManager
=
(
CameraManager
)
this
.
getSystemService
(
Context
.
CAMERA_SERVICE
);
// Open the camera device using the ARCore wrapped callback.
cameraManager
.
openCamera
(
cameraId
,
wrappedCallback
,
backgroundHandler
);
Kotlin
// Wrap the callback in a shared camera callback.
val
wrappedCallback
=
sharedCamera
.
createARDeviceStateCallback
(
cameraDeviceCallback
,
backgroundHandler
)
// Store a reference to the camera system service.
val
cameraManager
=
this
.
getSystemService
(
Context
.
CAMERA_SERVICE
)
as
CameraManager
// Open the camera device using the ARCore wrapped callback.
cameraManager
.
openCamera
(
cameraId
,
wrappedCallback
,
backgroundHandler
)
Use the camera device state callback
In the camera device state callback store a reference to the camera device, and start a new capture session.
Java
public
void
onOpened
(
@NonNull
CameraDevice
cameraDevice
)
{
Log
.
d
(
TAG
,
"Camera device ID "
+
cameraDevice
.
getId
()
+
" opened."
);
SharedCameraActivity
.
this
.
cameraDevice
=
cameraDevice
;
createCameraPreviewSession
();
}
Kotlin
fun
onOpened
(
cameraDevice
:
CameraDevice
)
{
Log
.
d
(
TAG
,
"Camera device ID "
+
cameraDevice
.
id
+
" opened."
)
this
.
cameraDevice
=
cameraDevice
createCameraPreviewSession
()
}
Create a new capture session
Build a new capture request. Use TEMPLATE_RECORD
to ensure that the capture request is compatible with ARCore, and to allow seamless
switching between non-AR and AR mode at runtime.
Java
void
createCameraPreviewSession
()
{
try
{
// Create an ARCore-compatible capture request using `TEMPLATE_RECORD`.
previewCaptureRequestBuilder
=
cameraDevice
.
createCaptureRequest
(
CameraDevice
.
TEMPLATE_RECORD
);
// Build a list of surfaces, starting with ARCore provided surfaces.
List<Surface>
surfaceList
=
sharedCamera
.
getArCoreSurfaces
();
// (Optional) Add a CPU image reader surface.
surfaceList
.
add
(
cpuImageReader
.
getSurface
());
// The list should now contain three surfaces:
// 0. sharedCamera.getSurfaceTexture()
// 1. …
// 2. cpuImageReader.getSurface()
// Add ARCore surfaces and CPU image surface targets.
for
(
Surface
surface
:
surfaceList
)
{
previewCaptureRequestBuilder
.
addTarget
(
surface
);
}
// Wrap our callback in a shared camera callback.
CameraCaptureSession
.
StateCallback
wrappedCallback
=
sharedCamera
.
createARSessionStateCallback
(
cameraSessionStateCallback
,
backgroundHandler
);
// Create a camera capture session for camera preview using an ARCore wrapped callback.
cameraDevice
.
createCaptureSession
(
surfaceList
,
wrappedCallback
,
backgroundHandler
);
}
catch
(
CameraAccessException
e
)
{
Log
.
e
(
TAG
,
"CameraAccessException"
,
e
);
}
}
Kotlin
fun
createCameraPreviewSession
()
{
try
{
// Create an ARCore-compatible capture request using `TEMPLATE_RECORD`.
previewCaptureRequestBuilder
=
cameraDevice
.
createCaptureRequest
(
CameraDevice
.
TEMPLATE_RECORD
)
// Build a list of surfaces, starting with ARCore provided surfaces.
val
surfaceList
:
MutableList<Surface>
=
sharedCamera
.
arCoreSurfaces
// (Optional) Add a CPU image reader surface.
surfaceList
.
add
(
cpuImageReader
.
getSurface
())
// The list should now contain three surfaces:
// 0. sharedCamera.getSurfaceTexture()
// 1. …
// 2. cpuImageReader.getSurface()
// Add ARCore surfaces and CPU image surface targets.
for
(
surface
in
surfaceList
)
{
previewCaptureRequestBuilder
.
addTarget
(
surface
)
}
// Wrap the callback in a shared camera callback.
val
wrappedCallback
=
sharedCamera
.
createARSessionStateCallback
(
cameraSessionStateCallback
,
backgroundHandler
)
// Create a camera capture session for camera preview using an ARCore wrapped callback.
cameraDevice
.
createCaptureSession
(
surfaceList
,
wrappedCallback
,
backgroundHandler
)
}
catch
(
e
:
CameraAccessException
)
{
Log
.
e
(
TAG
,
"CameraAccessException"
,
e
)
}
}
Start in non-AR or AR mode
To begin capturing frames, call captureSession.setRepeatingRequest()
from the camera capture session onConfigured()
state callback.
Resume the ARCore session within the onActive()
callback to start in AR mode.
Java
// Repeating camera capture session state callback.
CameraCaptureSession
.
StateCallback
cameraSessionStateCallback
=
new
CameraCaptureSession
.
StateCallback
()
{
// Called when ARCore first configures the camera capture session after
// initializing the app, and again each time the activity resumes.
@Override
public
void
onConfigured
(
@NonNull
CameraCaptureSession
session
)
{
captureSession
=
session
;
setRepeatingCaptureRequest
();
}
@Override
public
void
onActive
(
@NonNull
CameraCaptureSession
session
)
{
if
(
arMode
&&
!
arcoreActive
)
{
resumeARCore
();
}
}
};
// A repeating camera capture session capture callback.
CameraCaptureSession
.
CaptureCallback
cameraCaptureCallback
=
new
CameraCaptureSession
.
CaptureCallback
()
{
@Override
public
void
onCaptureCompleted
(
…
)
{
shouldUpdateSurfaceTexture
.
set
(
true
);
}
};
void
setRepeatingCaptureRequest
()
{
captureSession
.
setRepeatingRequest
(
previewCaptureRequestBuilder
.
build
(),
cameraCaptureCallback
,
backgroundHandler
);
}
void
resumeARCore
()
{
// Resume ARCore.
sharedSession
.
resume
();
arcoreActive
=
true
;
// Set the capture session callback while in AR mode.
sharedCamera
.
setCaptureCallback
(
cameraCaptureCallback
,
backgroundHandler
);
}
Kotlin
val
cameraSessionStateCallback
=
object
:
CameraCaptureSession
.
StateCallback
()
{
// Called when ARCore first configures the camera capture session after
// initializing the app, and again each time the activity resumes.
override
fun
onConfigured
(
session
:
CameraCaptureSession
)
{
captureSession
=
session
setRepeatingCaptureRequest
()
}
override
fun
onActive
(
session
:
CameraCaptureSession
)
{
if
(
arMode
&&
!
arcoreActive
)
{
resumeARCore
()
}
}
}
val
cameraCaptureCallback
=
object
:
CameraCaptureSession
.
CaptureCallback
()
{
override
fun
onCaptureCompleted
(
session
:
CameraCaptureSession
,
request
:
CaptureRequest
,
result
:
TotalCaptureResult
)
{
shouldUpdateSurfaceTexture
.
set
(
true
);
}
}
fun
setRepeatingCaptureRequest
()
{
captureSession
.
setRepeatingRequest
(
previewCaptureRequestBuilder
.
build
(),
cameraCaptureCallback
,
backgroundHandler
)
}
fun
resumeARCore
()
{
// Resume ARCore.
sharedSession
.
resume
()
arcoreActive
=
true
// Set the capture session callback while in AR mode.
sharedCamera
.
setCaptureCallback
(
cameraCaptureCallback
,
backgroundHandler
)
}
Switch seamlessly between non-AR or AR modes at runtime
To switch from non-AR to AR mode and resume a paused ARCore session:
Java
// Resume the ARCore session.
resumeARCore
();
Kotlin
// Resume the ARCore session.
resumeARCore
()
To switch from AR-mode to non-AR mode:
Java
// Pause ARCore.
sharedSession
.
pause
();
// Create the Camera2 repeating capture request.
setRepeatingCaptureRequest
();
Kotlin
// Pause ARCore.
sharedSession
.
pause
()
// Create the Camera2 repeating capture request.
setRepeatingCaptureRequest
()
