Stabilize camera images on Android SDK (Kotlin/Java)

ARCore now supports Electronic Image Stabilization (EIS), which helps produce a smooth camera preview. EIS achieves stabilization by observing phone movement using gyro and applying compensation homography mesh within the boundaries of camera texture that counters the minor shakes. EIS is only supported in the device's portrait orientation. All orientations will be supported in the 1.39.0 release of ARCore.

Query for EIS support and enable EIS

To enable EIS, configure your session to use ImageStabilizationMode.EIS . If the device doesn't support the EIS feature, this will cause an exception to be thrown from ARCore.

Java

 if 
  
 ( 
 ! 
 session 
 . 
 isImageStabilizationModeSupported 
 ( 
 Config 
 . 
 ImageStabilizationMode 
 . 
 EIS 
 )) 
  
 { 
  
 return 
 ; 
 } 
 Config 
  
 config 
  
 = 
  
 session 
 . 
 getConfig 
 (); 
 config 
 . 
 setImageStabilizationMode 
 ( 
 Config 
 . 
 ImageStabilizationMode 
 . 
 EIS 
 ); 
 session 
 . 
 configure 
 ( 
 config 
 ); 

Kotlin

 if 
  
 ( 
 ! 
 session 
 . 
 isImageStabilizationModeSupported 
 ( 
 Config 
 . 
 ImageStabilizationMode 
 . 
 EIS 
 )) 
  
 return 
 session 
 . 
 configure 
 ( 
  
 session 
 . 
 config 
 . 
 apply 
  
 { 
  
 imageStabilizationMode 
  
 = 
  
 Config 
 . 
 ImageStabilizationMode 
 . 
 EIS 
  
 } 
 ) 

Transform coordinates

When EIS is on, the renderer needs to use the modified device coordinates and matching texture coordinates that incorporate the EIS compensation when rendering the camera background. To get the EIS compensated coordinates, use Frame.transformCoordinates3d() , using OPENGL_NORMALIZED_DEVICE_COORDINATES as input and EIS_NORMALIZED_DEVICE_COORDINATES as output to get 3D device coordinates and EIS_TEXTURE_NORMALIZED as output to get 3D texture coordinates. For now, the only supported input coordinate type for Frame.transformCoordinates3d() is OPENGL_NORMALIZED_DEVICE_COORDINATES .

Java

 final 
  
 FloatBuffer 
  
 cameraTexCoords 
  
 = 
  
 ByteBuffer 
 . 
 allocateDirect 
 ( 
 COORDS_BUFFER_SIZE_3D 
 ) 
  
 . 
 order 
 ( 
 ByteOrder 
 . 
 nativeOrder 
 ()) 
  
 . 
 asFloatBuffer 
 (); 
 final 
  
 FloatBuffer 
  
 screenCoords 
  
 = 
  
 ByteBuffer 
 . 
 allocateDirect 
 ( 
 COORDS_BUFFER_SIZE_3D 
 ) 
  
 . 
 order 
 ( 
 ByteOrder 
 . 
 nativeOrder 
 ()) 
  
 . 
 asFloatBuffer 
 (); 
 final 
  
 FloatBuffer 
  
 NDC_QUAD_COORDS_BUFFER 
  
 = 
  
 ByteBuffer 
 . 
 allocateDirect 
 ( 
 COORDS_BUFFER_SIZE_2D 
 ) 
  
 . 
 order 
 ( 
 ByteOrder 
 . 
 nativeOrder 
 ()) 
  
 . 
 asFloatBuffer 
 () 
  
 . 
 put 
 ( 
  
 new 
  
 float 
 [] 
  
 { 
  
 /*0:*/ 
  
 - 
 1f 
 , 
  
 - 
 1f 
 , 
  
 /*1:*/ 
  
 + 
 1f 
 , 
  
 - 
 1f 
 , 
  
 /*2:*/ 
  
 - 
 1f 
 , 
  
 + 
 1f 
 , 
  
 /*3:*/ 
  
 + 
 1f 
 , 
  
 + 
 1f 
 , 
  
 }); 
 final 
  
 VertexBuffer 
  
 screenCoordsVertexBuffer 
  
 = 
  
 new 
  
 VertexBuffer 
 ( 
 render 
 , 
  
 /* numberOfEntriesPerVertex= */ 
  
 3 
 , 
  
 null 
 ); 
 final 
  
 VertexBuffer 
  
 cameraTexCoordsVertexBuffer 
  
 = 
  
 new 
  
 VertexBuffer 
 ( 
 render 
 , 
  
 /* numberOfEntriesPerVertex= */ 
  
 3 
 , 
  
 null 
 ); 
 NDC_QUAD_COORDS_BUFFER 
 . 
 rewind 
 (); 
 frame 
 . 
 transformCoordinates3d 
 ( 
  
 Coordinates2d 
 . 
 OPENGL_NORMALIZED_DEVICE_COORDINATES 
 , 
  
 NDC_QUAD_COORDS_BUFFER 
 , 
  
 Coordinates3d 
 . 
 EIS_NORMALIZED_DEVICE_COORDINATES 
 , 
  
 screenCoords 
 ); 
 screenCoordsVertexBuffer 
 . 
 set 
 ( 
 screenCoords 
 ); 
 NDC_QUAD_COORDS_BUFFER 
 . 
 rewind 
 (); 
 frame 
 . 
 transformCoordinates3d 
 ( 
  
 Coordinates2d 
 . 
 OPENGL_NORMALIZED_DEVICE_COORDINATES 
 , 
  
 NDC_QUAD_COORDS_BUFFER 
 , 
  
 Coordinates3d 
 . 
 EIS_TEXTURE_NORMALIZED 
 , 
  
 cameraTexCoords 
 ); 
 cameraTexCoordsVertexBuffer 
 . 
 set 
 ( 
 cameraTexCoords 
 ); 

Kotlin

 val 
  
 COORDS_BUFFER_SIZE_2D 
  
 = 
  
 2 
  
 * 
  
 4 
  
 * 
  
 Float 
 . 
 SIZE_BYTES 
 val 
  
 COORDS_BUFFER_SIZE_3D 
  
 = 
  
 3 
  
 * 
  
 4 
  
 * 
  
 Float 
 . 
 SIZE_BYTES 
 val 
  
 cameraTexCoords 
  
 = 
  
 ByteBuffer 
 . 
 allocateDirect 
 ( 
 COORDS_BUFFER_SIZE_3D 
 ) 
  
 . 
 order 
 ( 
 ByteOrder 
 . 
 nativeOrder 
 ()) 
  
 . 
 asFloatBuffer 
 () 
 val 
  
 screenCoords 
  
 = 
  
 ByteBuffer 
 . 
 allocateDirect 
 ( 
 COORDS_BUFFER_SIZE_3D 
 ) 
  
 . 
 order 
 ( 
 ByteOrder 
 . 
 nativeOrder 
 ()) 
  
 . 
 asFloatBuffer 
 () 
 val 
  
 cameraTexCoordsVertexBuffer 
  
 = 
  
 VertexBuffer 
 ( 
 render 
 , 
  
 /* numberOfEntriesPerVertex= */ 
  
 3 
 , 
  
 null 
 ) 
 val 
  
 screenCoordsVertexBuffer 
  
 = 
  
 VertexBuffer 
 ( 
 render 
 , 
  
 /* numberOfEntriesPerVertex= */ 
  
 3 
 , 
  
 null 
 ) 
 val 
  
 NDC_QUAD_COORDS_BUFFER 
  
 = 
  
 ByteBuffer 
 . 
 allocateDirect 
 ( 
 COORDS_BUFFER_SIZE_2D 
 ) 
  
 . 
 order 
 ( 
 ByteOrder 
 . 
 nativeOrder 
 ()) 
  
 . 
 asFloatBuffer 
 () 
  
 . 
 apply 
  
 { 
  
 put 
 ( 
  
 floatArrayOf 
 ( 
  
 /* 0: */ 
  
 - 
 1f 
 , 
  
 - 
 1f 
 , 
  
 /* 1: */ 
  
 + 
 1f 
 , 
  
 - 
 1f 
 , 
  
 /* 2: */ 
  
 - 
 1f 
 , 
  
 + 
 1f 
 , 
  
 /* 3: */ 
  
 + 
 1f 
 , 
  
 + 
 1f 
  
 ) 
  
 ) 
  
 } 
 NDC_QUAD_COORDS_BUFFER 
 . 
 rewind 
 () 
 frame 
 . 
 transformCoordinates3d 
 ( 
  
 Coordinates2d 
 . 
 OPENGL_NORMALIZED_DEVICE_COORDINATES 
 , 
  
 NDC_QUAD_COORDS_BUFFER 
 , 
  
 Coordinates3d 
 . 
 EIS_NORMALIZED_DEVICE_COORDINATES 
 , 
  
 screenCoords 
 ) 
 screenCoordsVertexBuffer 
 . 
 set 
 ( 
 screenCoords 
 ) 
 NDC_QUAD_COORDS_BUFFER 
 . 
 rewind 
 () 
 frame 
 . 
 transformCoordinates3d 
 ( 
  
 Coordinates2d 
 . 
 OPENGL_NORMALIZED_DEVICE_COORDINATES 
 , 
  
 NDC_QUAD_COORDS_BUFFER 
 , 
  
 Coordinates3d 
 . 
 EIS_TEXTURE_NORMALIZED 
 , 
  
 cameraTexCoords 
 ) 
 cameraTexCoordsVertexBuffer 
 . 
 set 
 ( 
 cameraTexCoords 
 ) 

When EIS is off, the output 3D coordinates are equivalent to their 2D counterparts, with z values set to produce no change.

Modify shaders

The 3D coordinates calculated about should be passed to background rendering shaders. The vertex buffers are now 3D with EIS:

  layout 
 ( 
 location 
  
 = 
  
 0 
 ) 
  
 in 
  
 vec4 
  
 a_Position 
 ; 
 layout 
 ( 
 location 
  
 = 
  
 1 
 ) 
  
 in 
  
 vec3 
  
 a_CameraTexCoord 
 ; 
 out 
  
 vec3 
  
 v_CameraTexCoord 
 ; 
 void 
  
 main 
 () 
  
 { 
  
 gl_Position 
  
 = 
  
 a_Position 
 ; 
  
 v_CameraTexCoord 
  
 = 
  
 a_CameraTexCoord 
 ; 
 } 
 

Additionally, the fragment shader needs to apply perspective correction:

  precision 
  
 mediump 
  
 float 
 ; 
 uniform 
  
 samplerExternalOES 
  
 u_CameraColorTexture 
 ; 
 in 
  
 vec3 
  
 v_CameraTexCoord 
 ; 
 layout 
 ( 
 location 
  
 = 
  
 0 
 ) 
  
 out 
  
 vec4 
  
 o_FragColor 
 ; 
 void 
  
 main 
 () 
  
 { 
  
 vec3 
  
 tc 
  
 = 
  
 ( 
 v_CameraTexCoord 
  
 / 
  
 v_CameraTexCoord 
 . 
 z 
 ); 
  
 o_FragColor 
  
 = 
  
 texture 
 ( 
 u_CameraColorTexture 
 , 
  
 tc 
 . 
 xy 
 ); 
 } 
 

See the hello_eis_kotlin sample app for more details.

Create a Mobile Website
View Site in Mobile | Classic
Share by: