You can use ML Kit to detect faces in images and video.
Try it out
- Play around with the sample app to see an example usage of this API.
- Try the code yourself with the codelab .
Before you begin
- Include the following ML Kit pods in your Podfile:
pod 'GoogleMLKit/FaceDetection', '8.0.0'
- After you install or update your project's Pods, open your Xcode project using its
.xcworkspace. ML Kit is supported in Xcode version 12.4 or greater.
Input image guidelines
For face recognition, you should use an image with dimensions of at least 480x360 pixels. For ML Kit to accurately detect faces, input images must contain faces that are represented by sufficient pixel data. In general, each face you want to detect in an image should be at least 100x100 pixels. If you want to detect the contours of faces, ML Kit requires higher resolution input: each face should be at least 200x200 pixels.
If you detect faces in a real-time application, you might also want to consider the overall dimensions of the input images. Smaller images can be processed faster, so to reduce latency, capture images at lower resolutions, but keep in mind the above accuracy requirements and ensure that the subject's face occupies as much of the image as possible. Also see tips to improve real-time performance .
Poor image focus can also impact accuracy. If you don't get acceptable results, ask the user to recapture the image.
The orientation of a face relative to the camera can also affect what facial features ML Kit detects. See Face Detection Concepts .
1. Configure the face detector
Before you apply face detection to an image, if you want to change any of the face detector's default settings, specify those settings with aFaceDetectorOptions
object. You can change
the following settings: performanceMode
fast
(default) | accurate
Favor speed or accuracy when detecting faces.
landmarkMode
none
(default) | all
Whether to attempt to detect the facial "landmarks"—eyes, ears, nose, cheeks, mouth—of all detected faces.
contourMode
none
(default) | all
Whether to detect the contours of facial features. Contours are detected for only the most prominent face in an image.
classificationMode
none
(default) | all
Whether or not to classify faces into categories such as "smiling" and "eyes open".
minFaceSize
CGFloat
(default: 0.1
) Sets the smallest desired face size, expressed as the ratio of the width of the head to width of the image.
isTrackingEnabled
false
(default) | true
Whether or not to assign faces an ID, which can be used to track faces across images.
Note that when contour detection is enabled, only one face is detected, so face tracking doesn't produce useful results. For this reason, and to improve detection speed, don't enable both contour detection and face tracking.
For example, build a FaceDetectorOptions
object like one of the following examples:
Swift
// High-accuracy landmark detection and face classification let options = FaceDetectorOptions () options . performanceMode = . accurate options . landmarkMode = . all options . classificationMode = . all // Real-time contour detection of multiple faces // options.contourMode = .all
Objective-C
// High-accuracy landmark detection and face classification MLKFaceDetectorOptions * options = [[ MLKFaceDetectorOptions alloc ] init ]; options . performanceMode = MLKFaceDetectorPerformanceModeAccurate ; options . landmarkMode = MLKFaceDetectorLandmarkModeAll ; options . classificationMode = MLKFaceDetectorClassificationModeAll ; // Real-time contour detection of multiple faces // options.contourMode = MLKFaceDetectorContourModeAll;
2. Prepare the input image
To detect faces in an image, pass the image as aUIImage
or a CMSampleBufferRef
to the FaceDetector
using either the process(_:completion:)
or results(in:)
method: Create a VisionImage
object using a UIImage
or a CMSampleBuffer
.
If you use a UIImage
, follow these steps:
- Create a
VisionImageobject with theUIImage. Make sure to specify the correct.orientation.Swift
let image = VisionImage(image: UIImage) visionImage.orientation = image.imageOrientation
Objective-C
MLKVisionImage * visionImage = [[ MLKVisionImage alloc ] initWithImage : image ]; visionImage . orientation = image . imageOrientation ;
If you use a CMSampleBuffer
, follow these steps:
-
Specify the orientation of the image data contained in the
CMSampleBuffer.To get the image orientation:
Swift
func imageOrientation ( deviceOrientation : UIDeviceOrientation , cameraPosition : AVCaptureDevice . Position ) -> UIImage . Orientation { switch deviceOrientation { case . portrait : return cameraPosition == . front ? . leftMirrored : . right case . landscapeLeft : return cameraPosition == . front ? . downMirrored : . up case . portraitUpsideDown : return cameraPosition == . front ? . rightMirrored : . left case . landscapeRight : return cameraPosition == . front ? . upMirrored : . down case . faceDown , . faceUp , . unknown : return . up } }
Objective-C
- ( UIImageOrientation ) imageOrientationFromDeviceOrientation :( UIDeviceOrientation ) deviceOrientation cameraPosition :( AVCaptureDevicePosition ) cameraPosition { switch (deviceOrientation) { case UIDeviceOrientationPortrait : return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationLeftMirrored : UIImageOrientationRight ; case UIDeviceOrientationLandscapeLeft : return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationDownMirrored : UIImageOrientationUp ; case UIDeviceOrientationPortraitUpsideDown : return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationRightMirrored : UIImageOrientationLeft ; case UIDeviceOrientationLandscapeRight : return cameraPosition == AVCaptureDevicePositionFront ? UIImageOrientationUpMirrored : UIImageOrientationDown ; case UIDeviceOrientationUnknown : case UIDeviceOrientationFaceUp : case UIDeviceOrientationFaceDown : return UIImageOrientationUp ; } }
- Create a
VisionImageobject using theCMSampleBufferobject and orientation:Swift
let image = VisionImage ( buffer : sampleBuffer ) image . orientation = imageOrientation ( deviceOrientation : UIDevice . current . orientation , cameraPosition : cameraPosition )
Objective-C
MLKVisionImage * image = [[ MLKVisionImage alloc ] initWithBuffer : sampleBuffer ]; image . orientation = [ self imageOrientationFromDeviceOrientation : UIDevice . currentDevice . orientation cameraPosition : cameraPosition ];
3. Get an instance of FaceDetector
Get an instance of FaceDetector
:
Swift
let faceDetector = FaceDetector.faceDetector(options: options)
Objective-C
MLKFaceDetector * faceDetector = [ MLKFaceDetector faceDetectorWithOptions : options ];
4. Process the image
Then, pass the image to theprocess()
method: Swift
weak var weakSelf = self faceDetector . process ( visionImage ) { faces , error in guard let strongSelf = weakSelf else { print ( "Self is nil!" ) return } guard error == nil , let faces = faces , ! faces . isEmpty else { // ... return } // Faces detected // ... }
Objective-C
[ faceDetector processImage : image completion : ^ ( NSArray<MLKFace * > * faces , NSError * error ) { if ( error != nil ) { return ; } if ( faces . count > 0 ) { // Recognized faces } }];
5. Get information about detected faces
If the face detection operation succeeds, the face detector passes an array ofFace
objects to the completion handler. Each Face
object represents a face that was detected in the image. For
each face, you can get its bounding coordinates in the input image, as well as
any other information you configured the face detector to find. For example: Swift
for face in faces {
let frame = face.frame
if face.hasHeadEulerAngleX {
let rotX = face.headEulerAngleX // Head is rotated to the uptoward rotX degrees
}
if face.hasHeadEulerAngleY {
let rotY = face.headEulerAngleY // Head is rotated to the right rotY degrees
}
if face.hasHeadEulerAngleZ {
let rotZ = face.headEulerAngleZ // Head is tilted sideways rotZ degrees
}
// If landmark detection was enabled (mouth, ears, eyes, cheeks, and
// nose available):
if let leftEye = face.landmark(ofType: .leftEye) {
let leftEyePosition = leftEye.position
}
// If contour detection was enabled:
if let leftEyeContour = face.contour(ofType: .leftEye) {
let leftEyePoints = leftEyeContour.points
}
if let upperLipBottomContour = face.contour(ofType: .upperLipBottom) {
let upperLipBottomPoints = upperLipBottomContour.points
}
// If classification was enabled:
if face.hasSmilingProbability {
let smileProb = face.smilingProbability
}
if face.hasRightEyeOpenProbability {
let rightEyeOpenProb = face.rightEyeOpenProbability
}
// If face tracking was enabled:
if face.hasTrackingID {
let trackingId = face.trackingID
}
}
Objective-C
for ( MLKFace * face in faces ) { // Boundaries of face in image CGRect frame = face . frame ; if ( face . hasHeadEulerAngleX ) { CGFloat rotX = face . headEulerAngleX ; // Head is rotated to the upward rotX degrees } if ( face . hasHeadEulerAngleY ) { CGFloat rotY = face . headEulerAngleY ; // Head is rotated to the right rotY degrees } if ( face . hasHeadEulerAngleZ ) { CGFloat rotZ = face . headEulerAngleZ ; // Head is tilted sideways rotZ degrees } // If landmark detection was enabled (mouth, ears, eyes, cheeks, and // nose available): MLKFaceLandmark * leftEar = [ face landmarkOfType : FIRFaceLandmarkTypeLeftEar ]; if ( leftEar != nil ) { MLKVisionPoint * leftEarPosition = leftEar . position ; } // If contour detection was enabled: MLKFaceContour * upperLipBottomContour = [ face contourOfType : FIRFaceContourTypeUpperLipBottom ]; if ( upperLipBottomContour != nil ) { NSArray<MLKVisionPoint *> * upperLipBottomPoints = upperLipBottomContour . points ; if ( upperLipBottomPoints . count > 0 ) { NSLog ( "Detected the bottom contour of the subject's upper lip." ) } } // If classification was enabled: if ( face . hasSmilingProbability ) { CGFloat smileProb = face . smilingProbability ; } if ( face . hasRightEyeOpenProbability ) { CGFloat rightEyeOpenProb = face . rightEyeOpenProbability ; } // If face tracking was enabled: if ( face . hasTrackingID ) { NSInteger trackingID = face . trackingID ; } }
Example of face contours
When you have face contour detection enabled, you get a list of points for each facial feature that was detected. These points represent the shape of the feature. See Face Detection Concepts for details about how contours are represented.
The following image illustrates how these points map to a face, click the image to enlarge it:
Real-time face detection
If you want to use face detection in a real-time application, follow these guidelines to achieve the best framerates:
-
Configure the face detector to use either face contour detection or classification and landmark detection, but not both:
Contour detection
Landmark detection
Classification
Landmark detection and classification
Contour detection and landmark detection
Contour detection and classification
Contour detection, landmark detection, and classification -
Enable
fastmode (enabled by default). -
Consider capturing images at a lower resolution. However, also keep in mind this API's image dimension requirements.
- For processing video frames, use the
results(in:)synchronous API of the detector. Call this method from theAVCaptureVideoDataOutputSampleBufferDelegate'scaptureOutput(_, didOutput:from:)function to synchronously get results from the given video frame. KeepAVCaptureVideoDataOutput'salwaysDiscardsLateVideoFramesastrueto throttle calls to the detector. If a new video frame becomes available while the detector is running, it will be dropped. - If you use the output of the detector to overlay graphics on the input image, first get the result from ML Kit, then render the image and overlay in a single step. By doing so, you render to the display surface only once for each processed input frame. See the updatePreviewOverlayViewWithLastFrame in the ML Kit quickstart sample for an example.
