Enable event-driven function retries (1st gen)
This document describes how to enable retrying for event-driven functions . Automatic retrying is not available for HTTP functions.
Semantics of retry
Cloud Run functions provides at-least-once execution of an event-driven function for each event emitted by an event source. By default, if a function invocation terminates with an error, the function is not invoked again and the event is dropped. When you enable retries on an event-driven function, Cloud Run functions retries a failed function invocation until it completes successfully or the retry window expires.
This retry window expires expires after 7 days. Cloud Run functions retries newly created event-driven functions using an exponential backoff strategy, with an increasing backoff of between 10 and 600 seconds. This policy is applied to new functions the first time you deploy them. It is not retroactively applied to existing functions that were first deployed before the changes described in this release note took effect, even if you redeploy the functions.
When retries are not enabled for a function, which is the default, the function
always reports that it executed successfully, and 200 OK
response codes might
appear in its logs. This occurs even if the function encountered an error. To
make it clear when your function encounters an error, be sure to report errors
appropriately.
Why event-driven functions fail to complete
On rare occasions, a function might exit prematurely due to an internal error, and by default the function might or might not be automatically retried.
More typically, an event-driven function might fail to successfully complete due to errors thrown in the function code itself. The reasons this might happen include:
- The function contains a bug and the runtime throws an exception.
- The function cannot reach a service endpoint, or times out while trying to do so.
- The function intentionally throws an exception (for example, when a parameter fails validation).
- A Node.js function returns a rejected promise, or passes a non-
nullvalue to a callback.
In any of these cases, the function stops executing by default and the event is discarded. To retry the function when an error occurs, you can change the default retry policy by setting the "retry on failure" property . This causes the event to be retried repeatedly until the function successfully completes or the retry timeout expires.
Enable or disable retries
To enable or disable retries, you can either use the gcloud
command-line tool
or the Google Cloud console. By default, retries are disabled.
Configure retries from the gcloud
command-line tool
To enable retries using the gcloud
command-line tool, include the --retry
flag
when deploying your function:
gcloud functions deploy FUNCTION_NAME --retry FLAGS ...
To disable retries, re-deploy the function without the --retry
flag:
gcloud functions deploy FUNCTION_NAME FLAGS ...
Configure retries from the console
If you're creating a new function:
- From the Create Function screen, under Triggerand choose the type of event to act as a trigger for your function.
- Select the Retry on failurecheckbox to enable retries.
If you're updating an existing function:
- From the Cloud Run functions Overview page, click the name of the function you're updating to open its Function detailsscreen, then choose Editfrom the menu bar to display Triggerpane.
- Select or clear the Retry on failurecheckbox to enable or disable retries.
Best practices
This section describes best practices for using retries.
Use retry to handle transient errors
Because your function is retried continuously until successful execution, permanent errors like bugs should be eliminated from your code through testing before enabling retries. Retries are best used to handle intermittent or transient failures that have a high likelihood of resolution upon retrying, such as a flaky service endpoint or timeout.
Set an end condition to avoid infinite retry loops
It is best practice to protect your function against continuous looping when using retries. You can do this by including a well-defined end condition, before the function begins processing. Note that this technique only works if your function starts successfully and is able to evaluate the end condition.
A simple yet effective approach is to discard events with timestamps older than a certain time. This helps to avoid excessive executions when failures are either persistent or longer-lived than expected.
For example, this code snippet discards all events older than 10 seconds:
Node.js
Python
Go
Java
C#
using CloudNative.CloudEvents ; using Google.Cloud.Functions.Framework ; using Google.Events.Protobuf.Cloud.PubSub.V1 ; using Microsoft.Extensions.Logging ; using System ; using System.Threading ; using System.Threading.Tasks ; namespace TimeBoundedRetries ; public class Function : ICloudEventFunction<MessagePublishedData> { private static readonly TimeSpan MaxEventAge = TimeSpan . FromSeconds ( 10 ); private readonly ILogger _logger ; // Note: for additional testability, use an injectable clock abstraction. public Function ( ILogger<Function> logger ) = > _logger = logger ; public Task HandleAsync ( CloudEvent cloudEvent , MessagePublishedData data , CancellationToken cancellationToken ) { string textData = data . Message . TextData ; DateTimeOffset utcNow = DateTimeOffset . UtcNow ; // Every PubSub CloudEvent will contain a timestamp. DateTimeOffset timestamp = cloudEvent . Time . Value ; DateTimeOffset expiry = timestamp + MaxEventAge ; // Ignore events that are too old. if ( utcNow > expiry ) { _logger . LogInformation ( "Dropping PubSub message '{text}'" , textData ); return Task . CompletedTask ; } // Process events that are recent enough. // If this processing throws an exception, the message will be retried until either // processing succeeds or the event becomes too old and is dropped by the code above. _logger . LogInformation ( "Processing PubSub message '{text}'" , textData ); return Task . CompletedTask ; } }
Ruby
PHP
/** * This function shows an example method for avoiding infinite retries in * Google Cloud Functions. By default, functions configured to automatically * retry execution on failure will be retried indefinitely - causing an * infinite loop. To avoid this, we stop retrying executions (by not throwing * exceptions) for any events that are older than a predefined threshold. */ use Google\CloudFunctions\CloudEvent; function avoidInfiniteRetries(CloudEvent $event): void { $log = fopen(getenv('LOGGER_OUTPUT') ?: 'php://stderr', 'wb'); $eventId = $event->getId(); // The maximum age of events to process. $maxAge = 10; // 10 seconds // The age of the event being processed. $eventAge = time() - strtotime($event->getTime()); // Ignore events that are too old if ($eventAge > $maxAge) { fwrite($log, 'Dropping event ' . $eventId . ' with age ' . $eventAge . ' seconds' . PHP_EOL); return; } // Do what the function is supposed to do fwrite($log, 'Processing event: ' . $eventId . ' with age ' . $eventAge . ' seconds' . PHP_EOL); // infinite_retries failed function executions $failed = true; if ($failed) { throw new Exception('Event ' . $eventId . ' failed; retrying...'); } }
Distinguish between functions that can be retried and fatal errors
If your function has retries enabled, any unhandled error will trigger a retry. Make sure that your code captures any errors that shouldn't result in a retry.
Node.js
Python
Go
Java
C#
using CloudNative.CloudEvents ; using Google.Cloud.Functions.Framework ; using Google.Events.Protobuf.Cloud.PubSub.V1 ; using Microsoft.Extensions.Logging ; using System ; using System.Text.Json ; using System.Threading ; using System.Threading.Tasks ; namespace Retry ; public class Function : ICloudEventFunction<MessagePublishedData> { private readonly ILogger _logger ; public Function ( ILogger<Function> logger ) = > _logger = logger ; public Task HandleAsync ( CloudEvent cloudEvent , MessagePublishedData data , CancellationToken cancellationToken ) { bool retry = false ; string text = data . Message ?. TextData ; // Get the value of the "retry" JSON parameter, if one exists. if ( ! string . IsNullOrEmpty ( text )) { JsonElement element = JsonSerializer . Deserialize<JsonElement> ( data . Message . TextData ); retry = element . TryGetProperty ( "retry" , out var property ) && property . ValueKind == JsonValueKind . True ; } // Throwing an exception causes the execution to be retried. if ( retry ) { throw new InvalidOperationException ( "Retrying..." ); } else { _logger . LogInformation ( "Not retrying..." ); } return Task . CompletedTask ; } }
Ruby
PHP
use Google\CloudFunctions\CloudEvent; function tipsRetry(CloudEvent $event): void { $cloudEventData = $event->getData(); $pubSubData = $cloudEventData['message']['data']; $json = json_decode(base64_decode($pubSubData), true); // Determine whether to retry the invocation based on a parameter $tryAgain = $json['some_parameter']; if ($tryAgain) { /** * Functions with automatic retries enabled should throw exceptions to * indicate intermittent failures that a retry might fix. In this * case, a thrown exception will cause the original function * invocation to be re-sent. */ throw new Exception('Intermittent failure occurred; retrying...'); } /** * If a function with retries enabled encounters a non-retriable * failure, it should return *without* throwing an exception. */ $log = fopen(getenv('LOGGER_OUTPUT') ?: 'php://stderr', 'wb'); fwrite($log, 'Not retrying' . PHP_EOL); }
Make retryable event-driven functions idempotent
Event-driven functions that can be retried must be idempotent. Here are some general guidelines for making such a function idempotent:
- Many external APIs (such as Stripe) let you supply an idempotency key as a parameter. If you are using such an API, you should use the event ID as the idempotency key.
- Idempotency works well with at-least-once delivery, because it makes it safe to retry. So a general best practice for writing reliable code is to combine idempotency with retries.
- Make sure that your code is internally idempotent. For example:
- Make sure that mutations can happen more than once without changing the outcome.
- Query database state in a transaction before mutating the state.
- Make sure that all side effects are themselves idempotent.
- Impose a transactional check outside the function, independent of the code. For example, persist state somewhere recording that a given event ID has already been processed.
- Deal with duplicate function calls out-of-band. For example, have a separate clean up process that cleans up after duplicate function calls.
Configure the retry policy
Depending on the needs of your function, you may want to configure the retry policy directly. This would allow you to set up any combination of the following:
- Shorten the retry window from 7 days to as little as 10 minutes.
- Change the minimum and maximum backoff time for the exponential backoff retry strategy.
- Change the retry strategy to retry immediately.
- Configure a dead-letter topic .
- Set a maximum and minimum number of delivery attempts.
To configure the retry policy:
- Write an HTTP function.
- Use the Pub/Sub API to create a Pub/Sub subscription, specifying the URL of the function as the target.
See Pub/Sub documentation on handling failures for a more information on configuring Pub/Sub directly.
