Background processing with .NET


Many apps need to do background processing outside of the context of a web request. This tutorial creates a web app that lets users input text to translate, and then displays a list of previous translations. The translation is done in a background process to avoid blocking the user's request.

The following diagram illustrates the translation request process.

Diagram of architecture.

Here is the sequence of events for how the tutorial app works:

  1. Visit the web page to see a list of previous translations, stored in Firestore.
  2. Request a translation of text by entering an HTML form.
  3. The translation request is published to Pub/Sub.
  4. A Cloud Run service subscribed to that Pub/Sub topic is triggered.
  5. The Cloud Run service uses Cloud Translation to translate the text.
  6. The Cloud Run service stores the result in Firestore.

This tutorial is intended for anyone who is interested in learning about background processing with Google Cloud. No prior experience is required with Pub/Sub, Firestore, App Engine, or Cloud Run functions. However, to understand all of the code, some experience with .NET, JavaScript, and HTML is helpful.

Objectives

  • Understand and deploy Cloud Run services.
  • Try the app.

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator .

New Google Cloud users might be eligible for a free trial .

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up .

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Verify that billing is enabled for your Google Cloud project .

  4. Enable the Firestore, Cloud Run, Pub/Sub, and Cloud Translation APIs.

    Enable the APIs

  5. Install the Google Cloud CLI.

  6. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity .

  7. To initialize the gcloud CLI, run the following command:

    gcloud  
    init
  8. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  9. Verify that billing is enabled for your Google Cloud project .

  10. Enable the Firestore, Cloud Run, Pub/Sub, and Cloud Translation APIs.

    Enable the APIs

  11. Install the Google Cloud CLI.

  12. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity .

  13. To initialize the gcloud CLI, run the following command:

    gcloud  
    init
  14. Update gcloud components:
    gcloud  
    components  
    update
  15. Prepare your development environment.

    Setting up a .NET development environment

Preparing the app

  1. In your terminal window, clone the sample app repository to your local machine:

    git  
    clone  
    https://github.com/GoogleCloudPlatform/getting-started-dotnet.git

    Alternatively, you can download the sample as a zip file and extract it.

  2. Change to the directory that contains the background task sample code:

     cd 
      
    getting-started-dotnet/BackgroundProcessing

Understanding the TranslateWorker service

  • The service starts by importing several dependencies like Firestore and Translation.

  • The Firestore and Translation clients are initialized so they can be reused between handler invocations. That way, you don't have to initialize new clients for every invocation, which would slow down execution.

      public 
      
     void 
      
     ConfigureServices 
     ( 
     IServiceCollection 
      
     services 
     ) 
     { 
      
     services 
     . 
     AddSingleton<FirestoreDb> 
     ( 
     provider 
      
     = 
    >  
     FirestoreDb 
     . 
     Create 
     ( 
     GetFirestoreProjectId 
     ())); 
      
     services 
     . 
     AddSingleton<TranslationClient> 
     ( 
      
     TranslationClient 
     . 
     Create 
     ()); 
      
     services 
     . 
     AddMvc 
     (). 
     SetCompatibilityVersion 
     ( 
     CompatibilityVersion 
     . 
     Version_2_1 
     ); 
     } 
     
    
  • The Translation API translates the string to the language you selected.

      var 
      
     result 
      
     = 
      
     await 
      
     _translator 
     . 
     TranslateTextAsync 
     ( 
     sourceText 
     , 
      
     "es" 
     ); 
     
    
  • The controller's constructor receives the Firestore and Pub/Sub clients.

    The Post method parses the Pub/Sub message to get the text to translate. It uses the message ID as a unique name for the translation request to make sure it doesn't store any duplicate translations.

      using 
      
      Google.Cloud.Firestore 
     
     ; 
     using 
      
      Google.Cloud.Translation.V2 
     
     ; 
     using 
      
     Microsoft.AspNetCore.Mvc 
     ; 
     using 
      
     Microsoft.Extensions.Logging 
     ; 
     using 
      
     System 
     ; 
     using 
      
     System.Collections.Generic 
     ; 
     using 
      
     System.Text 
     ; 
     using 
      
     System.Threading.Tasks 
     ; 
     namespace 
      
     TranslateWorker.Controllers 
     { 
      
     /// <summary> 
      
     /// The message Pubsub posts to our controller. 
      
     /// </summary> 
      
     public 
      
     class 
      
     PostMessage 
      
     { 
      
     public 
      
     PubsubMessage 
      
     message 
      
     { 
      
     get 
     ; 
      
     set 
     ; 
      
     } 
      
     public 
      
     string 
      
     subscription 
      
     { 
      
     get 
     ; 
      
     set 
     ; 
      
     } 
      
     } 
      
     /// <summary> 
      
     /// Pubsub's inner message. 
      
     /// </summary> 
      
     public 
      
     class 
      
     PubsubMessage 
      
     { 
      
     public 
      
     string 
      
     data 
      
     { 
      
     get 
     ; 
      
     set 
     ; 
      
     } 
      
     public 
      
     string 
      
     messageId 
      
     { 
      
     get 
     ; 
      
     set 
     ; 
      
     } 
      
     public 
      
     Dictionary<string 
     , 
      
     string 
    >  
     attributes 
      
     { 
      
     get 
     ; 
      
     set 
     ; 
      
     } 
      
     } 
      
     [Route("api/[controller] 
     ")] 
      
     [ApiController] 
      
     public 
      
     class 
      
     TranslateController 
      
     : 
      
     ControllerBase 
      
     { 
      
     private 
      
     readonly 
      
     ILogger<TranslateController> 
      
     _logger 
     ; 
      
     private 
      
     readonly 
      
      FirestoreDb 
     
      
     _firestore 
     ; 
      
     private 
      
     readonly 
      
      TranslationClient 
     
      
     _translator 
     ; 
      
     // The Firestore collection where we store translations. 
      
     private 
      
     readonly 
      
      CollectionReference 
     
      
     _translations 
     ; 
      
     public 
      
     TranslateController 
     ( 
     ILogger<TranslateController> 
      
     logger 
     , 
      
      FirestoreDb 
     
      
     firestore 
     , 
      
      TranslationClient 
     
      
     translator 
     ) 
      
     { 
      
     _logger 
      
     = 
      
     logger 
      
     ?? 
      
     throw 
      
     new 
      
     ArgumentNullException 
     ( 
     nameof 
     ( 
     logger 
     )); 
      
     _firestore 
      
     = 
      
     firestore 
      
     ?? 
      
     throw 
      
     new 
      
     ArgumentNullException 
     ( 
      
     nameof 
     ( 
     firestore 
     )); 
      
     _translator 
      
     = 
      
     translator 
      
     ?? 
      
     throw 
      
     new 
      
     ArgumentNullException 
     ( 
      
     nameof 
     ( 
     translator 
     )); 
      
     _translations 
      
     = 
      
     _firestore 
     . 
      Collection 
     
     ( 
     "Translations" 
     ); 
      
     } 
      
     /// <summary> 
      
     /// Handle a posted message from Pubsub. 
      
     /// </summary> 
      
     /// <param name="request">The message Pubsub posts to this process.</param> 
      
     /// <returns>NoContent on success.</returns> 
      
     [HttpPost] 
      
     public 
      
     async 
      
     Task<IActionResult> 
      
     Post 
     ([ 
     FromBody 
     ] 
      
     PostMessage 
      
     request 
     ) 
      
     { 
      
     // Unpack the message from Pubsub. 
      
     string 
      
     sourceText 
     ; 
      
     try 
      
     { 
      
     byte 
     [] 
      
     data 
      
     = 
      
     Convert 
     . 
     FromBase64String 
     ( 
     request 
     . 
     message 
     . 
     data 
     ); 
      
     sourceText 
      
     = 
      
     Encoding 
     . 
     UTF8 
     . 
     GetString 
     ( 
     data 
     ); 
      
     } 
      
     catch 
      
     ( 
     Exception 
      
     e 
     ) 
      
     { 
      
     _logger 
     . 
     LogError 
     ( 
     1 
     , 
      
     e 
     , 
      
     "Bad request" 
     ); 
      
     return 
      
     BadRequest 
     (); 
      
     } 
      
     // Translate the source text. 
      
     _logger 
     . 
     LogDebug 
     ( 
     2 
     , 
      
     "Translating {0} to Spanish." 
     , 
      
     sourceText 
     ); 
      
     var 
      
     result 
      
     = 
      
     await 
      
     _translator 
     . 
      TranslateTextAsync 
     
     ( 
     sourceText 
     , 
      
     "es" 
     ); 
      
     // Store the result in Firestore. 
      
     Translation 
      
     translation 
      
     = 
      
     new 
      
     Translation 
     () 
      
     { 
      
     TimeStamp 
      
     = 
      
     DateTime 
     . 
     UtcNow 
     , 
      
     SourceText 
      
     = 
      
     sourceText 
     , 
      
     TranslatedText 
      
     = 
      
     result 
     . 
      TranslatedText 
     
      
     }; 
      
     _logger 
     . 
     LogDebug 
     ( 
     3 
     , 
      
     "Saving translation {0} to {1}." 
     , 
      
     translation 
     . 
      TranslatedText 
     
     , 
      
     _translations 
     . 
      Path 
     
     ); 
      
     await 
      
     _translations 
     . 
      Document 
     
     ( 
     request 
     . 
     message 
     . 
     messageId 
     ) 
      
     . 
      SetAsync 
     
     ( 
     translation 
     ); 
      
     // Return a success code. 
      
     return 
      
     NoContent 
     (); 
      
     } 
      
     /// <summary> 
      
     /// Serve a root page so Cloud Run knows this process is healthy. 
      
     /// </summary> 
      
     [Route("/")] 
      
     public 
      
     IActionResult 
      
     Index 
     () 
      
     { 
      
     return 
      
     Content 
     ( 
     "Serving translate requests..." 
     ); 
      
     } 
      
     } 
     } 
     
    

Deploying the TranslateWorker service

  • In the BackgroundProcessing directory, run the PowerShell script to build and deploy the service to Cloud Run:

    PublishTo-CloudRun.ps1

Understanding the PublishTo-CloudRun.ps1 script

The PublishTo-CloudRun.ps1 script publishes the service to Cloud Run, and protects the TranslateWorker service from being abused. If the service permitted all incoming connections, then anyone could post translate requests to the controller and thereby incur costs. Therefore, you set up the service to only accept POST requests from Pub/Sub.

The script does the following:

  1. Builds the app locally using dotnet publish .
  2. Builds a container that runs the app using Cloud Build .
  3. Deploys the app to Cloud Run.
  4. Enables the project to create Pub/Sub authentication tokens.
  5. Creates a service account to represent the Pub/Sub subscription identity.
  6. Gives the service account permission to invoke TranslateWorker service.
  7. Creates a Pub/Sub topic and subscription.

     # 1. Build the application locally.
    dotnet publish -c Release
    
    # Collect some details about the project that we'll need later.
    $projectId = gcloud config get-value project
    $projectNumber = gcloud projects describe $projectId --format="get(projectNumber)"
    $region = "us-central1"
    
    # 2. Use Google Cloud Build to build the worker's container and publish to Google
    # Container Registry.
    gcloud builds submit --tag gcr.io/$projectId/translate-worker `
        TranslateWorker/bin/Release/netcoreapp2.1/publish
    
    # 3. Run the container with Google Cloud Run.
    gcloud beta run deploy translate-worker --region $region --platform managed `
        --image gcr.io/$projectId/translate-worker --no-allow-unauthenticated
    $url = gcloud beta run services describe translate-worker --platform managed `
        --region $region --format="get(status.address.hostname)"
    
    # 4. Enable the project to create pubsub authentication tokens.
    gcloud projects add-iam-policy-binding $projectId `
         --member=serviceAccount:service-$projectNumber@gcp-sa-pubsub.iam.gserviceaccount.com `
         --role=roles/iam.serviceAccountTokenCreator
    
    # 5. Create a service account to represent the Cloud Pub/Sub subscription identity.
    $serviceAccountExists = gcloud iam service-accounts describe `
        cloud-run-pubsub-invoker@$projectId.iam.gserviceaccount.com 2> $null
    if (-not $serviceAccountExists) {
        gcloud iam service-accounts create cloud-run-pubsub-invoker `
            --display-name "Cloud Run Pub/Sub Invoker"
    }
    
    # 6. For Cloud Run, give this service account permission to invoke 
    # translate-worker service.
    gcloud beta run services add-iam-policy-binding translate-worker `
         --member=serviceAccount:cloud-run-pubsub-invoker@$projectId.iam.gserviceaccount.com `
         --role=roles/run.invoker --region=$region
    
    # 7. Create a pubsub topic and subscription, if they don't already exist.
    $topicExists = gcloud pubsub topics describe translate-requests 2> $null 
    if (-not $topicExists) {
        gcloud pubsub topics create translate-requests
    }
    $subscriptionExists = gcloud pubsub subscriptions describe translate-requests 2> $null
    if ($subscriptionExists) {
        gcloud beta pubsub subscriptions modify-push-config translate-requests `
            --push-endpoint $url/api/translate `
            --push-auth-service-account cloud-run-pubsub-invoker@$projectId.iam.gserviceaccount.com
    } else {
        gcloud beta pubsub subscriptions create translate-requests `
            --topic translate-requests --push-endpoint $url/api/translate `
            --push-auth-service-account cloud-run-pubsub-invoker@$projectId.iam.gserviceaccount.com
    } 
    

Understanding the TranslateUI service

The TranslateUI service renders a web page that displays recent translations, and accepts requests for new translations.

  • The StartUp class configures an ASP.NET app and creates Pub/Sub and Firestore clients.

      using 
      
      Google.Apis.Auth.OAuth2 
     
     ; 
     using 
      
      Google.Cloud.Firestore 
     
     ; 
     using 
      
      Google.Cloud.PubSub.V1 
     
     ; 
     using 
      
     Microsoft.AspNetCore.Builder 
     ; 
     using 
      
     Microsoft.AspNetCore.Hosting 
     ; 
     using 
      
     Microsoft.AspNetCore.Mvc 
     ; 
     using 
      
     Microsoft.Extensions.Configuration 
     ; 
     using 
      
      Microsoft.Extensions.DependencyInjection 
     
     ; 
     using 
      
     System 
     ; 
     using 
      
     System.Net.Http 
     ; 
     namespace 
      
     TranslateUI 
     { 
      
     public 
      
     class 
      
     Startup 
      
     { 
      
     public 
      
     Startup 
     ( 
     IConfiguration 
      
     configuration 
     ) 
      
     { 
      
     Configuration 
      
     = 
      
     configuration 
     ; 
      
     } 
      
     public 
      
     IConfiguration 
      
     Configuration 
      
     { 
      
     get 
     ; 
      
     } 
      
     // This method gets called by the runtime. Use this method to add services to the container. 
      
     public 
      
     void 
      
     ConfigureServices 
     ( 
     IServiceCollection 
      
     services 
     ) 
      
     { 
      
     services 
     . 
     AddSingleton<FirestoreDb> 
     ( 
      
     provider 
      
     = 
    >  
      FirestoreDb 
     
     . 
      Create 
     
     ( 
     GetFirestoreProjectId 
     ())); 
      
     services 
     . 
     AddSingleton<PublisherClient> 
     ( 
      
     provider 
      
     = 
    >  
      PublisherClient 
     
     . 
      CreateAsync 
     
     ( 
     new 
      
      TopicName 
     
     ( 
      
     GetProjectId 
     (), 
      
     GetTopicName 
     ())). 
      Result 
     
     ); 
      
     services 
     . 
     AddMvc 
     (). 
     SetCompatibilityVersion 
     ( 
     CompatibilityVersion 
     . 
     Version_2_1 
     ); 
      
     } 
      
     // This method gets called by the runtime. Use this method to configure the HTTP request pipeline. 
      
     public 
      
     void 
      
     Configure 
     ( 
     IApplicationBuilder 
      
     app 
     , 
      
     IHostingEnvironment 
      
     env 
     ) 
      
     { 
      
     if 
      
     ( 
     env 
     . 
     IsDevelopment 
     ()) 
      
     { 
      
     app 
     . 
     UseDeveloperExceptionPage 
     (); 
      
     } 
      
     else 
      
     { 
      
     app 
     . 
     UseExceptionHandler 
     ( 
     "/Home/Error" 
     ); 
      
     // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts. 
      
     app 
     . 
     UseHsts 
     (); 
      
     } 
      
     app 
     . 
     UseHttpsRedirection 
     (); 
      
     app 
     . 
     UseStaticFiles 
     (); 
      
     app 
     . 
     UseCookiePolicy 
     (); 
      
     app 
     . 
     UseMvc 
     ( 
     routes 
      
     = 
    >  
     { 
      
     routes 
     . 
     MapRoute 
     ( 
      
     name 
     : 
      
     "default" 
     , 
      
     template 
     : 
      
     "{controller=Home}/{action=Index}/{id?}" 
     ); 
      
     }); 
      
     } 
      
     } 
     } 
     
    
  • The index handler Index gets all existing translations from Firestore and fills a ViewModel with the list:

      using 
      
      Google.Cloud.Firestore 
     
     ; 
     using 
      
      Google.Cloud.PubSub.V1 
     
     ; 
     using 
      
      Google.Protobuf 
     
     ; 
     using 
      
     Microsoft.AspNetCore.Mvc 
     ; 
     using 
      
     System.Diagnostics 
     ; 
     using 
      
     System.Linq 
     ; 
     using 
      
     System.Threading.Tasks 
     ; 
     using 
      
     TranslateUI.Models 
     ; 
     namespace 
      
     TranslateUI.Controllers 
     { 
      
     public 
      
     class 
      
     HomeController 
      
     : 
      
     Controller 
      
     { 
      
     private 
      
     readonly 
      
      FirestoreDb 
     
      
     _firestore 
     ; 
      
     private 
      
     readonly 
      
      PublisherClient 
     
      
     _publisher 
     ; 
      
     private 
      
      CollectionReference 
     
      
     _translations 
     ; 
      
     public 
      
     HomeController 
     ( 
      FirestoreDb 
     
      
     firestore 
     , 
      
      PublisherClient 
     
      
     publisher 
     ) 
      
     { 
      
     _firestore 
      
     = 
      
     firestore 
     ; 
      
     _publisher 
      
     = 
      
     publisher 
     ; 
      
     _translations 
      
     = 
      
     _firestore 
     . 
      Collection 
     
     ( 
     "Translations" 
     ); 
      
     } 
      
     [HttpPost] 
      
     [HttpGet] 
      
     public 
      
     async 
      
     Task<IActionResult> 
      
     Index 
     ( 
     string 
      
     SourceText 
     ) 
      
     { 
      
     // Look up the most recent 20 translations. 
      
     var 
      
     query 
      
     = 
      
     _translations 
     . 
      OrderByDescending 
     
     ( 
     "TimeStamp" 
     ) 
      
     . 
     Limit 
     ( 
     20 
     ); 
      
     var 
      
     snapshotTask 
      
     = 
      
     query 
     . 
     GetSnapshotAsync 
     (); 
      
     if 
      
     ( 
     ! 
     string 
     . 
     IsNullOrWhiteSpace 
     ( 
     SourceText 
     )) 
      
     { 
      
     // Submit a new translation request. 
      
     await 
      
     _publisher 
     . 
      PublishAsync 
     
     ( 
     new 
      
      PubsubMessage 
     
     () 
      
     { 
      
     Data 
      
     = 
      
      ByteString 
     
     . 
      CopyFromUtf8 
     
     ( 
     SourceText 
     ) 
      
     }); 
      
     } 
      
     // Render the page. 
      
     var 
      
     model 
      
     = 
      
     new 
      
     HomeViewModel 
     () 
      
     { 
      
     Translations 
      
     = 
      
     ( 
     await 
      
     snapshotTask 
     ). 
     Documents 
     . 
     Select 
     ( 
      
     doc 
      
     = 
    >  
     doc 
     . 
     ConvertTo<Translation> 
     ()). 
     ToList 
     (), 
      
     SourceText 
      
     = 
      
     SourceText 
      
     }; 
      
     return 
      
     View 
     ( 
     model 
     ); 
      
     } 
      
     [ResponseCache(Duration = 0, Location = ResponseCacheLocation.None, NoStore = true)] 
      
     public 
      
     IActionResult 
      
     Error 
     () 
      
     { 
      
     return 
      
     View 
     ( 
     new 
      
     ErrorViewModel 
      
     { 
      
     RequestId 
      
     = 
      
     Activity 
     . 
      Current 
     
     ?. 
     Id 
      
     ?? 
      
     HttpContext 
     . 
     TraceIdentifier 
      
     }); 
      
     } 
      
     } 
     } 
     
    
  • New translations are requested by submitting an HTML form. The request translation handler validates the request, and publishes a message to Pub/Sub:

      // Submit a new translation request. 
     await 
      
     _publisher 
     . 
     PublishAsync 
     ( 
     new 
      
     PubsubMessage 
     () 
     { 
      
     Data 
      
     = 
      
     ByteString 
     . 
     CopyFromUtf8 
     ( 
     SourceText 
     ) 
     }); 
     
    

Deploying the TranslateUI service

  • In the BackgroundProcessing directory, run the PowerShell script to build and deploy the service to Cloud Run:

    ./PublishTo-CloudRun.ps1

Understanding the PublishTo-CloudRun.ps1 script

The PublishTo-CloudRun.ps1 script publishes the app to Cloud Run.

The script does the following:

  1. Builds the app locally using dotnet publish .
  2. Builds a container that runs the app by using Cloud Build .
  3. Deploys the app to Cloud Run.

     # 1. Build the application locally.
    dotnet publish -c Release
    # 2. Use Google Cloud Build to build the UI's container and publish to Google
    # Container Registry. 
    gcloud builds submit --tag gcr.io/$projectId/translate-ui `
        TranslateUI/bin/Release/netcoreapp2.1/publish
    
    # 3. Run the container with Google Cloud Run.
    gcloud beta run deploy translate-ui --region $region --platform managed `
        --image gcr.io/$projectId/translate-ui --allow-unauthenticated 
    

Testing the app

After successfully running the PublishTo-CloudRun.ps1 script, try requesting a translation.

  1. The final command in the PublishTo-CloudRun.ps1 script tells you the URL for your UI service. In your terminal window, find the URL for the TranslateUI service:

    gcloud  
    beta  
    run  
    services  
    describe  
    translate-ui  
    --region  
     $region 
      
    --format = 
     "get(status.address.hostname)" 
    
  2. In your browser, go to the URL that you got from the previous step.

    There is a page with an empty list of translations and a form to request new translations.

  3. In the Text to translatefield, enter some text to translate, for example, Hello, World.

  4. Click Submit.

  5. To refresh the page, click Refresh . There is a new row in the translation list. If you don't see a translation, wait a few more seconds and try again. If you still don't see a translation, see the next section about debugging the app.

Debugging the app

If you cannot connect to your Cloud Run service or don't see new translations, check the following:

  • Check that the PublishTo-CloudRun.ps1 script successfully completed and didn't output any errors. If there were errors (for example, message=Build failed ), fix them, and try running again.

  • Check for errors in the logs:

    1. In the Google Cloud console, go to the Cloud Run page.

      Go to Cloud Run page

    2. Click the service name, translate-ui .

    3. Click Logs.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the Google Cloud project

  1. In the Google Cloud console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete .
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the Cloud Run services.

  • Delete the Cloud Run services you created in this tutorial:

    gcloud  
    beta  
    run  
    services  
    delete  
    --region = 
     $region 
      
    translate-ui
    gcloud  
    beta  
    run  
    services  
    delete  
    --region = 
     $region 
      
    translate-worker

What's next

Create a Mobile Website
View Site in Mobile | Classic
Share by: