Streaming and live tailing log entries

This document describes how to view your log entries in real time by streaming and live tailing. Streaming logs lets you view log entries in real time and is available in the Logs Explorer . Live tailing lets you view log entries in real time and is available as the gcloud CLI command gcloud alpha logging tail and as the Cloud Logging API method entries.tail .

When you view and analyze your logs using the Logs Explorer, gcloud logging read , or the API method entries.list , you're viewing log entries that Cloud Logging has stored. When you stream or live tail log entries, you're viewing log entries as your applications write them to the Cloud Logging API.

Stream logs in the Logs Explorer

In the Logs Explorer, you can view your logs data in real time using Stream logs. When you use Stream logs, you can add a query to stream only those logs that match the query. To stream logs, do the following:

1. In the Google Cloud console, go to the Logs Explorer page:

   Go to Logs Explorer

   If you use the search bar to find this page, then select the result whose subheading is Logging .

2. In the Queryfield, enter a query and then click Stream logs.

   As Logging writes the logs data, only logs that match the query are shown in the Query resultspane. If a query isn't provided, then Logging shows the recently stored log entries. Logs continue streaming until you select the scroll bar on the logs panel. When streaming has stopped, a Restart streamingbutton is shown.

   For information about queries, see Build queries in the Logs Explorer .

Use live tailing in the Google Cloud CLI

Live tailing lets you view your log entries in real time as Cloud Logging writes them, by using either the Google Cloud CLI or the Cloud Logging API.

Live tailing isn't supported for log buckets with field-level access controls , however you can stream logs for those buckets in the Logs Explorer.

For information on the API method for live tailing, see the entries.tail method.

Installing gcloud alpha logging tail

To use gcloud alpha logging tail , you need to have Python 3 and the grpcio Python package installed.

For instructions on how to install Python, see the Python page . For instructions on how to install the Python package manager, pip , that is needed to install the grpcio package, see The Python Package Installer page .

Complete the following steps to install gcloud alpha logging tail :

1. Verify that you have the Google Cloud CLI installed. For instructions on how to install the Google Cloud CLI, see Installing Google Cloud CLI .

2. Verify that you're using version 302.0.0 or greater of the gcloud CLI.

    gcloud version 
   

   For instructions on updating the gcloud CLI, see gcloud components update .

3. Install the gcloud CLI alpha components:

    gcloud components install alpha 
   

4. For MacOS, Linux, and Cloud Shell users:

   1. Install gRPC client libraries:

       sudo pip3 install grpcio 
      

   2. Set the environment variable CLOUDSDK_PYTHON_SITEPACKAGES to any value:

         
       export 
        
       CLOUDSDK_PYTHON_SITEPACKAGES 
       = 
       1 
       
      

5. Use the following commands to set your Google Cloud project ID and to authenticate:

    gcloud config set project PROJECT_ID 
   gcloud auth login 
   

   To get the project ID, see Creating and managing projects .

6. Verify that gcloud alpha logging tail is installed by running the following command:

    gcloud alpha logging tail 
   

   The command displays the following message:

   Initializing tail session.

   You are now viewing the log entries for your Google Cloud project as Logging writes them.

   

For more information on using live tailing, see the gcloud alpha logging tail reference guide .

Buffering and ordering

Because Logging can receive log entries out of chronological order, live tailing provides a buffer-window setting so you can balance the tradeoff between viewing the log entries as they are being written and viewing them in ascending order. You can set the buffer window between 0 and 60 seconds.

Note the following characteristics of the buffer window:

• The default buffer window is two seconds.

• Logging delays writing the log entries to log buckets for the duration of the buffer window.

• If a log entry is written outside of the buffer window, then Logging returns the log entries as they are received.

When configuring the buffer window, you make a tradeoff between viewing logs as they are written and viewing the entries out of order.

Limits and quotas

The following table lists the limits and quotas for live tailing:

Client limitations

For a Google Cloud project that writes lots of entries quickly, your client might be unable to consume them as quickly as they're being written. In this case, Logging limits the total number of entries sent, prioritizing the most recent entries. At the end of the tail session, Logging returns the number of entries that were not displayed due to the limits of the client.

Use live tailing with client libraries

Live tailing lets you view your log entries in real time as Cloud Logging writes them. For information on the API method for live tailing, see the entries.tail method.

Live tailing isn't supported for log buckets with field-level access controls , however you can stream logs for those buckets in the Logs Explorer.

This sample demonstrates live tailing log entries of a given logger.

  import 
  
 ( 
  
 "context" 
  
 "fmt" 
  
 "io" 
  
 logging 
  
 "cloud.google.com/go/logging/apiv2" 
  
 "cloud.google.com/go/logging/apiv2/loggingpb" 
 ) 
 // tailLogs creates a channel to stream log entries that were recently ingested for a project 
 func 
  
 tailLogs 
 ( 
 projectID 
  
 string 
 ) 
  
 error 
  
 { 
  
 // projectID := "your_project_id" 
  
 ctx 
  
 := 
  
 context 
 . 
 Background 
 () 
  
 client 
 , 
  
 err 
  
 := 
  
 logging 
 . 
  NewClient 
 
 ( 
 ctx 
 ) 
  
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 return 
  
 fmt 
 . 
 Errorf 
 ( 
 "NewClient error: %w" 
 , 
  
 err 
 ) 
  
 } 
  
 defer 
  
 client 
 . 
 Close 
 () 
  
 stream 
 , 
  
 err 
  
 := 
  
 client 
 . 
 TailLogEntries 
 ( 
 ctx 
 ) 
  
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 return 
  
 fmt 
 . 
 Errorf 
 ( 
 "TailLogEntries error: %w" 
 , 
  
 err 
 ) 
  
 } 
  
 defer 
  
 stream 
 . 
 CloseSend 
 () 
  
 req 
  
 := 
  
& loggingpb 
 . 
 TailLogEntriesRequest 
 { 
  
 ResourceNames 
 : 
  
 [] 
 string 
 { 
  
 "projects/" 
  
 + 
  
 projectID 
 , 
  
 }, 
  
 } 
  
 if 
  
 err 
  
 := 
  
 stream 
 . 
 Send 
 ( 
 req 
 ); 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 return 
  
 fmt 
 . 
 Errorf 
 ( 
 "stream.Send error: %w" 
 , 
  
 err 
 ) 
  
 } 
  
 // read and print two or more streamed log entries 
  
 for 
  
 counter 
  
 := 
  
 0 
 ; 
  
 counter 
 < 
 2 
 ; 
  
 { 
  
 resp 
 , 
  
 err 
  
 := 
  
 stream 
 . 
 Recv 
 () 
  
 if 
  
 err 
  
 == 
  
 io 
 . 
 EOF 
  
 { 
  
 break 
  
 } 
  
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 return 
  
 fmt 
 . 
 Errorf 
 ( 
 "stream.Recv error: %w" 
 , 
  
 err 
 ) 
  
 } 
  
 fmt 
 . 
 Printf 
 ( 
 "received:\n%v\n" 
 , 
  
 resp 
 ) 
  
 if 
  
 resp 
 . 
 Entries 
  
 != 
  
 nil 
  
 { 
  
 counter 
  
 += 
  
 len 
 ( 
 resp 
 . 
 Entries 
 ) 
  
 } 
  
 } 
  
 return 
  
 nil 
 } 
 

  import 
  
 com.google.cloud.logging. LogEntry 
 
 ; 
 import 
  
 com.google.cloud.logging. LogEntryServerStream 
 
 ; 
 import 
  
 com.google.cloud.logging. Logging 
 
 ; 
 import 
  
 com.google.cloud.logging. Logging 
. TailOption 
 
 ; 
 import 
  
 com.google.cloud.logging. LoggingOptions 
 
 ; 
 public 
  
 class 
 TailLogEntries 
  
 { 
  
 public 
  
 static 
  
 void 
  
 main 
 ( 
 String 
 [] 
  
 args 
 ) 
  
 throws 
  
 Exception 
  
 { 
  
 // TODO(developer): Optionally provide the logname as an argument. 
  
 String 
  
 logName 
  
 = 
  
 args 
 . 
 length 
 > 
 0 
  
 ? 
  
 args 
 [ 
 0 
 ] 
  
 : 
  
 "" 
 ; 
  
  LoggingOptions 
 
  
 options 
  
 = 
  
  LoggingOptions 
 
 . 
  getDefaultInstance 
 
 (); 
  
 try 
  
 ( 
  Logging 
 
  
 logging 
  
 = 
  
 options 
 . 
 getService 
 ()) 
  
 { 
  
 // Optionally compose a filter to tail log entries only from specific log 
  
  LogEntryServerStream 
 
  
 stream 
 ; 
  
 if 
  
 ( 
 logName 
  
 != 
  
 "" 
 ) 
  
 { 
  
 stream 
  
 = 
  
 logging 
 . 
  tailLogEntries 
 
 ( 
  
  TailOption 
 
 . 
 filter 
 ( 
  
 "logName=projects/" 
  
 + 
  
 options 
 . 
 getProjectId 
 () 
  
 + 
  
 "/logs/" 
  
 + 
  
 logName 
 )); 
  
 } 
  
 else 
  
 { 
  
 stream 
  
 = 
  
 logging 
 . 
  tailLogEntries 
 
 (); 
  
 } 
  
 System 
 . 
 out 
 . 
 println 
 ( 
 "start streaming.." 
 ); 
  
 for 
  
 ( 
  LogEntry 
 
  
 log 
  
 : 
  
 stream 
 ) 
  
 { 
  
 System 
 . 
 out 
 . 
 println 
 ( 
 log 
 ); 
  
 // cancel infinite streaming after receiving first entry 
  
 stream 
 . 
  cancel 
 
 (); 
  
 } 
  
 } 
  
 } 
 } 
 

  const 
  
 { 
 Logging 
 } 
  
 = 
  
 require 
 ( 
 ' @google-cloud/logging 
' 
 ); 
 const 
  
 logging 
  
 = 
  
 new 
  
  Logging 
 
 (); 
 /** 
 * TODO(developer): Replace logName with the name of your log. 
 */ 
 const 
  
 log 
  
 = 
  
 logging 
 . 
 log 
 ( 
 logName 
 ); 
 console 
 . 
 log 
 ( 
 'running tail log entries test' 
 ); 
 const 
  
 stream 
  
 = 
  
 log 
  
 . 
 tailEntries 
 ({ 
  
 filter 
 : 
  
 'timestamp > "2021-01-01T23:00:00Z"' 
 , 
  
 }) 
  
 . 
 on 
 ( 
 'error' 
 , 
  
 console 
 . 
 error 
 ) 
  
 . 
 on 
 ( 
 ' data 
' 
 , 
  
 resp 
  
 = 
>  
 { 
  
 console 
 . 
 log 
 ( 
 resp 
 . 
 entries 
 ); 
  
 console 
 . 
 log 
 ( 
 resp 
 . 
 suppressionInfo 
 ); 
  
 // If you anticipate many results, you can end a stream early to prevent 
  
 // unnecessary processing and API requests. 
  
  stream 
 
 . 
 end 
 (); 
  
 }) 
  
 . 
 on 
 ( 
 'end' 
 , 
  
 () 
  
 = 
>  
 { 
  
 console 
 . 
 log 
 ( 
 'log entry stream has ended' 
 ); 
  
 }); 
 // Note: to get all project logs, invoke logging.tailEntries