Console
    Contact Us Start free

    Cloud Storage FUSE configuration file

    This document describes how to use a Cloud Storage FUSE configuration file to configure the behavior of Cloud Storage FUSE in a persistent manner. To use the configuration file, specify the path to the configuration file in the --config-file option when you run the gcsfuse command .

    The Cloud Storage FUSE configuration file is a YAML file that uses the following format and fields. Behavior controlled by the configuration file fields can also be set using the Cloud Storage FUSE CLI .

    Cloud Storage FUSE configuration file format and fields

    The following YAML configuration example shows the format and fields available in Cloud Storage FUSE.

     app-name 
 : 
  
  "APP_NAME" 
 
 logging 
 : 
  
 file-path 
 : 
  
  "FILE_PATH" 
 
  
 format 
 : 
  
  FORMAT 
 
  
 severity 
 : 
  
  SEVERITY 
 
  
 log-rotate 
 : 
  
 max-file-size-mb 
 : 
  
  MAX_FILE_SIZE 
 
  
 backup-file-count 
 : 
  
  BACKUP_FILE_COUNT 
 
  
 compress 
 : 
  
  COMPRESS 
 
 cache-dir 
 : 
  
  "CACHE_DIR" 
 
 file-cache 
 : 
  
 max-size-mb 
 : 
  
  MAX_SIZE 
 
  
 cache-file-for-range-read 
 : 
  
  CACHE_FILE_FOR_RANGE_READ 
 
  
 enable-parallel-downloads 
 : 
  
  ENABLE_PARALLEL_DOWNLOADS 
 
  
 parallel-downloads-per-file 
 : 
  
  PARALLEL_DOWNLOADS_PER_FILE 
 
  
 max-parallel-downloads 
 : 
  
  MAX_PARALLEL_DOWNLOADS 
 
  
 download-chunk-size-mb 
 : 
  
  DOWNLOAD_CHUNK_SIZE 
 
 metadata-cache 
 : 
  
 enable-nonexistent-type-cache 
 : 
  
  ENABLE_NONEXISTENT_TYPE_CACHE 
 
  
 negative-ttl-secs 
 : 
  
  ENABLE_NEGATIVE_TTL_SECS 
 
  
 stat-cache-max-size-mb 
 : 
  
  STAT_CACHE_MAX_SIZE 
 
  
 ttl-secs 
 : 
  
  TTL_SECS 
 
  
 type-cache-max-size-mb 
 : 
  
  TYPE_CACHE_MAX_SIZE 
 
 only-dir 
 : 
  
  "ONLY_DIR" 
 
 gcs-auth 
 : 
  
 anonymous-access 
 : 
  
  ANONYMOUS_ACCESS 
 
  
 key-file 
 : 
  
  "KEY_FILE" 
 
  
 reuse-token-from-url 
 : 
  
  REUSE_TOKEN_FROM_URL 
 
  
 token-url 
 : 
  
  "TOKEN_URL" 
 
 gcs-connection 
 : 
  
 billing-project 
 : 
  
  "BILLING_PROJECT" 
 
  
 client-protocol 
 : 
  
  CLIENT_PROTOCOL 
 
  
 custom-endpoint 
 : 
  
  "CUSTOM_ENDPOINT" 
 
  
 http-client-timeout 
 : 
  
  HTTP_CLIENT_TIMEOUT 
 
  
 limit-bytes-per-sec 
 : 
  
  "LIMIT_BYTES_PER_SEC" 
 
  
 limit-ops-per-sec 
 : 
  
  "LIMIT_OPS_PER_SEC" 
 
  
 max-conns-per-host 
 : 
  
  MAX_CONNS_PER_HOST 
 
  
 max-idle-conns-per-host 
 : 
  
  MAX_IDLE_CONNS_PER_HOST 
 
  
 sequential-read-size-mb 
 : 
  
  SEQUENTIAL_READ_SIZE 
 
 implicit-dirs 
 : 
  
  IMPLICIT_DIRS 
 
 file-system 
 : 
  
 kernel-list-cache-ttl-secs 
 : 
  
  KERNEL_LIST_CACHE_TTL_SECS 
 
  
 ignore-interrupts 
 : 
  
  IGNORE_INTERRUPTS 
 
  
 dir-mode 
 : 
  
  "DIR_MODE" 
 
  
 file-mode 
 : 
  
  "FILE_MODE" 
 
  
 fuse-options 
 : 
  
  FUSE_OPTIONS 
 
  
 gid 
 : 
  
  GID 
 
  
 rename-dir-limit 
 : 
  
  RENAME_DIR_LIMIT 
 
  
 temp-dir 
 : 
  
  "TEMP_DIR" 
 
  
 uid 
 : 
  
  UID 
 
 foreground 
 : 
  
  FOREGROUND 
 
 gcs-retries 
 : 
  
 max-retry-attempts 
 : 
  
  MAX_RETRY_ATTEMPTS 
 
  
 max-retry-sleep 
 : 
  
  MAX_RETRY_SLEEP 
 
  
 multiplier 
 : 
  
  "MULTIPLIER" 
 
 metrics 
 : 
  
 cloud-metrics-export-interval-secs 
 : 
  
  CLOUD_METRICS_EXPORT_INTERVAL 
 
  
 prometheus-port 
 : 
  
  PROMETHEUS_PORT 
 
 debug 
 : 
  
 log-mutex 
 : 
  
  LOG_MUTEX 
 
  
 exit-on-invariant-violation 
 : 
  
  EXIT_ON_INVARIANT_VIOLATION 
 
 write 
 : 
  
 enable-streaming-writes 
 : 
  
  STREAMING_WRITES 
 
  
 global-max-blocks 
 : 
  
  MAXIMUM_GLOBAL_BLOCKS 
 
 read 
 : 
  
 enable-buffered-read 
 : 
  
  ENABLE_BUFFERED_READ 
 
  
 global-max-blocks 
 : 
  
  MAXIMUM_GLOBAL_READ_BLOCKS 
 
 profile 
 : 
  
  PROFILE

    Cloud Storage FUSE configuration file fields

    The following table describes the fields you can specify in your Cloud Storage FUSE configuration file. Unless stated otherwise, all fields are optional.

    Field
    Description
    Valid value
    Default value
    The application name of the mount.
    String value, for example: "my-bucket-mount" .
    ""
    The path to the log file where logs will be written. If this field is unspecified, the logs are routed to stdout when Cloud Storage FUSE runs in foreground mode and to syslogs when Cloud Storage FUSE runs in background mode.
    String value, for example: "/var/log" .
    ""
    Specifies the format of the log file.
    • text
    • json
    json

    The severity level you want Cloud Storage FUSE to generate logs for. The severity levels are ordered from lowest severity to highest severity. For example, when you specify warning , Cloud Storage FUSE generates logs for warnings and errors. Generally, we recommend using the info severity level.

    • off : disables all logging.
    • error : contains messages about serious issues that prevent Cloud Storage FUSE from completing an operation, or indicate a failure such as mount failures, permission errors, and critical input/output (I/O) errors.
    • warning : contains messages indicating potential issues that aren't critical but can lead to issues if not addressed. Potential issues include non-fatal connection issues, deprecation warnings, resource contention that might cause slight performance degradation, and retries for transient failures. This severity level also includes information provided in the error severity level.
    • info : contains logging information such as startup and shutdown messages, or information about successful mount operations, progress updates, and configuration settings. This severity level also includes information about the warning and error severity levels.
    • debug : contains logging information such as startup and shutdown messages, or information about successful mount operations, progress updates, and configuration settings. This severity level also includes the information provided in the info , warning , and error severity levels.
    • trace : contains granular details about each Cloud Storage FUSE operation and function call, outlining the gcsfuse interaction with the kernel FUSE driver and Cloud Storage. This severity level also includes the details provided in debug , info , warning , and error severity levels.
    info
    The maximum size in megabytes (MB) that log files can reach before being rotated.
    Integer. The minimum value is 1 .
    512
    The maximum number of rotated log files to retain, excluding the active file that logs are written to.
    • Integer
    • 0 : Retains all rotated log files
    10
    Specifies whether rotated log files are compressed using gzip .
    Boolean value: true , false
    true

    Enables the file cache and specifies the directory for storing file cache data.

    A path, for example: "/tmp/gcsfuse-cache-path" . An empty value indicates this field is disabled. This field is disabled by default.

    Specifies the maximum size in MiB that the file cache can use and lets you limit the total capacity the file cache can use within its mounted directory.

    • Integer
    • -1 : Specifies the use of the cache's entire available capacity in the directory you specify for cache-dir . This is the default only if cache-dir is passed.
    • 0 : Disables the file cache.
    -1
    Determines whether the full object should be downloaded asynchronously and stored in the Cloud Storage FUSE cache directory when the first read is completed from a non-zero offset. This field should be set to true if you plan to perform several random reads or partial reads.
    Boolean value: true , false
    false

    Accelerates reads of large files by using the file cache directory as a prefetch buffer using multiple workers to download multiple parts of a file in parallel.

    Parallel downloads are automatically enabled when you enable file caching. To learn more about file caching, see Use Cloud Storage FUSE file caching .

    To learn more about parallel downloads and how to configure supporting properties, Parallel downloads .

    Boolean value: true , false .
    true
    Specifies the number of maximum goroutines to spawn per file to download the object from Cloud Storage into the file cache.
    Integer
    16
    The maximum number of goroutines that can be spawned at any given time across all the download jobs of files.
    • Integer
    • -1 : Specifies unlimited parallel downloads.
    • 0 : Disables parallel downloads. Can only be used if --enable-parallel-downloads is not passed or is passed as false .
    Twice the number of CPU cores on your machine or 16 , whichever is higher.
    Specifies the size of each read request in MiB that each goroutine makes to Cloud Storage when downloading the object into the file cache.
    Integer
    200
    Creates a type cache entry with the type NonexistentType if a file isn't found in Cloud Storage. If the file gets created in Cloud Storage but the NonexistentType entry for the file is cached, then Cloud Storage FUSE cannot request that file until the NonexistentType entry is removed from the type cache.
    Boolean value: true , false
    false
    The maximum size memory that the stat cache can use, in MiB. The stat cache is always entirely kept in memory.
    • Integer. We recommend the following:
      • 32 if your workload involves up to 20,000 files.
      • If your workload is larger than 20,000 files, increase the size by values of 10 for every additional 6,000 files, where the stat cache uses an average of 1,500 MiB per file.
    • -1 : Sets no limit, where the stat cache use as much memory as needed.
    • 0 : Disables the stat cache.
    32

    Defines the time to live (TTL) in seconds of negative stat cache entries, which store results for non-existent files in the cache.

    • Integer representing seconds, for example: 10 (10 seconds).
    • 0 : Disables negative stat caching.
    • -1 : Allows unlimited negative stat caching and disables a TTL expiration.
    5
    Defines the time to live (TTL) in seconds of cached metadata entries.
    • Integer representing seconds, for example: 30 (30 seconds).
    • -1 : Bypass TTL expiration and serve files from the cache whenever they're available.
    • 0 : Use the most up-to-date file. Using this value issues a Get metadata call to make sure that the object generation for the file in the cache matches what's stored in Cloud Storage.
    60
    The maximum size in MiB per directory that the type cache can use. The type cache is always entirely kept in memory.
    • Integer. We recommend the following:
      • 4 if the maximum number of files within a single directory from the bucket you're mounting contains 20,000 files or less.
      • If the maximum number of files within a single directory that you're mounting contains more than 20,000 files, increase the value by 1 for every 5,000 files, which is an average of around 200 bytes per file.
    • -1 : Specifies no limit and lets the type cache use as much memory as needed.
    • 0 : Disables the type cache.
    4
    Mounts only a specific directory within a bucket.
    A path, for example: "/etc/gcsfuse.yaml" .
    Disables authentication for requests. This field should be set if you're using a custom endpoint that doesn't support authentication. This field should also be set if you're using Cloud Storage FUSE with public buckets.
    Boolean value: true , false
    false
    Specifies an absolute path to the credential JSON key file for authenticating requests to Cloud Storage. By default, Cloud Storage FUSE uses Application Default Credentials to authenticate requests.
    A URL.
    When this field is not set, Application Default Credentials are used.
    Specifies whether to reuse the token acquired from --token-url .
    Boolean value: true , false
    true
    Specifies a URL for getting an access token when the --key-file is absent.
    A URL.
    Specifies a project to use for billing when the mounted bucket is accessed. This field is often required when mounting a bucket enabled with Requester Pays .
    String value representing a "project ID" .
    ""
    Specifies the protocol used for communicating with the Cloud Storage backend.
    http1
    Specifies an alternative custom endpoint for fetching data. The custom endpoint must support the equivalent resources and operations as the Cloud Storage JSON endpoint, storage. UNIVERSE_DOMAIN_NAME :443 . If a custom endpoint isn't specified, Cloud Storage FUSE uses the global Cloud Storage JSON API endpoint, storage.googleapis.com:443 . If authentication isn't supported on the custom endpoint you specify, set the anonymous-access field to true to bypass authentication.
    An endpoint, for example: storage.googleapis.com:443 .
    Specifies how long the Cloud Storage FUSE HTTP client can wait to get a response from the server before timing out.
    Duration, for example: 1h10m10s for 1 hour, 10 minutes, and 10 seconds. 0s specifies no timeout.
    0s , which specifies no timeout
    Specifies the bandwidth limit at which Cloud Storage FUSE can read data from Cloud Storage, measured over a 30-second window.
    "-1" , which specifies no limit.
    Specifies a limit for operations performed per second, measured over a 30-second window.
    Floating point number. -1 specifies no limit.
    "-1"
    Specifies the maximum number of TCP connections allowed per server. This becomes effective when --client-protocol is set to http1 .
    0
    Specifies the maximum number of idle connections allowed per server. This becomes effective when --client-protocol is set to http1 .
    Integer between 0 and 2147483647 . 0 specifies no limit on TCP connections.
    0
    Specifies the chunk size of the data to be downloaded from Cloud Storage, in megabytes (MB).
    Integer between 1 and 1024 .
    200
    Implicitly includes folders and managed folders . See Files and directories in the Cloud Storage FUSE GitHub documentation for more information.
    Boolean value: true , false
    false
    Enables the list cache and defines the time to live (TTL) in seconds of cached list entries. The list cache is kept in memory in the page cache, which is controlled by the kernel based on available memory.
    • Integer representing seconds, for example: 10 (10 seconds).
    • 0 : Disables list caching.
    • -1 : Bypasses entry expiration and always returns the list response from the cache when it's available.
    0
    Instructs Cloud Storage FUSE to ignore system interrupt signals, like SIGINT triggered by Control+C . This prevents signals from terminating in-flight operations.
    Boolean value: true , false .
    true
    Permissions bits for directories, in octal.
    Integer between 000 and 777 (inclusive).
    "755"
    Specifies permissions bits for files, in octal.
    Integer between 000 and 777 (inclusive).
    "644"
    Specifies additional system-specific mount options.
    Specifies the Group Identifier (GID) owner of all inodes.
    • Integer representing a GID.
    • -1 : The GID of the caller is used.
    -1
    Allows the renaming of directories containing fewer descendants than the specified limit.
    Integer between 0 and 2147483647 .
    0
    Specifies a path to the temporary directory where writes are staged prior to being uploaded to Cloud Storage.
    A string path, for example: "/mnt/ssd/example-user-gcsfuse-temp-dir" .
    "/tmp"
    Specifies the User Identifier (UID) owner of all inodes.
    • Integer representing a UID.
    • -1 : The UID of the caller is used.
    -1
    Runs the gcsfuse command in the foreground.
    Boolean value: true , false
    false
    Specifies the maximum number of times an operation is retried if the operation fails, preventing endless retry loops.
    • Integer representing the maximum number of retry attempts, for example: 10 .
    • 0 : Allows unlimited retry attempts.
    0
    Specifies the maximum duration that Cloud Storage FUSE is allowed to sleep in a retry loop with exponential backoff . Once the backoff duration exceeds the specified maximum duration, the retry continues with the specified maximum duration.
    Duration, for example: 1h5m50s (1 hour, 5 minutes, and 50 seconds) or 60s (60 seconds).
    30s
    Specifies the multiplier for exponential backoff between consecutive retries.
    Floating-point number
    "2"

    Exports metrics to Cloud Monitoring with the specified interval.

    Integer representing a value in seconds, for example: 10 (10 seconds). 0 specifies no exporting.
    0

    Exposes Prometheus metrics endpoint on the specified port and /metrics path.

    Integer representing the port you want to specify.
    0
    Prints debug messages when a mutex is held too long. If this field is specified, the severity level of logs is automatically set to trace , which includes trace logs, debug logs, info logs, warning logs, and error logs.
    Boolean value: true , false .
    false
    Exits the program when internal variant violations are detected.
    Boolean value: true , false .
    false
    Controls the write path flow so that data is uploaded directly to Cloud Storage as its written instead of fully staging the write locally and uploading it upon close() or fsync() . For more information about streaming writes, see Reads/Writes in the Cloud Storage FUSE GitHub documentation.
    Boolean value: true , false .
    true

    Specifies the maximum number of data chunks available for streaming writes across all files. Each file uses one 32 MiB data chunk by default.

    • Integer
    • 0 : disables streaming writes.
    • -1 : sets the value to unlimited chunks.
    • Default value for low-specification machines with limited resources: 4
    • Default value for high-specification machines with a large amount of resources: 1600
    Specifies asynchronous prefetching of parts of a Cloud Storage object into an in-memory buffer, allowing subsequent reads to be served from the buffer instead of requiring network calls.
    Boolean value: true , false
    false

    Specifies the maximum number of blocks available for buffered reads across all file handles.

    • Integer
    • 0 : disables buffered reads.
    • -1 : sets the value to unlimited blocks.
    40
    Applies a predefined, optimized set of Cloud Storage FUSE configurations for caching, threading, and buffer sizes to help you achieve high performance for a specific workload type, such as training, serving, and checkpointing. To learn more about each predefined configuration based on your workload type, see Profile-based configurations for AI/ML workloads .
    • A string, for example, "aiml-checkpointing" .
    • "aiml-training" : optimizes performance for high throughput reads of large datasets and prevents Cloud GPUs and Cloud TPU hardware from waiting for data.
    • aiml-checkpointing : optimizes performance for high throughput writes for large files by drastically reducing the time it takes to save multi-gigabyte checkpoints, minimizing training pauses.
    • aiml-serving : optimizes performance for serving workloads by streamlining data access and applying caching mechanisms.
    ""

    Back to top

    Design a Mobile Site
    View Site in Mobile | Classic
    Share by: