Go

This page documents rules for adding third-party Go code to //piper/third_party/golang.

IMPORTANT: Read go/thirdparty first. When creating a new directory, set *********************************** in the CL. For Go-specific questions, email emailremoved@.

What it's for

The third_party/golang directory is for code that is developed outside of Piper and imported into Piper using the //piper/third_party/golang/import.go tool. The purpose of this tree is to enforce common standards about how new versions of Go code are imported into Piper.

Open source code that is developed with Piper as the source of truth (and possibly exported) does not need to adhere to the requirements of the third_party/golang tree and should generally live elsewhere in third_party .

Code which is being temporarily developed in Piper with the intention of an eventual move to an external repository may be added to third_party/golang .

Go requirements

Go code checked in to third_party/golang must satisfy the following requirements:

1. Have the necessary required top-level files: OWNERS , BUILD , LICENSE , and METADATA .

2. Build using Blaze with no special options.

3. Follow a directory structure that is compatible with the open source Go toolchain . In particular, every directory may only have at most one Go package. It's a convention that the Go package itself be given a name identical to the directory that it lives under. This may cause the Go import path inside Google to stutter (e.g., "google3/third_party/golang/protobuf/v2/proto/proto" ) but is a consequence of differences between Blaze and the Go toolchain.

4. Have an entry under the Modules section of //piper/third_party/golang/CONFIGURATION, which is a mapping of directories under third_party/golang to the root Go import path for that module as respected by the open source Go toolchain. For example, the Go protobuf module has an external Go import path rooted at "google.golang.org/protobuf" , but is stored internally under the protobuf/v2 directory.

The last two rules exist to enable a systematic and reliable way to resolve open source Go import paths to Google-specific Go import paths.

Directory naming

Each external module should be stored in Piper in a sub-directory of third_party/golang . The directory name should not start with go . If upstream is gofoo , just call it foo if foo is unique enough. If there are many foo s, you can import "github.com/bar/go-foo" as barfoo . Groups of closely related packages are sometimes imported as bar/foo , though this is not required.

Importing code

Directories under third_party/golang must use the //piper/third_party/golang/import.go tool to manage importing of open source Go code into Piper. Entire modules should be imported even if only a small subset of the packages are needed. If the module contains more packages than necessary, a filter specified in the import configuration file can be used to import only the desired packages.

This section only covers functionally provided by the import.go tool. Users who use other tools (e.g., Copybara) to import or export code do so on at their own initiative and risk. The most technically challenging part of the import/export process is the mapping of internal and external Go import paths, which the import.go tool is specifically designed to handle. If you choose not to use import.go , please contact the emailremoved@ and let them know why the tool was insufficient.

First-time import

To import a Go module with import.go , first add an entry to the Modules section in //piper/third_party/golang/CONFIGURATION that maps the chosen Piper sub-directory to the open-source Go module path. For example:

  [ 
 Modules 
 ] 
 foo 
 : 
  
 github 
 . 
 com 
 / 
 user 
 / 
 foo 
 

Once an entry has been added, an import can be initiated with:

  go 
  
 run 
  
 third_party 
 / 
 golang 
 / 
 import 
 . 
 go 
  
 - 
 dir 
 = 
 foo 
 

It will automatically create the following files:

• GOIMPORT/CONFIGURATION : This file specifies how the import ought to operate. Every new import copies the contents of //piper/third_party/golang/CONFIGURATION.default.

• METADATA : An initial metadata file is created for the import. It is your responsibility to populate the "description" with something meaningful and any other attributes you deem necessary.

The following file may be present:

• LICENSE : The default configuration file has a rule that attempts to rename common names for the license file as LICENSE . However, it does not catch all cases and it may be your responsibility to provide the correct renaming.

The following files will need manual modification:

• BUILD : See /documentation/reference/thirdparty/documentation/#build.

• OWNERS : See /documentation/reference/thirdparty/documentation/#owners.

By default, import.go pulls the latest version of the module. If a specific version is desired, it can be specified with the -version flag. For example:

  go 
  
 run 
  
 third_party 
 / 
 golang 
 / 
 import 
 . 
 go 
  
 - 
 dir 
 = 
 foo 
  
 - 
 version 
 = 
 v1 
 .0.0 
 

The -version flag understands the special values "same" and "latest", which specifies that import.go should import at the current google3 version or the latest version, respectively. Specifying "same" is an error for the first import.

Updating imported code

Pull and merge the latest version of the module.

  go 
  
 run 
  
 third_party 
 / 
 golang 
 / 
 import 
 . 
 go 
  
 - 
 dir 
 = 
 foo 
 

Any google specific patches may result in merge conflicts. Review and fix any files that failed to patch. After fixing the conflicts, test your module and regenerate the MANIFEST and AUTOPATCHES files.

  blaze 
  
 { 
 build 
 , 
 test 
 } 
  
 //third_party/golang/foo/... 
 go 
  
 run 
  
 third_party 
 / 
 golang 
 / 
 import 
 . 
 go 
  
 - 
 dir 
 = 
 foo 
  
 - 
 version 
 = 
 same 
 

Getting a review

To declare that you know the rules on this page (or at least to get a faster review), make your CL description like:

  Import 
  
 `github.com/bar/foo` 
  
 as 
  
 `third_party/golang/foo` 
 This 
  
 package 
  
 will 
  
 be 
  
 used 
  
 by 
 : 
 [ 
  
 ] 
  
 Other 
  
 third_party 
  
 code 
 [ 
  
 ] 
  
 Packages 
  
 in 
  
 Google3 
 Running 
  
 `import.go --dir=foo -check` 
  
 reports 
  
 success 
 . 
 Checked 
  
 that 
  
 `blaze {build,test} //third_party/golang/foo/...` 
  
 works 
 . 
 

Send your CL to *********************************** for review.

If you need to add multiple packages, put each new package into a separate CL. If you're adding a package and its dependencies, you may find it helpful to use go/fig to manage the stack of CLs.

Visibility

Depending on the intended scope of the imported code, we recommend handling visibility in one of two ways:

• If the code is imported for third_party dependencies only:

  This is the default, and is what import.go will generate:

    visibility 
    
   = 
    
   [ 
   "//third_party/golang:__subpackages__" 
   ] 
   
  

  This will allow the packages to be imported by current and future third-party code, but the package cannot be used from the rest of google3.

• If the code will be used by non-third_party google3 code:

  We recommend creating a package group in the top-level BUILD file and referencing this in each package or rule that is part of the "public" API of the imported code.

    package_group 
   ( 
    
   name 
    
   = 
    
   "users" 
   , 
    
   packages 
    
   = 
    
   [ 
    
   "//third_party/golang/..." 
   , 
    
   "//path/to/your/project/..." 
   , 
    
   ], 
   ) 
   
  

  This keeps the visibility list centralized and provides an opportunity to explain who can use it, how to get added to the allowed users list, etc.

Reference

This section provides detailed documentation regarding the import.go tool.

Configuration files

The syntax of the configuration files is loosely based on INI files. It consists of a sequence of section headers (e.g., [Section] , where each section consists of an ordered list of entries (e.g., name: val0 val1 ). The entry itself consists of an entry name, followed by zero or more values, and must be a single line. The value is the raw bytes of the value up until the next Unicode whitespace. If whitespace needs to be encoded in the value, it can be escaped using double-quoted or backtick-quoted strings. Comments are any bytes following an # character up until the next newline.

Example configuration file:

  # 
  
 This 
  
 is 
  
 a 
  
 comment 
  
 regarding 
  
 Section 
 . 
 [ 
 Section 
 ] 
 entry_name1 
 : 
  
 value1a 
  
 "value1b has spaces" 
  
 # 
  
 trailing 
  
 comment 
 entry_name2 
 : 
  
 value2 
 

Global configuration file

The //piper/third_party/golang/CONFIGURATION file is the global configuration file for the import.go tool. It has the following sections:

Modules

Modules is a mapping of sub-directories under third_party/golang to the root Go import path for an open-source Go module. Directory names and module import paths must be unique.

Example:

  [ 
 Modules 
 ] 
  
 foo 
 : 
  
 github 
 . 
 com 
 / 
 user 
 / 
 foo 
  
 bar 
 : 
  
 github 
 . 
 com 
 / 
 user 
 / 
 bar 
 

PkgPathRewrites

PkgPathRewrites is a filter that specifies whether to rewrite an open-source Go import path to some Google-specific Go import path. A rewrite rule is rarely needed at the global level except to rewrite import paths to point to Go packages that live outside third_party/golang . This filter is comprised of a set of replace rules:

• replace specifies a Go import path to match on and an import path to replace it with. It will always be matched against the open-source Go import path, and the output must be an import path that operates correctly with Blaze. A replace rule cannot replace the output of another replace rule.

  If the upstream package and the replaced package have different package names, then the original package name must be appended to the resulting path delimited with a semi-colon. This is commonly the case for rewrites to go_proto_library rules where the upstream package name may differ from the downstream package name, which typically has a _go_proto suffix.

Example:

  [ 
 PkgPathRewrites 
 ] 
  
 replace 
 : 
  
 golang 
 . 
 org 
 / 
 x 
 / 
 net 
 / 
 context 
  
 = 
>  
 context 
  
 replace 
 : 
  
 google 
 . 
 golang 
 . 
 org 
 / 
 protobuf 
 / 
 types 
 / 
 known 
 / 
 anypb 
  
 = 
>  
 //piper/.../any_go_proto;anypb 
 

Import configuration file

All modules imported by import.go must have an import configuration file located at GOIMPORT/CONFIGURATION within the sub-directory under third_party/golang that the module is imported into. The first time import.go imports a module, it copies the default //piper/third_party/golang/CONFIGURATION.default configuration into the module directory. Users are free to modify the import configuration file as they see fit.

It may have the following sections:

ImportFiles

ImportFiles specifies which files to include from the open-source module and which to exclude. By default all files are imported. This filter consists of a sequence of include and exclude rules:

• include is followed by a regular expression pattern used to match files to include into Piper.

• exclude is followed by a regular expression pattern used to match files to exclude from Piper.

The sequence of rules are evaluated in the order defined in the filter until the first match is found, resulting in either inclusion or exclusion. The regular expression pattern is matched against a path that is always relative to the module root with a leading '/' (e.g., /path/to/file.txt ). The benefit of having a '/' prefix is that directories can be always be matched with /dirname/ and files always be matched with /filename$ .

This filter is applied against paths as they appear in the open-source Go module.

Example:

  [ 
 ImportFiles 
 ] 
  
 # 
  
 Exclude 
  
 hidden 
  
 files 
  
 and 
  
 directories 
  
 ( 
 e 
 . 
 g 
 ., 
  
 ".gitignore" 
 ). 
  
 exclude 
 : 
  
 / 
 [.] 
  
 # 
  
 Include 
  
 / 
 api 
 / 
 services 
 , 
  
 but 
  
 exclude 
  
 everything 
  
 else 
  
 in 
  
 / 
 api 
 . 
  
 include 
 : 
  
 ^ 
 / 
 api 
 / 
 services 
 / 
  
 exclude 
 : 
  
 ^ 
 / 
 api 
 / 
  
 # 
  
 Include 
  
 everything 
  
 else 
  
 ( 
 implicit 
 ; 
  
 shown 
  
 for 
  
 clarity 
 ). 
  
 include 
 : 
  
 . 
 * 
 

ImportRenames

ImportRenames specifies whether to rename certain files or directories. The use of this feature is sometimes needed to work around Piper or Blaze limitations on allowed filenames. This feature implies that the relative path as seen in the open-source Go module may differ from the path as stored in Piper. When import.go performs the Go import path translation, it knows to consult this filter to derive the Google-specific Go import path. This filter consists of a sequence of sed commands:

• sed is followed by a edit script in the same syntax as the popular sed tool .

The open-source relative path is passed through all sed commands in sequence where the output of the previous command is passed as the input to the next command. The final result is the path used. The lack of any commands implies that no renaming takes place.

Example:

  [ 
 ImportRenames 
 ] 
  
 # 
  
 Rename 
  
 dashes 
  
 as 
  
 underscores 
 . 
  
 sed 
 : 
  
 s 
 : 
 - 
 : 
 _ 
 : 
 g 
  
 # 
  
 Strip 
  
 parenthesis 
 . 
  
 sed 
 : 
  
 s 
 :[()]:: 
 g 
 

The above filter renames /(foo-bar)/baz as /foo_bar/baz . Note that both sed commands were used to produce the output path. Also note that this feature can be used to rename a directory by changing its name in the middle of a file path.

RewriteFiles

RewriteFiles specifies which Go source files to apply import path rewriting upon. The Glaze tool is run upon any directories containing files matched by this filter. By default, all Go source files are matched. It has syntax and semantics identical to the ImportFiles filter. The filter is matched against the potentially renamed file paths from the ImportRenames filter.

Example:

  [ 
 RewriteFiles 
 ] 
  
 # 
  
 Exclude 
  
 testdata 
  
 and 
  
 hidden 
  
 files 
  
 per 
  
 behavior 
  
 of 
  
 the 
  
 go 
  
 tool 
  
 itself 
 . 
  
 exclude 
 : 
  
 / 
 testdata 
 / 
  
 exclude 
 : 
  
 / 
 [. 
 _ 
 ][^ 
 / 
 ] 
 * 
 $ 
  
 include 
 : 
  
 . 
 * 
 

GlobalPkgPathRewrites

GlobalPkgPathRewrites provides the module the ability to specify a Go import path rewrite for all code (both those within this module and also in other modules that may depend on this module). It consists of a set of replace rules similar to the PkgPathRewrites filter in the global configuration. These rewrites are only relevant for import paths that are rooted within the current module. This filter take precedence over the PkgPathRewrites filter in the global configuration.

WARNING: Carefully think about rules being added to GlobalPkgPathRewrite as they will affect all other imports that depend on this module. It is possible that LocalPkgPathRewrite better satisfies your use case.

Example:

  [ 
 GlobalPkgPathRewrites 
 ] 
  
 replace 
 : 
  
 github 
 . 
 com 
 / 
 golang 
 / 
 protobuf 
 / 
 proto 
  
 = 
>  
 //piper/.../proto 
 

Assuming this occurs within the module for "github.com/golang/protobuf" , this rewrite alters all imports of "github.com/golang/protobuf/proto" to use "google3/net/proto2/go/proto" instead. However, if this filter were placed in the module for "github.com/user/foo" , it would have no affect since "github.com/golang/protobuf" is outside the module path scope.

LocalPkgPathRewrites

LocalPkgPathRewrites provides the imported module the ability to specify Go import path rewrites for all code only within this module. It can be used to rewrite any import path (including standard library import paths). It consists of a set of replace rules similar to the PkgPathRewrites filter in the global configuration. This filter takes precedence over the GlobalPkgPathRewrites mentioned above.

GoogleFiles

GoogleFiles specifies which files are internal to Google. The import.go tool will delete any files not present in the upstream repository and not covered by this filter. It has syntax and semantics identical to the ImportFiles filter. The filter is matched against the potentially renamed file paths from the ImportRenames filter.

Example:

  [ 
 GoogleFiles 
 ] 
  
 # 
  
 BUILD 
  
 files 
  
 are 
  
 special 
  
 to 
  
 Blaze 
 . 
  
 include 
 : 
  
 / 
 BUILD 
 $ 
  
 # 
  
 Treat 
  
 files 
  
 with 
  
 a 
  
 google_ 
  
 prefix 
  
 as 
  
 internal 
 . 
  
 # 
  
 This 
  
 enables 
  
 the 
  
 injection 
  
 of 
  
 internal 
  
 code 
  
 in 
  
 a 
  
 separate 
  
 file 
  
 # 
  
 without 
  
 having 
  
 to 
  
 worry 
  
 about 
  
 patch 
  
 conflicts 
 . 
  
 include 
 : 
  
 / 
 google_ 
 [^ 
 / 
 ] 
 + 
  
 # 
  
 treat 
  
 files 
  
 with 
  
 google_ 
  
 prefix 
  
 as 
  
 internal 
  
 # 
  
 Exclude 
  
 all 
  
 other 
  
 files 
  
 as 
  
 internal 
  
 ( 
 implicit 
 ; 
  
 shown 
  
 for 
  
 clarity 
 ). 
  
 exclude 
 : 
  
 . 
 * 
 

Example import configuration:

  # 
  
 ImportFiles 
  
 specifies 
  
 which 
  
 files 
  
 to 
  
 import 
  
 from 
  
 the 
  
 upstream 
  
 source 
 . 
 [ 
 ImportFiles 
 ] 
  
 # 
  
 Exclude 
  
 dot 
 - 
 prefixed 
  
 files 
  
 and 
  
 directories 
  
 ( 
 e 
 . 
 g 
 ., 
  
 ".gitignore" 
 ). 
  
 exclude 
 : 
  
 / 
 [.] 
  
 # 
  
 Include 
  
 everything 
  
 else 
  
 ( 
 this 
  
 is 
  
 implicit 
  
 behavior 
 ; 
  
 shown 
  
 for 
  
 clarity 
 ). 
  
 include 
 : 
  
 . 
 * 
 # 
  
 LocalPkgPathRewrites 
  
 specifies 
  
 a 
  
 set 
  
 of 
  
 import 
  
 path 
  
 rewrites 
  
 to 
  
 apply 
  
 locally 
 . 
 [ 
 LocalPkgPathRewrites 
 ] 
  
 # 
  
 Replace 
  
 all 
  
 usages 
  
 of 
  
 glog 
  
 with 
  
 the 
  
 Google 
 - 
 specific 
  
 log 
  
 package 
 . 
  
 replace 
 : 
  
 github 
 . 
 com 
 / 
 golang 
 / 
 glog 
  
 = 
>  
 //piper/.../log 
 # 
  
 ImportRenames 
  
 specifies 
  
 whether 
  
 to 
  
 rename 
  
 any 
  
 source 
  
 files 
  
 or 
  
 directories 
 . 
 [ 
 ImportRenames 
 ] 
  
 # 
  
 Rename 
  
 common 
  
 variants 
  
 of 
  
 license 
  
 names 
  
 to 
  
 simply 
  
 be 
  
 LICENSE 
 . 
  
 sed 
 : 
  
 s 
 :^ 
 / 
 LICENSE 
 ([.]( 
 gpl 
 | 
 md 
 | 
 txt 
 )) 
 ?$ 
 : 
 / 
 LICENSE 
 : 
 I 
 # 
  
 GoogleFiles 
  
 specifies 
  
 files 
  
 added 
  
 to 
  
 the 
  
 import 
  
 to 
  
 support 
  
 use 
  
 within 
  
 google3 
 . 
 [ 
 GoogleFiles 
 ] 
  
 # 
  
 Include 
  
 files 
  
 special 
  
 to 
  
 import 
 . 
 go 
 . 
  
 include 
 : 
  
 ^ 
 / 
 GOIMPORT 
 / 
  
 # 
  
 Include 
  
 files 
  
 special 
  
 to 
  
 Piper 
 . 
  
 include 
 : 
  
 ^ 
 / 
 METADATA 
 $ 
  
 # 
  
 see 
  
 go 
 / 
 metadata 
  
 include 
 : 
  
 ^ 
 / 
 OWNERS 
 $ 
  
 # 
  
 see 
  
 go 
 / 
 owners 
  
 include 
 : 
  
 / 
 BUILD 
 $ 
  
 # 
  
 see 
  
 go 
 / 
 build 
  
 include 
 : 
  
 / 
 g3doc 
 / 
  
 # 
  
 see 
  
 go 
 / 
 g3doc 
  
 # 
  
 Include 
  
 files 
  
 with 
  
 "google3_" 
  
 prefix 
  
 ( 
 helps 
  
 with 
  
 local 
  
 changes 
 ). 
  
 include 
 : 
  
 / 
 google3_ 
 [^ 
 / 
 ] 
 + 
 $ 
  
 # 
  
 Exclude 
  
 everything 
  
 else 
  
 ( 
 this 
  
 is 
  
 implicit 
  
 behavior 
 ; 
  
 shown 
  
 for 
  
 clarity 
 ). 
  
 exclude 
 : 
  
 . 
 * 
 

Process pipeline

The import.go tool performs an import by running through the following sequence of operations.

Downloading the source

Go modules are downloaded from the Go module proxy (http://linkremoved/) rather than being cloned from a Git or Hg repository. Using the module proxy speeds up imports by an order of magnitude compared to cloning a repository (which needs to download the entire history of the module). Furthermore, it provides a stable backup for the original source should the upstream repository one day disappear (which is the case for some code already imported into Piper).

Using the module proxy is generally advantageous, but it does mean that trying to import the absolute latest version may be delayed by a latency on the order of minutes.

Applying manual patches

Manual patches may be applied to the downloaded source. Patches must be located in the GOIMPORT/PATCHES sub-directory and be named with a .patch suffix. They are applied in sorted filename order directly upon the downloaded source (prior to any file path renamings or Go import path rewrites).

The patch must be in the unified diff format with a single path prefix added to the file paths. This is the format typically provided by git diff or as downloaded from GitHub or Geritt.

Example patch:

  diff --git a/integration_test.go bug ...est.go 
 index 9df20d0..f62e072 100644 
 --- a/integration_test.go 
 +++ bug ...est.go 
 @@ -28,8 +28,8 @@ 
  
var (  
regenerate = flag.Bool("regenerate", false, "regenerate files") -   protobufVersion = "3.7.0" 
 -   golangVersions  = []string{"1.9.7", "1.10.8", "1.11.5", "1.12"} 
 +   protobufVersion = "3.7.1" 
 +   golangVersions  = []string{"1.9.7", "1.10.8", "1.11.6", "1.12.1"} 
  
golangLatest    = golangVersions[len(golangVersions)-1]  
// purgeTimeout determines the maximum age of unused sub-directories. 

Filtering and renaming files

To determine the set of files to import, the tool consults the ImportFiles filter to determine whether to include or exclude a file. If a file is to be kept, the ImportRenames filter is consulted next to determine whether the file (or the directory that it lives under) should be renamed.

Rewriting Go imports paths

All Go source files being imported (except for those excluded by the RewriteFiles filter) are reformatted to have Go import paths translated from the open-source paths to ones that are compatible with Blaze. For example, it is at this stage that an import path may be rewritten from "google.golang.org/protobuf/proto" to "google3/third_party/golang/protobuf/v2/proto/proto" .

The translation goes through the following process:

1. It checks whether there is an applicable LocalPkgPathRewrites rule from the current import configuration and uses it if possible.
2. It checks whether there is an applicable GlobalPkgPathRewrites rule from the module that the import path is rooted within and uses it if possible.
3. It checks whether there is an applicable PkgPathRewrites rule in the global configuration and uses it if possible.
4. It attempts to resolve the Go import path by locating the Go module that the import path is rooted in and walking the file tree until it (hopefully) finds the specified Go package. For example, to resolve "google.golang.org/protobuf/encoding/protojson" , we identify that it has a root prefix match with "google.golang.org/protobuf" for code located in Piper under the protobuf/v2 directory. Using the suffix, we identify the Go package located in protobuf/v2/encoding/protojson as protojson . Concatenating these pieces of information together results in an import path of "google3/third_party/golang/protobuf/v2/encoding/protojson/protojson" .

Most import paths are resolved using the last mechanism. The need to specify a custom rewrite should be the exception, not the norm.

Applying automatic patches

A feature of import.go is the ability to perform auto-patching. It provides a generally easier user experience in making local changes to an import. It operates in two stages. For the sake of explanation, let v1.0 be the current version that is already imported into Piper and v1.1 be the version that is about to be imported into Piper.

1. The module is downloaded at the current version, v1.0 , and all stages prior to this are performed on that temporary import (i.e., filtering, renaming, and import path rewriting). A patch is derived between that temporary import and the current import within Piper. This patch file contains the set of local changes that have been made on top of v1.0 . If no local changes were made in prior imports, then this patch is empty.

2. The contents of that patch file are applied to the current working import of v1.1 (after filtering, renaming, and import path rewriting). In the common case, the patch of local changes relative to v1.0 will apply cleanly to v1.1 . If not, the merge conflict will be reflecting the file with <<< and >>> markers.

This process is similar to how git rebase works.

Synchronizing to Piper

Up until this point, all changes are being made in a temporary working directory. At this stage, the tool compares the contents of the working import against the contents of the import directory in Piper:

• Files that exist only in the working import, but not in Piper are copied to Piper as an added file.
• Files that exist in both the working import and Piper, but differ in content will be modified (with the imported copy over-writing the Piper copy).
• Files that exist only in Piper, but not the working import are deleted unless the GoogleFiles filter in the import configuration specifies that this is Google-internal file, in which case no changes are made.

Lastly, glaze is executed on all directories containing imported Go source files. This generates BUILD files with go_library and go_test rules derived from the imported Go source code.

Generated files

After importing a module, import.go generates several metadata files:

• GOIMPORT/AUTOPATCHES is a directory containing diffs for all of the automatic patches that were applied. The diffs are a trimmed variant of the unified diff format where line numbers have been removed to reduce the probability of noise when unrelated changes occur within the same file. If there are no local changes to the module, then this directory is not generated.

• GOIMPORT/MANIFEST is a list of all files within the import directory along with the expected checksum for those files. In the future, a pre-submit check will be added to third_party/golang to verify the contents of the MANIFEST file. The purpose of this check is to ensure that local changes are properly reflected in either manual patches or automatically-generated patches and surfaced for proper review.

If local changes have been made and the above files are stale, then they can be regenerated by running:

 go  
run  
third_party/golang/import.go  
-dir = 
foo  
-version = 
same 

To verify that the manifest is up-to-date, then run:

 go  
run  
third_party/golang/import.go  
-dir = 
foo  
-check 

Filtering external and internal files

If you need to exclude certain files or directories from being imported from the upstream source, specify so in the ImportFiles section of the import configuration.

You may want to consider the following files as being excluded from import:

• The vendor directory (dependencies should be added explicitly to third_party instead of vendored)
• BUILD files (if the project uses bazel )
• GitHub OWNERS files
• Dot-prefixed hidden files (e.g., .gitignore )

If you need to make sure the import does not delete any files that should be kept for Google-specific usages, then specify so in the GoogleFiles section of the import configuration.

You may want to consider the following files as Google-specific:

• The GOIMPORT directory (for use by import.go )
• OWNER , BUILD , LICENSE , or METADATA files
• The /g3doc/ directory
• Files with a google_ prefix (for use with Google-specific logic)

See existing configuration files for example usages.

Making local changes

There are two primary mechanisms for making local changes:

• Manual patches:These patches are applied on the vanilla copy of the upstream source. The recommended use case for manual patches is the cherry-picking of upstream changes that have not yet landed in the release imported into Google. Keeping cherry-picks as discrete patches enables them to be easily added and removed.

• Automatic patches:These patches are applied at the end of the process pipeline after all Go import path rewrites have occurred. Automatic patches provide an easy way for users to make local changes (i.e., they just directly modify the files in their CITC). However, they lack the ability to distinguish the purpose of one local changes from another. We recommend that users comment judiciously about the purpose of the change. For substantial changes, we recommend putting the main functionality in a Google-specific file (e.g., google_internal.go ) and patch in the minimal number of hooks in the upstream code to call over to the internal logic. Doing so should reduce the probability of a merge conflict in future imports.

  After making local changes, the auto-patches need to be regenerated:

   go  
  run  
  third_party/golang/import.go  
  -dir = 
  foo  
  -version = 
  same 
  

Changes to just import paths are generally more maintainable if performed through the use of LocalPkgPathRewrites in the import configuration file.

Auto-patching issues

• If a patch won't apply due to merge conflicts, import.go will leave diff markers <<< and >>> in your files where the conflicts occur. You'll have to manually resolve the conflicts by editing the files.
• To revert local changes that you've made to a file, you can run import.go with a sequence of -autopatch_include or -autopatch_exclude flags to enable or disable patching on specific files. The auto-patching filter follows the same semantics as the ImportFiles and GoogleFiles filter in the import configuration.