Fetch lineage links from multiple regions

Fetch data lineage links across multiple Google Cloud regions using a fan-out approach. You use the Data Lineage API to retrieve and aggregate lineage data from all available regions. This is useful when your data assets are distributed across different locations and you need a unified view of their lineage.

Procedure

The procedure makes parallel requests to the projects.locations.searchLinks endpoint in each region. You collect results, handle pagination across regions, and aggregate the found links. Manage a regionalPageToken to ensure consistent pagination for subsequent calls, accounting for unreachable regions.

The following pseudocode outlines a procedure you can follow using the Data Lineage API to fetch data lineage links from all available regions.

1. Input setup:
   • linksToFetch = 5 (number of links to fetch from all regions)
   • regionalPageToken = "" (last received regional page token used for pagination)
2. Create:
   1. list of all regions to fan-out called regions .
   2. list of unreachable regions called unreachable .
   3. an empty list of found links called foundLinks .
   4. an empty variable nextPageToken and nextRegionalPageToken .
   5. an empty variable lastRegion .
   6. an empty variable lastRegionLinksCount .
3. Sort regions alphabetically.
4. Parse regionalPageToken
   1. If it is empty then continue to step 5
   2. If it is not empty then
      1. parse regionalPageToken by splitting it by first dot into two segments
      2. filter out regions by taking only the ones that are after (in alphabetical order) the first segment from parsed regionalPageToken .
      3. store the second segment aside to use it in all projects.locations.searchLinks calls.
5. Make a call to projects.locations.searchLinks in each region in the regions list in parallel.
6. Wait for all requests to complete.
7. Filter out successful responses and store failed region names in the unreachable list.
8. For each of the responses (starting with first region in alphabetical order)
   1. if no links were returned along with a non empty page token
      1. store region name in lastRegion .
      2. save received pageToken in nextPageToken .
      3. ignore the remaining responses.
   2. otherwise
      1. store region name in lastRegion .
      2. save received links in foundLinks (up to linksToFetch ).
      3. save received pageToken in nextPageToken .
      4. store number of taken links from the response in lastRegionLinksCount .
      5. if foundLinks is less than linksToFetch
         1. continue the pseudocode with the next received response in the alphabetical order.
      6. if foundLinks equals linksToFetch
         1. make another request to projects.locations.searchLinks endpoint in the same region but set the pageSize parameter to the value of lastRegionLinksCount .
         2. store received pageToken in nextPageToken .
         3. continue to step 9
9. Prepend nextPageToken with the lastRegion (i.e. [region].[receivedNextPageToken] ) to create nextRegionalPageToken to handle pagination in consecutive calls.