- 1.57.1 (latest)
- 1.57.0
- 1.56.3
- 1.55.0
- 1.54.0
- 1.53.0
- 1.52.0
- 1.51.0
- 1.50.0
- 1.49.0
- 1.48.0
- 1.47.0
- 1.46.0
- 1.45.0
- 1.44.0
- 1.43.0
- 1.42.0
- 1.41.0
- 1.40.0
- 1.39.1
- 1.38.0
- 1.37.0
- 1.36.0
- 1.35.1
- 1.34.1
- 1.33.0
- 1.32.0
- 1.31.0
- 1.30.1
- 1.29.0
- 1.28.1
- 1.27.0
- 1.26.0
- 1.25.0
- 1.24.0
- 1.23.0
- 1.22.1
- 1.21.0
- 1.20.0
- 1.19.0
- 1.18.2
- 1.17.0
- 1.16.1
- 1.15.0
- 1.14.0
- 1.13.0
- 1.12.0
Package dataflux provides an easy way to parallelize listing in Google Cloud Storage.
More information about Google Cloud Storage is available at https://cloud.google.com/storage/docs .
See https://pkg.go.dev/cloud.google.com/go for authentication, timeouts, connection pooling and similar aspects of this package.
NOTE: This package is in preview. It is not stable, and is likely to change.
Lister
type
Lister
struct
{
// contains filtered or unexported fields
}
Lister is used for interacting with Dataflux fast-listing. The caller should initialize it with NewLister() instead of creating it directly.
func NewLister
func
NewLister
(
c
*
storage
.
Client
,
in
*
ListerInput
)
*
Lister
NewLister creates a new dataflux Lister to list objects in the give bucket.
func (*Lister) Close
func
(
c
*
Lister
)
Close
()
Close closes the range channel of the Lister.
func (*Lister) NextBatch
NextBatch runs worksteal algorithm and sequential listing in parallel to quickly return a list of objects in the bucket. For smaller dataset, sequential listing is expected to be faster. For larger dataset, worksteal listing is expected to be faster.
ListerInput
type
ListerInput
struct
{
// BucketName is the name of the bucket to list objects from. Required.
BucketName
string
// Parallelism is number of parallel workers to use for listing.
// Default value is 10x number of available CPU. Optional.
Parallelism
int
// BatchSize is the number of objects to list. Default value returns
// all objects at once. The number of objects returned will be
// rounded up to a multiple of gcs page size. Optional.
BatchSize
int
// Query is the query to filter objects for listing. Default value is nil.
// Use ProjectionNoACL for faster listing. Including ACLs increases
// latency while fetching objects. Optional.
Query
storage
.
Query
// SkipDirectoryObjects is to indicate whether to list directory objects.
// Default value is false. Optional.
SkipDirectoryObjects
bool
}
ListerInput contains options for listing objects.
