Path

class verta.dataset.Path(paths, base_path=None, enable_mdb_versioning=False)

Captures metadata about files.

Note

If relative paths are passed in, they will not be converted to absolute paths.

Parameters:

  • paths (list of str) – List of filepaths or directory paths.

  • base_path (str, optional) – Directory path to be removed from the beginning of paths before saving to ModelDB.

  • enable_mdb_versioning (bool, default False) – Whether to upload the data itself to ModelDB to enable managed data versioning.

Examples

from verta.dataset import Path
dataset1 = Path([
    "../datasets/census-train.csv",
    "../datasets/census-test.csv",
])
dataset2 = Path([
    "../datasets",
])
dataset += other

Updates the dataset, adding paths from other.

dataset + other + ...

Returns a new dataset with paths from the dataset and all others.

add(paths, base_path=None)

Adds paths to this dataset.

Parameters:

  • paths (list of str) – List of filepaths or directory paths.

  • base_path (str, optional) – Directory path to be removed from the beginning of paths before saving to ModelDB.

static blob_msg_to_object(blob_msg)

Deserialize a blob protobuf message into an instance.

Parameters:

blob_msg (VersioningService_pb2.Blob) –

Returns:

instance of subclass of Blob

download(component_path=None, download_to_path=None)

Downloads component_path from this dataset if ModelDB-managed versioning was enabled.

Parameters:

  • component_path (str, optional) – Original path of the file or directory in this dataset to download. If not provided, all files will be downloaded.

  • download_to_path (str, optional) – Path to download to. If not provided, the file(s) will be downloaded into a new path in the current directory. If provided and the path already exists, it will be overwritten.

Returns:

downloaded_to_path (str) – Absolute path where file(s) were downloaded to. Matches download_to_path if it was provided as an argument.

list_components()

Returns the components in this dataset.

Returns:

components (list of Component) – Components.

list_paths()

Returns the paths of all components in this dataset.

Returns:

component_paths (list of str) – Paths of all components.

classmethod with_spark(sc, paths)

Creates a dataset blob with a SparkContext instance.

Parameters:

  • sc (pyspark.SparkContext) – SparkContext instance.

  • paths (list of strs) – List of paths to binary input data file(s).

Returns:

dataset (dataset) – Dataset blob capturing the metadata of the binary files.