HDFSPath

class verta.dataset.HDFSPath(hdfs_client, paths, base_path=None)

Captures metadata about files from HDFS.

Parameters:

• hdfs_client (hdfs.client.Client) – Instance of HDFS client.

• paths (list of strs) – List of paths to files from HDFS.

• base_path (str, optional) – Directory path to be removed from the beginning of paths before saving to ModelDB.

add(paths, base_path=None)

Adds paths to this dataset.

Parameters:

• paths (list of str) – List of filepaths or directory paths.

• base_path (str, optional) – Directory path to be removed from the beginning of paths before saving to ModelDB.

static blob_msg_to_object(blob_msg)

Deserialize a blob protobuf message into an instance.

Parameters:

blob_msg (VersioningService_pb2.Blob) –

Returns:

instance of subclass of Blob

download(component_path=None, download_to_path=None)

Downloads component_path from this dataset if ModelDB-managed versioning was enabled.

Parameters:

• component_path (str, optional) – Original path of the file or directory in this dataset to download. If not provided, all files will be downloaded.

• download_to_path (str, optional) – Path to download to. If not provided, the file(s) will be downloaded into a new path in the current directory. If provided and the path already exists, it will be overwritten.

Returns:

downloaded_to_path (str) – Absolute path where file(s) were downloaded to. Matches download_to_path if it was provided as an argument.

list_components()

Returns the components in this dataset.

Returns:

components (list of Component) – Components.

list_paths()

Returns the paths of all components in this dataset.

Returns:

component_paths (list of str) – Paths of all components.

classmethod with_spark(sc, paths)

Creates a dataset blob with a SparkContext instance.

Parameters:

• sc (pyspark.SparkContext) – SparkContext instance.

• paths (list of strs) – List of paths to binary input data file(s).

Returns:

dataset (dataset) – Dataset blob capturing the metadata of the binary files.