wandb_utils.commands.backup

Module Contents

wandb_utils.commands.backup.logger
class wandb_utils.commands.backup.PyClone(*, binPath: str = None, binSuffix: str = '', messageBufferSize: int = 5, global_flags: Optional[Dict] = None)

Bases: pyclone.PyClone

Same as pyclone.PyClone but with some methods overridden.

line
command(self, action: str, source: str, target: str = '', target_remote: str = '', source_remote: str = '') → str
copy(self, source: str, target: str, source_remote: str, target_remote: str) → None

Copy files from source to dest, skipping already copied.

Parameters

  • source (str) – Files to transfer.

  • remote (str) – Configured service name.

  • path (str) – Destination to save to.

Note

This is a convenience method that wraps around PyClone.__launchThread(). For more information about this action, please read rclone’s documentation.

move(self, source: str, target: str, source_remote: str, target_remote: str) → None
signalsDict(self, k=None)

Used for looking up a signal by its integer or string value, or a dictionary listing of all available signals.

Parameters

k (int or str, optional) –

Returns

Returns a dictionary by default, or the desired signal lookup (string if an integer is given, or integer if a string is given).

Return type

int, str, or dict

sigTrap(self, sigNum, currFrame)

This serves as the process’ signal trap for several default signals (e.g. SIGINT and SIGTERM).

While you can override this method, you probably won’t ever need to, and can leave this to call upon PyClone.stop().

Parameters

  • sigNum (int) – Signal number (e.g. 2 for SIGINT, or 15 for SIGTERM.)

  • currFrame (frame) – Python Stack frame.

See also

signals(self, cb, *keys)

Bind a callback to an arbitrary number of signals.

Parameters

  • cb (Function or Method) – Callback that’s executed when a signal occurs that it’s been bound to.

  • *keys (str) – Variable length list of signals to bind callback to.

Returns

True if successful, False if an error occurs.

Return type

bool

An example for registering a callback to multiple signals:

def myCallback( self, sigNum, currFrame ):

    print( f'My callback received sigNum={ sigNum } and currFrame={ currFrame }' )

    pass # END METHOD : My callback

def __init__( self ):

    self.signals(
        self.myCallback,
        'SIGINT',   # ^C
        'SIGTERM',  # `kill procID` or `pkill myApp.py` and systemd's default kill signal.
    )

    pass # END CONSTRUCTOR

See also

addFlag(self, key, value=None)

Rclone has many global flags available to every command, if a flag doesn’t have the option for a value, simply leave it unset.

Examples:

  • Dry run without any changes: PyClone.addFlag( 'dry-run' )

  • Throttling your total bandwidth: PyClone.addFlag( 'bwlimit', '10M' )

Parameters

  • key (str) –

  • value (str, optional) –

See also

Returns

True if key doesn’t exist and has been added, False if key already exists and couldn’t be added.

Return type

bool

removeFlag(self, key)

Removes a global flag that was set with PyClone.addFlag()

Parameters

key (str) –

See also

Returns

True if key was found and removed, otherwise False.

Return type

bool

updateFlag(self, key, value=None)

Overwrites a flag that was set with PyClone.addFlag()

Parameters

  • key (str) –

  • value (str, optional) –

See also

Returns

True if PyClone.removeFlag() and PyClone.addFlag() both returned True, otherwise False.

Return type

bool

binPath(self, binPath=None)

Gets (or if a value is provided, sets) path to rclone binary.

Examples:

  • Host: /usr/bin/env rclone

  • Container: docker-compose --no-ansi --file /path/to/docker-compose.yaml run rclone

Parameters

binPath (str, optional) – Path to rclone binary.

Returns

Binary path stored in PyClone.__binPath().

Return type

str

stop(self)

Used for shutting down rclone (including interrupting transfers, if needed).

Returns

If a process was successfully spawned, and then successfully shut down with this method, a tuple containing the exit status and the signal status are returned.

Return type

tuple or None

tailing(self)

Used by your program for determining if a loop should continue checking for output from rclone, based upon multiple conditions.

An example for continuously printing out data from rclone:

while rclone.tailing():

    if rclone.readline():
        print( rclone.line, flush=True )

    time.sleep( 0.5 )

See also

PyClone.readline()

Returns

Returns True if a process is still running and there’s the potential for messages to be added to the buffer, else this returns False.

Return type

bool

readline(self)

Mostly used in conjunction with PyClone.tailing(), this retrieves the oldest line from the message buffer that’s filled by rclone, with a buffer size set when initializing the class.

See also

PyClone.tailing()

Returns

Returns True if a line was removed from the message buffer and stored in PyClone.line, otherwise returns False.

Return type

bool

clearBuffer(self)

Clear all messages in the buffer that were added by an rclone action.

remotes(self)

Lists configured remotes in rclone.

Returns

Dictionary of available remotes, where the key is the name and the value is the type.

Return type

dict

ls(self, remote, path='')

List files and directories in a given path. If no path is provided, the root directory of your remote destination will be listed.

Parameters

  • remote (str) – Name of remote service that’s configured in rclone.

  • path (str, optional) – Remote path to list.

Returns

Files and directories.

Return type

list

sync(self, *, source, remote, path)

Make source and dest identical, modifying destination only.

Parameters

  • source (str) – Files to transfer.

  • remote (str) – Configured service name.

  • path (str) – Destination to save to.

Note

This is a convenience method that wraps around PyClone.__launchThread(). For more information about this action, please read rclone’s documentation.

delete(self, *, remote, path, rmdirs=False)

Remove the contents of path.

Parameters

  • remote (str) – Configured service name.

  • path (str) – Destination to delete.

  • rmdirs (bool) – If set to True, rclone will remove all empty directories.

See also

PyClone.purge() for deleting directories.

Note

This is a convenience method that wraps around PyClone.__launchThread(). For more information about this action, please read rclone’s documentation.

purge(self, *, remote, path)

Remove the path and all of its contents.

Parameters

  • remote (str) – Configured service name.

  • path (str) – Destination to delete.

See also

PyClone.delete() for deleting files.

Note

This is a convenience method that wraps around PyClone.__launchThread(). For more information about this action, please read rclone’s documentation.

touch(self, *, remote, path, create=True, timestamp=None)

Create new file or change file modification time.

Parameters

  • remote (str) – Configured service name.

  • path (str) – Destination to save to.

  • create (bool) – If file doesn’t exist, it will be created.

  • timestamp (str or datetime) – If timestamp is provided, it will be converted to UTC and applied to the remote service’s path.

Returns

True if successful, False if unsuccessful.

Return type

bool

Note

For more information about this action, please read rclone’s documentation.

Examples:

# No timezone provided, will use current UTC.
rclone.touch( remote='myGoogleDrive', path='/path/to/file.txt' )

# Date provided, but time (and time zone) has not been.  Time is in UTC.
rclone.touch( remote='myGoogleDrive', path='/path/to/file.txt', timestamp='2000-01-01' )

# ISO 8601, in human format with microseconds and without a time zone.  Time is in UTC.
rclone.touch( remote='myGoogleDrive', path='/path/to/file.txt', timestamp='2000-01-01 12:34:56.789' )

# ISO 8601, with microseconds and with a time zone.  Time is in Arizona's time zone (they don't waste their time on DST).
rclone.touch( remote='myGoogleDrive', path='/path/to/file.txt', timestamp='2000-01-01T12:34:56.789-07:00' )
wandb_utils.commands.backup.copy_to_remote(rclone: pyclone.PyClone, source: pathlib.Path, target: Optional[pathlib.Path] = None, source_remote: Optional[str] = None, target_remote: Optional[str] = None) → bool
wandb_utils.commands.backup.rclone(ctx: click.Context, include_filter: List[str], exclude_filter: List[str], filter: List[str], dump: List[str]) → None

Perform operations on remote storage using Rclone.

wandb_utils.commands.backup.copy(rclone: PyClone, source: pathlib.Path, source_remote: str, target: Optional[pathlib.Path], target_remote: str) → None

Copy from SOURCE to REMOTE:TARGET using rclone.

wandb_utils.commands.backup.move(rclone: PyClone, source: pathlib.Path, source_remote: str, target: Optional[pathlib.Path], target_remote: str) → None

Copy from SOURCE to REMOTE:TARGET using rclone.