dalood package

Subpackages

• dalood.loader package
  • Submodules
  • dalood.loader.base module
    • LoaderBase
      • LoaderBase.get_mtime()
      • LoaderBase.load()
      • LoaderBase.patterns
      • LoaderBase.register_patterns()
  • dalood.loader.file module
    • FileLoaderBase
      • FileLoaderBase.DEFAULT_ENCODING
      • FileLoaderBase.__init__()
      • FileLoaderBase.get_mtime()
      • FileLoaderBase.stream()
  • dalood.loader.json module
    • JSONFileLoader
      • JSONFileLoader.__init__()
      • JSONFileLoader.load()
      • JSONFileLoader.patterns
    • JSONUrlLoader
      • JSONUrlLoader.__init__()
      • JSONUrlLoader.load()
  • dalood.loader.map module
    • MapLoaderBase
      • MapLoaderBase.__init__()
      • MapLoaderBase.map()
      • MapLoaderBase.patterns
  • dalood.loader.memory module
    • MemoryLoader
      • MemoryLoader.load()
  • dalood.loader.pandas module
    • DataFrameCSVLoader
      • DataFrameCSVLoader.__init__()
      • DataFrameCSVLoader.load()
      • DataFrameCSVLoader.patterns
    • DataFrameSQLLoader
      • DataFrameSQLLoader.__init__()
      • DataFrameSQLLoader.load()
  • dalood.loader.text module
    • TextFileLoader
      • TextFileLoader.__init__()
      • TextFileLoader.load()
      • TextFileLoader.patterns
    • TextUrlLoader
      • TextUrlLoader.__init__()
      • TextUrlLoader.load()
  • dalood.loader.url module
    • UrlLoaderBase
      • UrlLoaderBase.__init__()
      • UrlLoaderBase.get_mtime()
      • UrlLoaderBase.stream()
  • dalood.loader.yaml module
    • YAMLFileLoader
      • YAMLFileLoader.__init__()
      • YAMLFileLoader.load()
      • YAMLFileLoader.patterns
    • YAMLUrlLoader
      • YAMLUrlLoader.__init__()
      • YAMLUrlLoader.load()
  • Module contents

Submodules

dalood.data_wrapper module

Data wrapper class.

class dalood.data_wrapper.DataWrapper(loader, src)

Bases: object

Wrapper around loaded data that stores that data and a loader reference, and tracks loading and access times.

__init__(loader, src)

access()

Access the data. It will be loaded first if necessary.

load()

Load the data for the given source.

Parameters:

src – The data source (e.g. a file or URI, or whatever else the matching loader can handle).

refresh()

Reload data if the loader reports that the data has changed since it was last loaded.

dalood.exception module

Custom exceptions.

class dalood.exception.ExpectedExceptionContext(*expected, error_msg=None)

Bases: object

Context manager to convert expected exceptions other exception types.

EXCEPTION_MAP = {<class 'KeyError'>: <class 'dalood.exception.LoaderKeyError'>, <class 'OSError'>: <class 'dalood.exception.LoaderOSError'>, <class 'ValueError'>: <class 'dalood.exception.LoaderValueError'>}

__init__(*expected, error_msg=None)

Parameters:

• *expected – The expected exceptions to transform into custom exceptions. These must be exception classes that are already in EXCEPTION_MAP.

• error_msg – An error message to prepend to the caught exception when instantiating the custom exception. It should provide context for the exception when it is displayed to the user.

classmethod map_exception(ex_from, ex_to)

Add an exception pair to the exception map.

Parameters:

• ex_from – The exception to map.

• ex_to – The exception to which ex_from should be mapped.

exception dalood.exception.LoaderError

Bases: Exception

Custom exception base class.

exception dalood.exception.LoaderKeyError

Bases: KeyError

Custom KeyError exception.

exception dalood.exception.LoaderOSError

Bases: OSError

Custom OSError exception.

exception dalood.exception.LoaderValueError

Bases: ValueError

Custom ValueError exception.

dalood.manager module

File manager class.

class dalood.manager.Manager

Bases: object

File manager for loading files on demand and managing memory.

loaders

An OrderedDict mapping argument regular expressions to loaders.

__init__()

clear_cache(pattern=None, age=None, by_access_time=False, pattern_type=PatternType.REGEX)

Clear cached data from memory.

Parameters:

• pattern – Same as for register_loader(). If given, only cached items matching this pattern will be cleared.

• pattern_type – Same as for register_loader().

• age – An optional datetime.timedelta object (or a dict of keyword arguments that can be passed to datetime.timedelta(). If the data has been loaded for longer than this time, it will be cleared.

• by_access_time – If True, clear data by last access time when clearing by age, otherwise clear by load time.

clear_cache_with_glob_pattern(pattern, age=None, by_access_time=False)

Wrapper around clear_cache() for GLOB patterns.

clear_cache_with_literal_pattern(pattern, age=None, by_access_time=False)

Wrapper around clear_cache() for LITERAL patterns.

clear_cache_with_regex_pattern(pattern, age=None, by_access_time=False)

Wrapper around clear_cache() for REGEX patterns.

get(src, refresh=False, reload=False)

Get the data from the given source. The data will be loaded via the loader if necessary.

Parameters:

• src – The data source.

• refresh – If True, reload data if the reported modification time is newer

• data. (H than the loaded)

• reload – If True, force a reload.

Returns:

The data.

get_loader(src)

Get the loader that matches the source.

Parameters:

src – The data source (e.g. a file or URI, or whatever else the matching loader can handle).

Raises:

ValueError – The given source is not matched by any registered regular expressions.

get_mtime(src)

Get the modification time of a source.

Parameters:

src – The data source.

Returns:

The modification time as a datetime.datetime object, or None.

refresh(pattern=None, pattern_type=PatternType.REGEX)

Refresh data for which the loader reports a source modification since the time the data was loaded.

Parameters:

• pattern – Same as for register_loader(). If given, only cached items matching this pattern will be refreshed.

• pattern_type – Same as for register_loader().

refresh_with_glob_pattern(pattern)

Wrapper around refresh() for GLOB patterns.

refresh_with_literal_pattern(pattern)

Wrapper around refresh() for LITERAL patterns.

refresh_with_regex_pattern(pattern)

Wrapper around refresh() for REGEX patterns.

register_loader(pattern, loader, *, prioritize=False, pattern_type=PatternType.REGEX)

Register a loader to handle a pattern.

Parameters:

• pattern – Either a string, in which case the pattern type is determined by the pattern_type argument, or an instance of RegexPattern or one of its subclasses.

• loader – A subclass of the LoaderBase.

• prioritize – If True, insert this regex at the start of the list of registred regexes, otherwise it will be inserted at the end.

• pattern_type – The pattern type, either as an instance of PatternType or a string that is recognized by regex.PatternType.from_str() (e.g. “glob”, “literal”).

register_loader_with_glob_pattern(pattern, loader, *, prioritize=False)

Wrapper around register_loader() for GLOB patterns.

register_loader_with_literal_pattern(pattern, loader, *, prioritize=False)

Wrapper around register_loader() for LITERAL patterns.

register_loader_with_regex_pattern(pattern, loader, *, prioritize=False)

Wrapper around register_loader() for REGEX patterns.

dalood.regex module

Regular expression types and functions.

class dalood.regex.GlobPattern(pattern)

Bases: RegexPattern

Glob pattern.

TYPE = 2

__init__(pattern)

class dalood.regex.LiteralPattern(pattern)

Bases: RegexPattern

Literal pattern.

TYPE = 3

__init__(pattern)

class dalood.regex.PatternType(*values)

Bases: Enum

Recognized pattern types.

GLOB = 2

LITERAL = 3

REGEX = 1

classmethod from_str(arg)

Convert a string to a PatternType.

class dalood.regex.RegexPattern(pattern, flags=0)

Bases: object

Regular expression pattern.

TYPE = 1

__init__(pattern, flags=0)

fullmatch(*args, **kwargs)

Wrapper around re.Pattern.fullmatch

dalood.regex.get_extension_pattern_for_filepath(ext, escape=True)

Get a regular expression pattern for filepaths that end with the given extension. This will exclude URIs, including file URIs.

Parameters:

• ext – The extention to recognize, e.g. “.txt”.

• escape – If True, escape the extension for the pattern. This can be set to false whena pre-escaped pattern is passed in, e.g. r”.[tc]sv”.

Returns:

A RegexPattern or a subclass thereof.

dalood.regex.get_regex(pattern, pattern_type=PatternType.REGEX)

Get the regular expression corresponding to the given pattern.

Parameters:

• pattern – A string containing a pattern of the specified type that should be converted to a regular expression.

• pattern_type – An instance of PatternType or an equivalent string.

Returns:

An re.Pattern regular expression object.

Module contents

Package stub.