- dask.bag.read_text(urlpath, blocksize=None, compression='infer', encoding='utf-8', errors='strict', linedelimiter=None, collection=True, storage_options=None, files_per_partition=None, include_path=False)[source]¶
Read lines from text files
- urlpathstring or list
Absolute or relative filepath(s). Prefix with a protocol like
s3://to read from alternative filesystems. To read from multiple files you can pass a globstring or a list of paths, with the caveat that they must all have the same protocol.
- blocksize: None, int, or str
Size (in bytes) to cut up larger files. Streams by default. Can be
Nonefor streaming, an integer number of bytes, or a string like “128MiB”
- compression: string
Compression format like ‘gzip’ or ‘xz’. Defaults to ‘infer’
- encoding: string
- errors: string
- linedelimiter: string or None
- collection: bool, optional
Return dask.bag if True, or list of delayed values if false
- storage_options: dict
Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc.
- files_per_partition: None or int
If set, group input files into partitions of the requested size, instead of one partition per file. Mutually exclusive with blocksize.
- include_path: bool
Whether or not to include the path in the bag. If true, elements are tuples of (line, path). Default is False.
- dask.bag.Bag or list
dask.bag.Bag if collection is True or list of Delayed lists otherwise.
Build bag from Python sequence
>>> b = read_text('myfiles.1.txt') >>> b = read_text('myfiles.*.txt') >>> b = read_text('myfiles.*.txt.gz') >>> b = read_text('s3://bucket/myfiles.*.txt') >>> b = read_text('s3://key:secret@bucket/myfiles.*.txt') >>> b = read_text('hdfs://namenode.example.com/myfiles.*.txt')
Parallelize a large file by providing the number of uncompressed bytes to load into each partition.
>>> b = read_text('largefile.txt', blocksize='10MB')
Get file paths of the bag by setting include_path=True
>>> b = read_text('myfiles.*.txt', include_path=True) >>> b.take(1) (('first line of the first file', '/home/dask/myfiles.0.txt'),)