dask.dataframe.DataFrame.repartition
dask.dataframe.DataFrame.repartition¶
- DataFrame.repartition(divisions=None, npartitions=None, partition_size=None, freq=None, force=False)¶
Repartition dataframe along new divisions
- Parameters
- divisionslist, optional
The “dividing lines” used to split the dataframe into partitions. For
divisions=[0, 10, 50, 100]
, there would be three output partitions, where the new index contained [0, 10), [10, 50), and [50, 100), respectively. See https://docs.dask.org/en/latest/dataframe-design.html#partitions. Only used if npartitions and partition_size isn’t specified. For convenience if given an integer this will defer to npartitions and if given a string it will defer to partition_size (see below)- npartitionsint, optional
Approximate number of partitions of output. Only used if partition_size isn’t specified. The number of partitions used may be slightly lower than npartitions depending on data distribution, but will never be higher.
- partition_size: int or string, optional
Max number of bytes of memory for each partition. Use numbers or strings like 5MB. If specified npartitions and divisions will be ignored. Note that the size reflects the number of bytes used as computed by
pandas.DataFrame.memory_usage
, which will not necessarily match the size when storing to disk.Warning
This keyword argument triggers computation to determine the memory size of each partition, which may be expensive.
- freqstr, pd.Timedelta
A period on which to partition timeseries data like
'7D'
or'12h'
orpd.Timedelta(hours=12)
. Assumes a datetime index.- forcebool, default False
Allows the expansion of the existing divisions. If False then the new divisions’ lower and upper bounds must be the same as the old divisions’.
Notes
Exactly one of divisions, npartitions, partition_size, or freq should be specified. A
ValueError
will be raised when that is not the case.Also note that
len(divisions)
is equal tonpartitions + 1
. This is becausedivisions
represents the upper and lower bounds of each partition. The first item is the lower bound of the first partition, the second item is the lower bound of the second partition and the upper bound of the first partition, and so on. The second-to-last item is the lower bound of the last partition, and the last (extra) item is the upper bound of the last partition.Examples
>>> df = df.repartition(npartitions=10) >>> df = df.repartition(divisions=[0, 5, 10, 20]) >>> df = df.repartition(freq='7d')